1.

Introduction

The proliferation of ubiquitous communication infrastructures makes various video services increasingly popular on portable multimedia devices. Although recent technological advancements have made real-time video playback possible on many portable multimedia devices, when it comes to high resolution and/or high quality video content, especially on mobile devices, there are still impending issues in real-time playback due to limited resources of portable devices in battery capacity, processor speed, and memory.

The problem addressed in this paper is slightly different from the conventional usage scenario of video decoders since it addresses what to do if an H.264/AVC video decoder is given a compressed bitstream exceeding its level specification. The level in a video coding standard specifies the minimal resources with which a given standard-conformant decoder shall be equipped. Encountering video contents exceeding a decoder’s level specification is not rare in reality due to the recent ubiquity of various communication networks and the coexistence of portable devices with wide variability in their computation capacity. That is because video content available on mobile networks is accessible basically to every sort of devices, and some content may have a higher level specification of H.264/AVC than that of a receiving decoder. In such cases, currently a conventional decoder just refuses to decode. However, it would be much friendlier to users if the decoder were able to show the decoded pictures at a slightly lower quality that its available resources could provide with the best efforts. In order to do this, a video decoder should be able to flexibly decode the bitstream exceeding the level specification according to its available computing resources. This kind of simplified decoding capability is also quite essential when a decoder knows a priori that fully compliant decoding is not necessary, e.g., in fast-forwarding of video, making thumbnails, or skimming through a video playback. As a whole, the issue in this problem is complexity scalable video decoding in accordance with the available resources in a decoder.

The aforementioned video playback capability itself is already implementable in a sense if the play-back quality is not concerned much. However, such a less careful approach would be practically useless due to the significant degradation of both objective and subjective quality resulting from the processing mismatch between the encoder and the decoder. A quality distortion caused by a complexity-reduced decoding process is propagated to subsequent pictures, and pretty soon, the quality degradation will become unbearable. Therefore, it is very important to carefully design a complexity scalable decoding algorithm which can manage optimal complexity control depending on the resource availability of the device.

Note that various complexity scalable video decoding algorithms have been already developed.^1–19 A major approach to complexity scalable decoding is to control the computational complexity of one or two decoding processes. Peng¹ proposed a discrete cosine transform (DCT)-based complexity scalable video decoder which controlled the decoding complexity by pruning out some DCT data in order to skip the inverse discrete cosine transform (IDCT) process. Peng and Zhong² proposed a selective B-residual decoding method based on computational resources and the energy level of B-residual blocks. Chen et al.³ realized the complexity scalability by using both IDCT pruning and a simpler interpolation filter based on the frame type. Lei et al.⁴ proposed a complexity scalable algorithm for the audio and video coding standard in China (AVS) by using a loop filter along with a luminance interpolation filter scaling method. In this approach, the encoder sends complexity information about the loop filter and interpolation filter to the decoder for the complexity control. Meanwhile, Ji et al.⁵ developed an energy scalable video decoding strategy for multimedia devices. Lee et al. ⁶ also worked towards complexity scalability by controlling the complexity of motion compensation and of the deblocking filter. Mahjoub et al.⁷ proposed a complexity reduction method of the deblocking filter, and Lu et al.⁸ optimized the context adaptive variable length coding (CAVLC) lookup tables to reduce the decoding complexity. de Oliveira et al.⁹ optimized the inverse transform (IT) matrix for each frame, as a function of both content and quantization noise.

Several approaches also designed the complexity model of decoder for complexity scalable video decoding.^10–13 They include complexity models of H.264/AVC decoding parts such as motion compensation,^10,11 entropy decoding,¹² or the whole H.264/AVC decoder. ¹³

As another approach to the complexity scalable algorithms, Park et al.¹⁴ reduced the energy consumption of the MPEG-4 decoding process by using a re-quantization process that reduced the amount of data to process. Nam et al.¹⁵ proposed a method using spatial downsizing decoding.

Some other approaches addressed the reduction of decoding complexity from the viewpoint of hardware architecture design.^16–19 Chao et al.¹⁶ designed an optimized IT architecture to support multistandard video coding applications, and Wei et al.¹⁷ proposed parallel decoding algorithms for multicore processors. Tsai et al.¹⁸ designed a parallel level decoding method of CAVLC, and Sze and Chandrakasan¹⁹ proposed a parallel context adaptive binary arithmetic coding (CABAC) decoding method. However, these hardware perspective approaches are not easy for achieving flexible control because of their hardwired characteristics.

In this paper, an algorithmic complexity scalable video decoding scheme for the H.264/AVC decoder of portable multimedia devices is investigated. First, each decoding element of H.264/AVC was studied in terms of both its complexity versus degradation performance from the viewpoint of complexity scalability and its complexity control parameters. By adjusting the complexity control parameters in a complexity-distortion (C-D) optimized way, the optimum complexity control levels which give the best performance in complexity reduction with minimal quality degradation were found.

The rest of this paper is structured as follows. Section 2 investigates the video decoding elements from the viewpoint of complexity control and develops a complexity reduction method. Section 3 presents the proposed complexity scalable decoding method of each control element and the optimal complexity control level. Section 4 presents experimental results, and Sec. 5 concludes with discussion and some directions for possible future work.

2.

Complexity Control of Video Decoding Elements

The H.264/AVC decoder is composed of several decoding elements as depicted in Fig. 1: a variable length decoder (VLD), an inverse quantizer (IQ), an IT, motion compensation, intra prediction, reconstruction, and deblocking filter. In previous research,⁶ complexity of the H.264/AVC decoding elements in the main profile was evaluated, and its complexity profiling result⁶ is summarized in Table 1 to show that the most complex decoding element is the motion compensation and variable length decoding (CABAC decoding), and that the second major complex element is the deblocking filter. “Others” in Table 1 represent operating system overheads such as file I/O for reading bitstreams from the file system.²⁰ Based on this profiling result, the motion compensation and deblocking filtering were chosen in this paper as targets for complexity control.

Fig. 1

Block diagram of the H.264/AVC decoding process.

Table 1

Complexity profile result of H.264/AVC decoding elements (Ref. 6).

However, by noting the lossless nature of the entropy decoder, simplification of CABAC is not considered since even small degradation from compromised entropy decoding can result in fatal decoding errors. Since the complexity control range using the motion compensation and deblocking filter were not sufficient, a macroblock (MB) decoding skipping method was further developed to provide more flexibility in the complexity control.

2.1.

Complexity Reduction of the Motion Compensation

In the H.264/AVC motion compensation, motion predicted values at half-pel samples were generated by horizontal or vertical one-dimensional 6-tap finite impulse response (FIR) interpolation filtering, while quarter-pel samples were generated by averaging the nearest half-pel and integer-pel samples.²¹ Figure 2 depicts quarter-pel samples for luma components. Since the interpolation process accounts for most of the complexity of motion compensation, its complexity for the luma component is modeled as

Eq. (1)

$$C_{\mathrm{mc}\_\mathrm{luma}}=C_{\mathrm{Int}\_\mathrm{pel}}+C_{\text{Half}\_\mathrm{pel}}+C_{\text{Quarter}\_\mathrm{pel}},$$

where $C_{\mathrm{mc}\_\mathrm{luma}}$ is the total computational complexity of the motion compensation in a decoder; $C_{\mathrm{Int}\_\mathrm{pel}}$, $C_{\text{Half}\_\mathrm{pel}}$, and $C_{\text{Quarter}\_\mathrm{pel}}$ are the computational complexity for generating integer-pel samples (i.e., for sample copy from its reference picture), 6-tap FIR interpolation filtering to generate half-pel samples, and an additional averaging process of quarter-pel samples after generating half-pel samples, respectively. Table 2 shows the interpolation filtering process based on the quarter-pel sample positions in Fig. 2 and their normalized complexity with respect to $C_{\mathrm{Int}\_\mathrm{pel}}$.⁶ Note the nonidentical computational complexity: it depends on the sample positions. Quarter-pel positions at $f$, $i$, $k$, $q$ are the most complex since they require seven 6-tap filtering and one 2-tap filtering. On the other hand, those half-pel samples labeled $b$, $h$ are the least complex ones — they require just one time of 6-tap filtering. This observation suggests that the motion compensation complexity depends critically on the complexity of the interpolation filter. Therefore, to reduce the complexity, the 6-tap interpolation filter can be replaced by 2-tap or 4-tap filter, depending on the subpel position, with filter coefficients as shown in Table 3. Note that the filter coefficients of the 4-tap filter are used in a scalable extension of H.264/AVC²² for inter-layer intra prediction, and those of the 2-tap filter are generated by simplifying the H.264/AVC interpolation filter with adjacent int-pel samples.⁶

Fig. 2

Fractional sample positions for quarter sample luma interpolation.

Table 2

Complexity comparison of interpolation in H.264/AVC decoding.

Sample position	Interpolation operation	Normalized Complexity w.r.t. G(0.0)
G(0,0)	Integer (pixel copy)	1.00
b(0.5,0)	6-tap	2.96
h(0.5,0)	6-tap	2.87
a(0.25,0)	$6\text{-}\mathrm{tap}+2\text{-}\mathrm{tap}$	3.64
c(0.75,0)	$6\text{-}\mathrm{tap}+2\text{-}\mathrm{tap}$	3.50
d(0,0.25)	$6\text{-}\mathrm{tap}+2\text{-}\mathrm{tap}$	3.49
n(0,0.75)	$6\text{-}\mathrm{tap}+2\text{-}\mathrm{tap}$	3.60
e(0.25,0.25)	$(6\text{-}\mathrm{tap})\times 2+2\text{-}\mathrm{tap}$	5.59
g(0.25,0.75)	$(6\text{-}\mathrm{tap})\times 2+2\text{-}\mathrm{tap}$	5.70
p(0.75,0.25)	$(6\text{-}\mathrm{tap})\times 2+2\text{-}\mathrm{tap}$	5.64
r(0.75,0.75)	$(6\text{-}\mathrm{tap})\times 2+2\text{-}\mathrm{tap}$	5.82
j(0.5,0.5)	$(6\text{-}\mathrm{tap})\times 6+6\text{-}\mathrm{tap}$	7.19
f(0.5,0.25)	$(6\text{-}\mathrm{tap})\times 6+6\text{-}\mathrm{tap}+2\text{-}\mathrm{tap}$	7.79
i(0.5,0.75)	$(6\text{-}\mathrm{tap})\times 6+6\text{-}\mathrm{tap}+2\text{-}\mathrm{tap}$	7.28
k(0.25,0.5)	$(6\text{-}\mathrm{tap})\times 6+6\text{-}\mathrm{tap}+2\text{-}\mathrm{tap}$	7.11
q(0.75,0.5)	$(6\text{-}\mathrm{tap})\times 6+6\text{-}\mathrm{tap}+2\text{-}\mathrm{tap}$	7.78

Table 3

Complexity reduction method for interpolation filtering.

position	1-D filter coefficient
	2-tap	4-tap
$1/4$	$[48,16]/64$	$[-3,51,19,-3]/64$
$1/2$	$[32,32]/64$	$[-3,19,19,-3]/64$
$3/4$	$[16,48]/64$	$[-3,19,51,-3]/64$

Prediction values for chroma sample positions are generated by bilinear interpolation of four neighboring integer samples using

Eq. (2)

$$a=((8-x{\mathrm{Frac}}_c)\times (8-y{\mathrm{Frac}}_c)\times A+x{\mathrm{Frac}}_c\times (8-y{\mathrm{Frac}}_c)\times B+(8-x{\mathrm{Frac}}_c)\times y{\mathrm{Frac}}_c\times C+x{\mathrm{Frac}}_c\times y{\mathrm{Frac}}_c\times D+32)/64,$$

where $a$ is a predicted chroma sample value and $A,B,C,D$ are the integer-pel samples. $x{\mathrm{Frac}}_c$ and $y{\mathrm{Frac}}_c$ are the fractional offsets of the predicted sample. To reduce the complexity for chroma interpolation filtering, the predicted chroma sample is just copied from the nearest neighboring integer samples.

2.2.

Complexity Reduction of the Deblocking Filter

Since H.264/AVC performs block-based transform and lossy quantization of integer DCT coefficients, blocking artifacts occur in the reconstructed picture. To eliminate the blocking artifacts, a deblocking filter is applied both in the encoder and in the decoder. This deblocking filter has two processes: block boundary strength (Bs) decision and actual pixel filtering using the determined Bs. At each $4\times 4$ block boundary, Bs is determined as one integer between 0 and 4 according to the rules²³ based on whether or not it is a MB boundary, whether its blocks have intra/inter prediction mode, whether it has nonzero DCT coefficients, its motion vector, its reference picture index, etc. Subsequently, the actual filtering process is applied to each $4\times 4$ block boundary depending on the selected value of Bs. In case of $\mathrm{Bs}=4$, a special filter is applied according to the specific condition.²³ When Bs is from 1 to 3, a normal filter is applied.²³ Therefore, the computational complexity of the deblocking filter is modeled as

Eq. (3)

$$C_{\text{deblocking}\_\text{filter}}=C_{\mathrm{Bs}\_\text{decision}}+C_{\text{filtering}},$$

where $C_{\text{deblocking}\_\text{filter}}$ is the computational complexity of the deblocking filtering, $C_{\mathrm{Bs}\_\text{decision}}$ is the computational complexity of the Bs decision process, and $C_{\text{filtering}}$ is the computational complexity of the actual pixel filtering process. The complexity reduction of deblocking can be designed in two ways — by reducing $C_{\mathrm{Bs}\_\text{decision}}$ or $C_{\text{filtering}}$.

In order to reduce $C_{\mathrm{Bs}\_\text{decision}}$, a simplified Bs decision process in the previous research⁶ was proposed based on some observations.^6,24 The proposed simplified Bs decision process contained the same Bs decision rules as the previous research,⁶ but different Bs values according to the rules as depicted in Fig. 3.

Fig. 3

Proposed simplified block boundary strength (Bs) decision.

A slightly different filtering method compared to the previous research⁶ was also designed by adjusting filtering tap size and the number of samples to be filtered according to Bs to reduce $C_{\text{filtering}}$. When $\mathrm{Bs}\ge 3$, the proposed simplified filtering method is the same as that of H.264/AVC. On the other hand, if $\mathrm{Bs}=2$, only one nearest pixel from each side of the block boundary (i.e., $p_0$ and $q_0$) is filtered²³ — that is, unlike the H.264/AVC, the second immediate pixels from the block boundaries ($p_1$ and $q_1$) are not filtered.²³ In the $\mathrm{Bs}=1$ case, the filtering complexity is reduced by using a 2-tap FIR filter which is applied only to the $p_0$ and $q_0$ samples.

3.

Proposed Complexity Scalable Decoding Scheme

The complexity scalable video decoding scheme should satisfy the following optimality criterion

3.4.

Proposed Complexity Scalable Decoding Scheme

The previous subsections discussed how to realize decoding complexity scalability individually for each key decoding element. In this subsection, the same problem is discussed but focusing specifically on the best scalable control of total video decoding by optimally adjusting the three control parameters ${\mathrm{MCR}}_{\text{Level}}$, ${\mathrm{DFR}}_{\text{Level}}$, and ${\mathrm{MDR}}_{\text{Level}}$ together. A block diagram of the proposed complexity scalable video decoder is shown in Fig. 4.

Fig. 4

Block diagram of the proposed complexity scalable H.264/AVC decoder.

Let us define a total complexity reduction level (${\mathrm{TCR}}_{\text{Level}}$) as a function of the three levels,

Eq. (6)

$${\mathrm{TCR}}_{\text{Level}}=f({\mathrm{MCR}}_{\text{Level}},{\mathrm{DFR}}_{\text{Level}},{\mathrm{MDR}}_{\text{Level}}),$$

where $f(·)$ is a complexity control function with all three individual control levels as input. Note that each level has different effectiveness in complexity control and that consequentially the degradation varies in quality. To find the optimal combination of the control parameters to maximize the total complexity reduction subject to a constraint on the minimum quality degradation, the C-D performance^6,32 was compared by using $\mathrm{AST}$ and $\mathrm{\Delta }\mathrm{PSNR}$ which are respectively defined as

Eq. (7)

$$\mathrm{AST}(\%)=\frac{\mathrm{DT}(\text{Anchor})-\mathrm{DT}(\text{proposed})}{\mathrm{DT}(\text{Anchor})}\times 100,$$

Eq. (8)

$$\mathrm{\Delta }\mathrm{PSNR}[\mathrm{dB}]=\mathrm{PSNR}(\text{proposed})-\mathrm{PSNR}(\text{Anchor}),$$

where $\mathrm{AST}(\%)$ is an average saving time and $\mathrm{DT}(·)$ is a total decoding time of a decoder. $\mathrm{\Delta }\mathrm{PSNR}$ [dB]is the difference in objective quality as measured in the peak signal-to-noise ratio (PSNR) between the proposed method and the anchor. Here the “anchor” represents the conventional H.264/AVC decoder which corresponds to the maximum complexity (i.e., ${\mathrm{TCR}}_{\text{Level}}={\mathrm{MCR}}_{\text{Level}}={\mathrm{DFR}}_{\text{Level}}={\mathrm{MDR}}_{\text{Level}}=0$), and the “proposed” represents the complexity-reduced decoding scheme according to the selected control level.

The C-D curves are shown in Fig. 5; each curve represents the complexity reduction versus quality degradation obtained by controlling the individual level parameter independently (${\mathrm{MCR}}_{\text{Level}}$ in Table 4, ${\mathrm{DFR}}_{\text{Level}}$ in Table 5, and ${\mathrm{MDR}}_{\text{Level}}$ in Table 6, respectively). By controlling each parameter, the decoder complexity is reduced by up to 13.6% by ${\mathrm{MCR}}_{\text{Level}}$, 12.9% by ${\mathrm{DFR}}_{\text{Level}}$, and 25.9% by ${\mathrm{MDR}}_{\text{Level}}$. The quality degradations compared to the maximum complexity reduction are $-1.66$, $-1.84$, and $-3.19\mathrm{dB}$, respectively. As in Fig. 5, it was verified that these three parameters were suitable for complexity control of a decoder.

Fig. 5

Complexity-distortion curve of each individual control level.

The joint C-D curve in terms of all three control parameters together (${\mathrm{MCR}}_{\text{Level}}$, ${\mathrm{DFR}}_{\text{Level}}$, and ${\mathrm{MDR}}_{\text{Level}}$) is shown in Fig. 6. The optimal complexity control points which have the maximum complexity reduction subject to minimum distortion are specified by a line in Fig. 6. Therefore, if a decoder adjusts the control parameters following the line, the optimal decoding complexity scalability can be attained. Table 7 shows the control level of each parameter according to ${\mathrm{TCR}}_{\text{Level}}$, while its relative complexity reduction and quality degradation are shown in Fig. 6. As shown in Table 7, the expected maximum complexity reduction was up to 41% compared with ${\mathrm{TCR}}_{\text{Level}}=0$. Since the choice of ${\mathrm{TCR}}_{\text{Level}}=15$ in Table 7, compared to ${\mathrm{TCR}}_{\text{Level}}=14$, reduced the complexity by not more than 1% but incurred additional quality degradation of about 0.6 dB, the maximum complexity reduction level was set to 14 instead of 15. Therefore, the complexity of a decoder was adjusted in up to 15 steps. When ${\mathrm{TCR}}_{\text{Level}}$ was 0, the decoder has no complexity reduction, that is, the same complexity as conventional H.264/AVC decoder. As the ${\mathrm{TCR}}_{\text{Level}}$ increased, the complexity control level of each parameter was adjusted according to Table 7.

Fig. 6

Joint complexity-distortion curve.

Table 7

Joint complexity reduction level.

${\mathrm{TCR}}_{\text{Level}}$	${\mathrm{MDR}}_{\text{Level}}$	${\mathrm{MCR}}_{\text{Level}}$	${\mathrm{DFR}}_{\text{Level}}$	$\mathrm{\Delta }\mathrm{PSNR}$ [dB]	AST(%)
0	0	0	0	0.00	0.00
1	0	1	0	0.00	7.29
2	0	2	0	$-0.13$	13.17
3	0	2	1	$-0.71$	14.34
4	0	3	2	$-1.98$	20.45
5	0	2	4	$-1.03$	21.64
6	0	2	5	$-1.82$	25.29
7	0	3	5	$-2.77$	26.44
8	1	1	2	$-3.32$	31.43
9	1	2	3	$-3.33$	32.45
10	1	2	5	$-3.47$	33.76
11	2	1	3	$-3.72$	34.78
12	2	1	4	$-3.86$	36.33
13	1	1	5	$-4.07$	37.52
14	2	2	5	$-4.45$	40.47
15	2	3	5	$-5.06$	41.33

4.

Experimental Results

In order to assess the performance of the proposed scheme, it was implemented on JM18.0 H.264/AVC reference software. Bitstreams for experiments were coded for the H.264/AVC main profile with group of pictures (GOP) $\text{size}=60$ under IBPBP structure (Here, I, B, and P represents Intra, Bi-predictive, and Predictive picture, respectively). The number of reference frames was 5, and one picture was coded as one slice. The quantization parameter (QP) was set to 22, 27, 32, and 37. The video sequences used for performance evaluation were Bigships, City_corr, Night, and Crew ($1280\times 720@60\mathrm{fps}$). The performance of the proposed scheme was measured by AST (%) in Eq. (7) and $\mathrm{\Delta }\mathrm{PSNR}$ [dB] in Eq. (8).

Figure 7 shows how the subjective quality changes when each control parameter of complexity (${\mathrm{MCR}}_{\text{Level}}$, ${\mathrm{DFR}}_{\text{Level}}$, and ${\mathrm{MDR}}_{\text{Level}}$) is individually adjusted. Figure 7(a) corresponds to the maximum complexity level (${\mathrm{TCR}}_{\text{Level}}=0$, i.e., ${\mathrm{MCR}}_{\text{Level}}=0$, ${\mathrm{DFR}}_{\text{Level}}=0$, and ${\mathrm{MDR}}_{\text{Level}}=0$) while the others, Fig. 7(b)–7(d), respectively, correspond to when each control parameter is changed to its maximum reduction value individually [Fig. 7(b): ${\mathrm{MCR}}_{\text{Level}}=3$; Fig. 7(c): ${\mathrm{DFR}}_{\text{Level}}=5$; Fig. 7(d): ${\mathrm{MDR}}_{\text{Level}}=2$]. Figure 7 shows that while the maximum complexity reduction of deblocking filtering (${\mathrm{DFR}}_{\text{Level}}=5$) in Fig. 7(c) gives similar subjective quality as the no complexity reduction case in Fig. 7(a), the sharpness of the picture is degraded when ${\mathrm{MCR}}_{\text{Level}}$ or ${\mathrm{MDR}}_{\text{Level}}$ are adjusted to those of the minimum complexity level (${\mathrm{MCR}}_{\text{Level}}=3$, ${\mathrm{MDR}}_{\text{Level}}=2$). The sharpness degradation appears in all regions in Fig. 7(b), but it is less apparent in the low motion area in Fig. 7(d) (see the middle building in the picture). As depicted in Fig. 7, although control parameters had some degradation in the minimum complexity level, they still maintained proper overall subjective quality.

Fig. 7

Subjective quality comparison between the maximum complexity level and the minimum complexity level of control parameters (city sequence, 118th frame, QP22) (a) ${\mathrm{TCR}}_{\text{Level}}=0$ (maximum complexity), (b) ${\mathrm{MCR}}_{\text{Level}}=3$, (c) ${\mathrm{DFR}}_{\text{Level}}=5$, (d) ${\mathrm{MDR}}_{\text{Level}}=2$.

Table 8 shows experimental results of the proposed scheme, indicating that it attains complexity scalability. The proposed scheme reduces the decoding complexity by up to 44% (in City_corr sequence, ${\mathrm{TCR}}_{\text{Level}}=14$) as compared with the conventional H.264/AVC decoder. In Table 9, $C_{\text{target}}$ represents the target complexity reduction and ${\mathrm{TCR}}_{\text{Level}}$ is the complexity control level to achieve the complexity reduction up to $C_{\text{target}}$. Reduced complexity for the ${\mathrm{TCR}}_{\text{Level}}$ is similar to the $C_{\text{target}}$ in most cases. However, the reduced complexity does not reach the target complexity $C_{\text{target}}$ in the Crew sequence. This is because the proposed scheme does not consider an intra-coded MB as a candidate for complexity control, therefore, there may exist a problem of insufficiency in the adjustable range of complexity especially if there are many intra MBs. The many intra-coded MBs (boxed areas) in the Crew sequence are shown in Fig. 8 in an inter slice. This behavior is due to flash lights). This is one area for future extension of the proposed method.

Table 8

Experimental results of the proposed scheme.

Table 9

Performance comparison of the proposed scheme with previous methods (Refs. 4, 6).

Fig. 8

Intra coded macroblock (shown in box) in an inter slice.

Table 9 shows the distortion and complexity reduction performance of the proposed method compared to the previous research.^4,6 For a fair comparison, previous research^4,6 was implemented on the JM18.0 H.264/AVC reference software. In Table 9, methods 1 and 2, respectively represent methods of research from Refs. 4 and 6. The reduced complexity according to the $C_{\text{target}}$ and the relative distortion compared with the conventional H.264/AVC decoder were measured. Since method 2 reduces the complexity by up to 25% and method 1 exploits the $B$ picture decoding skip method to reduce the complexity by more than 30%, $C_{\text{target}}$ was only considered for 10% and 20% in this experiment. Compared to methods 1 and 2, the proposed method is better in objective quality loss. Since those methods reduce the complexity of the interpolation filtering process regardless of picture type — that is, through the $P$ picture or stored-$B$ picture which are used as references of following pictures — error will be propagated to the following pictures and the objective quality loss was much higher than the proposed method. Complexity control of the proposed method was also more accurate than methods 1 and 2, and the reduced complexity of proposed method was more similar to $C_{\text{target}}$ than these two methods.

Figure 9 shows a subjective quality comparison according to the ${\mathrm{TCR}}_{\text{Level}}$ represented in Table 8 from the 118th (the last decoded frame in one GOP) frame in City_corr sequence coded with QP 22 (high quality). Even if the ${\mathrm{TCR}}_{\text{Level}}$ increases, the decoded pictures still have acceptable subjective quality, although the objective quality degradation becomes larger.

Fig. 9

Comparison of subjective quality of decoded frames (city sequence, 118th frame, QP22) (a) ${\mathrm{TCR}}_{\text{Level}}=0$ (maximum complexity), (b) ${\mathrm{TCR}}_{\text{Level}}=2$, (c) ${\mathrm{TCR}}_{\text{Level}}=4$, (d) ${\mathrm{TCR}}_{\text{Level}}=6$, (e) ${\mathrm{TCR}}_{\text{Level}}=11$, (f) ${\mathrm{TCR}}_{\text{Level}}=14$ (minimum complexity).

To verify the proposed scheme on mobile devices, it was also implemented on a mobile device. Figure 10 shows subjective quality with the maximum complexity (${\mathrm{TCR}}_{\text{Level}}=0$) and the minimum complexity (${\mathrm{TCR}}_{\text{Level}}=14$) on a mobile device. Since the mobile device is not able to show the original resolution of the video sequence, it only displays a downsized picture after performing a downsizing process. As shown in Fig. 10, the subjective quality of the decoded picture compared to the one of maximum complexity was acceptable even with the minimum complexity.

Fig. 10

Comparison of subjective quality on mobile device. (a) ${\mathrm{TCR}}_{\text{Level}}=0$ (maximum complexity) (b) ${\mathrm{TCR}}_{\text{Level}}=14$ (minimum complexity).

5.

Conclusion

This paper presented a complexity scalable H.264/AVC decoding scheme for portable multimedia devices. The proposed method controls the motion compensation, deblocking filtering, and MB decoding skipping process by adjusting these three complexity control parameters. Its C-D performance was evaluated according to the controlling parameters, and the optimal complexity control levels for each parameter were sought. The proposed scheme can control the decoding complexity with variable complexity control levels without significant subjective quality loss. Since the current scheme can adjust the decoding complexity of inter MBs only, future work may extend it to include also the decoding complexity controlling capability of intra MBs.