Distortion-Complexity Optimization of the H.264/MPEG-4 AVC Encoder using the GBFOS Algorithm Rahul Vanam, Eve Riskin, Sheila Hemami and Richard Ladner University of Washington, Cornell University
Data Compression Conference 2007
1
Problem Raw video Bit rate Encoder Parameters
H.264 Encoder (x264)
Compressed Video at given rate
2
Problem Raw video Bit rate Encoder Parameters Real time application
H.264 Encoder (x264) Fast encoding speed
Compressed Video at given rate
Best quality
3
Problem Raw video Bit rate Encoder Parameters
?
H.264 Encoder (x264) Fast encoding speed
Compressed Video at given rate
Best quality
4
Problem Raw video Bit rate Encoder Parameters
H.264 Encoder (x264)
Compressed Video at given rate
Fast encoding Best quality ? speed Parameters are similar to knobs. As a knob is moved from lower to higher setting: quality improves while speed suffers H.264 Encoder
5
Selecting the Encoder Parameters
Distortion
data point Convex hull
Exhaustive search: – Run the encoder for all possible settings. – Obtain MSE vs. encode time plot – Choose settings on the Distortion-Complexity convex hull – Extremely time consuming!
Complexity
6
Selecting Encoder Parameters • GBFOS-basic and GBFOS-iterative algorithms select good encoder parameter settings with fewer encodings • Based on the GBFOS (Generalized Breiman, Friedman, Olshen and Stone) algorithm • GBFOS Assumption: Distortion and Complexity of parameters are additive/independent (untrue!) 7
Selecting Encoder Parameters • Both algorithms are explained using an example: – Four parameter setting variables: number of reference frames, partition size, sub pixel motion estimation and quantization method – Other settings are constant
8
GBFOS-Basic Algorithm • Obtain D-C plot for each parameter by setting other parameters to their best PSNR option • Choose convex hull points and corresponding parameter settings and slopes
Distortion-complexity plot for Number of Reference frames
9
GBFOS-Basic Algorithm Reference frames MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
16
Time per frame
Subme
Time per frame
Trellis MSE
MSE
7
Time per frame
First parameter setting selected
2
Time per frame 10
GBFOS-Basic Algorithm Reference frames MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
16
Time per frame
Subme
Time per frame
Trellis MSE
MSE
7
Time per frame
2
Compare slopes Find the parameter with the least slope
Time per frame 11
GBFOS-Basic Algorithm Reference frames MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
16
Time per frame
Subme
Time per frame
Trellis MSE
MSE
7
Time per frame
2
Compare slopes Find the parameter with the least slope
Time per frame 12
GBFOS-Basic Algorithm Reference frames frames Reference MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
7
Time per frame
Subme
Time per frame
Update the slope and setting of Ref. frames
Trellis MSE
MSE
7
Time per frame
2
Time per frame 13
GBFOS-Basic Algorithm Reference frames MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
7
Time per frame
Subme
Time per frame
Trellis MSE
MSE
7
Time per frame
2
Update the slope and setting of Ref. frames We now have the second parameter setting
Time per frame 14
GBFOS-Basic Algorithm Reference frames MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
7
Time per frame
Subme
Time per frame
Trellis MSE
MSE
7
Time per frame
Compare the slopes again and find the parameter with least slope
2
Time per frame 15
GBFOS-Basic Algorithm Reference frames MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
7
Time per frame
Subme
Time per frame
Trellis MSE
MSE
4
Time per frame
2
Compare the slopes again and find the parameter with least slope Update the slope and setting of Subme
Time per frame 16
GBFOS-Basic Algorithm Reference frames MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
7
Time per frame
Subme
Time per frame
Trellis MSE
MSE
4
Time per frame
2
Compare the slopes again and find the parameter with least slope Update the slope and setting of Subme
Time per frame 17
GBFOS-Basic Algorithm Reference frames MSE
1
MSE
Time per frame
Subme MSE
Partition sizes P8x8
Time per frame
Trellis MSE
1
Time per frame
1
The process is repeated to obtain the final parameter setting
Time per frame 18
GBFOS-Iterative Algorithm • More number of encodings • Better performance • Initialization and first iteration same as the GBFOS-Basic Algorithm
19
GBFOS-Iterative Algorithm Reference frames MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
16
Time per frame
Subme
Time per frame
Trellis MSE
MSE
7
Time per frame
First parameter setting selected
2
Time per frame 20
GBFOS-Iterative Algorithm Reference frames MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
16
Time per frame
Subme
Time per frame
Trellis MSE
MSE
7
Time per frame
2
Compare slopes Find the parameter with the least slope
Time per frame 21
GBFOS-Iterative Algorithm Reference frames MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
16
Time per frame
Subme
Time per frame
Trellis MSE
MSE
7
Time per frame
2
Compare slopes Find the parameter with the least slope
Time per frame 22
GBFOS-Iterative Algorithm Reference frames frames Reference MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
16
Time per frame
Subme
Time per frame
Update the slope and setting of Ref. frames
Trellis MSE
MSE
7
Time per frame
2
Time per frame 23
GBFOS-Iterative Algorithm Reference frames frames Reference MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
7
Time per frame
Subme
Time per frame
Update the slope and setting of Ref. frames
Trellis MSE
MSE
7
Time per frame
2
Time per frame 24
GBFOS-Iterative Algorithm Reference frames MSE
Partition sizes
MSE P8x8,P4x4,B8x8, I8x8,I4x4
7
Time per frame
Subme
Time per frame
Trellis MSE
MSE
7
Time per frame
2
Update the slope and setting of Ref. frames We now have the second parameter setting
Time per frame 25
GBFOS-Iterative Algorithm Second Iteration Reference frames MSE
Partition sizes
MSE 7
Time per frame
Subme MSE
Time per frame
Trellis MSE
Time per frame
Regenerate the D-C plots for partition size, subme and trellis using Ref. frames = 7 Other parameters are obtained from the previous step
Time per frame 26
GBFOS-Iterative Algorithm Second Iteration Reference frames MSE
MSE 7
Time per frame
Subme MSE
7
Partition sizes P8x8,P4x4,B8x8, I8x8,I4x4
Time per frame
Trellis MSE
Time per frame
2
Regenerate the D-C plots for partition size, subme and trellis using Ref. frames = 7 Other parameters are obtained from the previous step
Time per frame 27
GBFOS-Iterative Algorithm Second Iteration Reference frames MSE
MSE 7
Time per frame
Subme MSE
7
Partition sizes P8x8,P4x4,B8x8, I8x8,I4x4
Time per frame
Trellis
Find the parameter with the least slope and update its slope and setting
MSE
Time per frame
2
Time per frame 28
GBFOS-Iterative Algorithm Second Iteration Reference frames MSE
MSE P8x8,P4x4,I8x8,I4x4
7
Time per frame
Subme MSE
Partition sizes
7
Time per frame
Trellis
Find the parameter with the least slope and update its slope and setting
MSE
Time per frame
2
Time per frame 29
GBFOS-Iterative Algorithm Second Iteration Reference frames MSE
MSE P8x8,P4x4,I8x8,I4x4
7
Time per frame
Subme MSE
Partition sizes
7
Time per frame
Trellis MSE
Time per frame
2
Find the parameter with the least slope and update its slope and setting We now have the new parameter setting
Time per frame 30
GBFOS-Iterative Algorithm Final Iteration Reference frames MSE
1
MSE
Time per frame
Subme MSE
1
Partition sizes P8x8
Time per frame
Trellis MSE
Time per frame
1
The process is repeated till we obtain the final parameter setting
Time per frame 31
Results • In our test, we use 3 data sets – ASL-1 (10 QCIF files), ASL-2 (10 320x240 files) and Standard (15 QCIF files) and 3 bitrates – 30, 150 and 300 kb/s • Four variable encoding parameters: number of reference frames (16 options), Partition sizes (10 options), Sub-pixel motion estimation (7 options) and trellis (3 options)
32
Results • If n1,...,nM are M parameter options, then the number of encodings per video sequence for M Convex hull: (product) ∏ ni i=1
GBFOS-basic:
1- M+
∑
M n i=1 i
(sum!)
• In our example, the number of encodings for Convex hull = 3360 (constant) GBFOS - basic = 33 (constant) ≈ 1% Convex hull GBFOS - iterative = 268 (maximum) ≈ 8% of Convex hull
33
Results • Maximum PSNR difference with respect to convex hull for: GBFOS-basic = 0.707 dB GBFOS-iterative = 0.575 dB PSNR vs. encoding time for x264 encoder 37.6 37.4
PSNR (dB)
37.2 37 36.8 36.6 36.4
Convex hull GBFOS-iterative GBFOS-basic
36.2 36 35.8 0
0.01
0.02
0.03
0.04
0.05
Average Encoding time per frame (seconds)
34
Results • Robustness test done using 3 different (data set, training set) pairs at 3 different bitrates • Max PSNR difference between either GBFOS algorithms and convex hull points < 0.55 dB 46.2
PSNR (dB)
46 45.8 45.6 45.4 Convex hull of test data GBFOS-iterative GBFOS-basic
45.2 45 0
0.02
0.04
0.06
0.08
0.1
Average Encoding time per frame (seconds)
35
Application • Video cell phones have limited processing speed • Finding convex hull points is infeasible! • GBFOS algorithms can be used for finding closeto-optimal encoder settings • Goal: Real-time communication of American Sign Language videos over the GPRS network
http://www.cs.washington.edu/research/MobileASL/index.html
36
Conclusion • Proposed two fast algorithms: GBFOS-basic and GBFOS-iterative for choosing encoder parameter settings • GBFOS-basic and GBFOS-iterative algorithms take 1% and 8% of encodings required for obtaining the D-C convex hull points • For same test and training data: Max PSNR difference for both algorithms < 0.71 dB • For different test and training data: Max PSNR difference for both algorithms < 0.55 dB 37
Thanks! • To National Science Foundation • To Anna Cavender, Neva Cherniavsky and Loren Merritt
38
