Super high resolution video codec system with multiple ...

Viewer
Transcript

SUPER HIGH RESOLUTION VIDEO CODEC SYSTEM WITH MULTIPLE MPEG-2 HDTV CODEC LSI’S Ken Nakamura, Takeshi Yoshitome, Yoshiyuki Yashima NTT Cyber Space Laboratories, NTT Corporation 1-1 Hikarinooka Yokosuka-city Kanagawa 239-0847 Japan 3840 pixel

1. INTRODUCTION Interest in Super High Resolution (SHR) video systems has grown in the last few years. SHR video has approximately double the height and width of conventional HDTV and has a 24 - 60 fps frame rate. Because of the high quality and the sense of reality they convey to the viewer, SHR systems are expected to be platforms for many new video communication applications, like digital cinema, virtual museums and public viewing of sports, concert and other events[1]. For SHR communication applications, it will be very important to reduce network bandwidth, because raw SHR video data is sent at 6 Gbit/s - 12 Gbit/s. Thus a CODEC system is indispensable for realization of these applications. We have already developed a coding and transmission system for SHR or over-HDTV video, which consists of several conventional MPEG-2 HDTV CODEC equipments and a new multi-channel frame synchronizer [2]. This system was economical, but it was also large and complex. The present report deals with a new SHR CODEC system that uses multiple MPEG-2 HDTV CODEC LSI’s. We have already developed MPEG-2 HDTV encoder system

;,(((

1 1 stream

2 ....

3

4

1

2

3

4

1

....

picture data

4

i) one stream output system 3840 pixel 2160 pixel

This super high resolution (SHR) video CODEC system is based on a spatial image division and multiple stream output approach and is implemented with multiple MPEG2 HDTV CODEC LSI’s. The encoder has two operation modes, a non-multiplexing mode, and a multiplexing mode. In the non-multiplexing mode, the encoder outputs four constant bit rate transport streams (CBR-TS’s) for which it can use two channel-synchronization schemes, system time clock (STC) sharing and sync-marker. In the multiplexing mode, the encoder outputs one CBR-TS comprised of four variable bit rate elementary streams (VBR-ES’s), where cooperative operation between the pre-encoder and main-encoder realize adaptive bit allocation between four channels. The encoder also has a video shift and padding function which reduces distortions caused by the image division. These functions enable high-efﬁciency, high-quality SHR video coding.

2160 pixel

ABSTRACT

1

3

2

4

4 streams ....

1

1

1

....

....

2

2

2

....

....

3

3

3

....

....

4

4

4

....

picture data

ii) multiple stream output system

Fig. 1. Spatial image division approach.

that uses multiple MPEG-2 SDTV encoder LSI’s [3][4], and have developed the world’s ﬁrst MPEG-2 422P@HL CODEC LSI’s [5][6]. This time, we have constructed the SHR CODEC system with them. This system is small and energy efﬁcient, yet enables high efﬁciency coding. In this paper, ﬁrst, we describe the basic idea of our CODEC system, which is spatial image division and multiple stream output. Next, we explain the video shift and padding function, which solves the boundary distortion problem caused by spatial image division. Then we show how to synchronize the multiple streams in non-multiplexing mode and explain the strategy of the adaptive bit allocation in multiplexing mode. Finally, we discuss the system’s implementation. 2. ARCHITECTURE OF SHR CODEC SYSTEM This CODEC system adopts the spatial image division and multi-stream output approach. In spatial image division, the input image is divided into multiple sub-images and the encoder modules encode them in parallel. This approach is reasonable in terms of cost performance and extensibility. Spatial image division can use a one-stream output system, in which the sub-streams generated by the encoder modules are reconstructed into one SHR elementary stream (ES), or it can use a multiple-stream output system, where

,,,

,6&$6

Encoder

Decoder

1st/2nd Field

1

ENC1 1

2

ENC2

3

4

ENC3 ENC4

DEC1

line copy

2 DEC2

1

2

DEC3

3

4

3

DIST SRC1

4

4 CBR-TS

DEC4

active line

i) non-multiplexing mode

PRE

Encoder MAIN

ENC1

ENC1

Decoder

1

2

ENC2

ENC2

1 2 3

3

4

ENC3

ENC3

4

ENC4

ENC4

coding information.

SRC2

DEC1 DEC2

1

2

DEC3

3

4

line copy

Fig. 3. Video shift and padding function.

DEC4 1 CBR-TS (4 VBR-ES) ii)multiplexing mode

Fig. 2. Overview of the SHR CODEC system.

several HDTV streams generated by the encoder modules are output directly in parallel or multiplexed into one transport stream (TS), as shown in Fig.1. For SHR communication applications, we use the multiplestream output system, because the conventional HDTV decoders can decode its output stream, whereas the one-stream SHR output system needs dedicated SHR decoders. Fig.2 depicts the overview of our CODEC system. The CODEC has two output modes, multiplexing mode and nonmultiplexing mode. In non-multiplexing mode, encoder modules output constant bit rate (CBR) TS’s which are transmitted in parallel. In multiplexing mode, pre-encoder analyzes the input images and each encoder module in main-encoder outputs a variable bit rate (VBR) ES, and these are multiplexed into one CBR-TS. The multiplexing mode has the advantages of transmitter availability and coding efﬁciency. The non-multiplexing mode is needed when the bitrate of each stream is too high for the bandwidth of the network channel. 3. VIDEO SHIFT AND PADDING FUNCTION In spatial image division, there is a problem that the coding distortion caused by the mismatch of the image boundary and DCT block boundary might be visible at the boundary of the divided images under certain coding conditions. Generally, SHR or over-HDTV image camera systems output divided images by using conventional HDTV video signal interfaces, like HD-SDI’s. In some camera systems, the output image data does not ﬁll the active line, e.g., 1920 1032 image data on 1920 1080, and the remaining lines are ﬁlled with black data or zero data. Because the MPEG-2 encoders transform the 8 8 image block by using a DCT and cut off the high-frequency components, if the edge of the image and the DCT block boundary do not match, the border will not be sharp and the image data will be eroded

by black data. Thus, the coding distortion will be visible at the boundary of the divided images. This problem also occurs at the border of ordinary coded images, but it is more visible in this kind of system because the boundaries of the divided images are positioned at the center of the image. In our encoder system, the video shift and padding function modules are placed before the encoder modules. This module can copy the top and bottom lines of image data onto the non-image data area, as shown in Fig. 3. The decoder system does nothing in regard to this operation because the copied data are aborted later. This module also can shift all the image data vertically so that the image border matches the DCT block boundary. This function can provide better quality than the line copy function can, although the image must be shifted back to its original position at the projector system. With this function, the distortion at the divided images’ boundary can be almost completely removed. 4. SYNCHRONIZATION STRATEGY IN NON-MULTIPLEXING MODE In non-multiplexing mode, the encoder outputs 4 CBR-TS’s. When the multiple streams are transmitted on different channels in parallel, the synchronization between channels is a very important issue. This CODEC system has two scheme for synchronization between channels. One method is that the encoder modules share a common STC (System Time Clock), as shown in Fig. 4. On the encoder side, a 27-MHz system clock and STC value are generated from the input video signal at the master encoder module and distributed to the other encoder modules because the input video signals are synchronized. Each encoder generates and outputs a PCR (Program Clock Reference) and PTS/DTS (Presenting Time Stamp / Decoding Time Stamp) from the 27-MHz system clock and STC value. On the decoder side, the master decoder module generates a 27-MHz system clock and STC from the received PCR’s and distributes the system clock and STC value to the other decoder modules. In order to absorb the disparity in the receiving timing between channels and to avoid under-

,,,

VCLK

Pre-Encoder

PLL STC_VALUE Video

VCXO

CLK27MHz

TS

Decoder Module

TS

Encoder Module

TS

Decoder Module

TS

Encoder Module

TS

Decoder Module

TS

Video

Video

Proce ssor

Proce ssor

CODEC LSI

CODEC LSI

CODEC LSI

bitrate, target_bit, Q

CLK27MHz

CLK27MHz Encoder Module

CODEC LSI

Video

Proce ssor

CLK27MHz

CLK27MHz

Video

target_bit, picture_biit, Q

Video

CLK27MHz

CLK27MHz

Video

CLK27MHz

STC_VALUE Encoder Module

Proce ssor

TS

Decoder Module

TS

Proce ssor

Proce ssor

Proce ssor

Proce ssor

CODEC LSI

CODEC LSI

CODEC LSI

CODEC LSI

Video

Fig. 4. Synchronization by STC sharing.

Main Encoder

1

2

Fig. 6. Adaptive bit allocation using pre-encoder. 3

1 bit3

4

0 bit2

1 bit1

main-encoder encodes the input image at a variable bitrate, which is given by the control processor. The bitrates of the encoder modules are changed simultaneously so that the bitrate of the multiplexed TS will be constant. These output streams are multiplexed into one CBR-TS with the cascade multiplexing function in the CODEC LSI’s.

0 bit0

Fig. 5. Sync marker. 6. IMPLEMENTATION ﬂow or overﬂow of the buffer, the encoder system generates the PTS/DTS with a large enough timing margin. The other synchronization scheme is the sync marker scheme, which is useful mainly with multiple conventional HDTV decoders. Many conventional HDTV decoders do not have an STC sharing scheme and can’t deal with ancillary data like time code information. Our encoder put a sync marker on the top or bottom of the active line of each channel of the image, as shown in Fig. 5. On the receiving side, the decoded images are synchronized with the multichannel frame synchronizer that we have developed [2]. 5. BIT ALLOCATION STRATEGY IN MULTIPLEXING-MODE In multiplexing mode, four VBR-ES’s are multiplexed into one CBR-TS. For higher efﬁciency coding, an adaptive bit allocation between channels is needed. Our approach is to use a two-pass bit allocation system, as shown in Fig. 6. A pre-encoder encodes and analyzes the input images, and a control processor calculates the bitrate of each channel so as to make the image quality constant between channels. The

The SHR encoder system is comprised of the pre-encoder unit and the main encoder unit, and these units are realized with the same hardware; only the ﬁrmwares are different. Fig. 7 is a block diagram of the encoder unit hardware. Four synchronized video signals are input through the HD-SDI to the video buffers. These video buffers can delay the signals up to 20 frames. At the pre-encoder unit, the delay is 0 frames and at the main-encoder unit, it is about 15-20 frames for utilizing the pre-encoding information. This video buffer also has the video shift and padding function and can put the sync marker onto the input images. The video signals are encoded by the corresponding encoder LSI’s. In the multiplexing mode, the main-encoder module multiplexes the streams into one TS and outputs it. The encoder unit hardware is composed of four channel modules, each of which has a video buffer and CODEC LSI, the CODEC’s external memories, host processor, and other communication modules. The channel modules are connected to each other; serial cables are used for the high speed communication between the CODEC LSI’s , which is the cooperative coding control and the cascade bitstream multiplexing, and 100 Base-TX is used for the communica-

,,,

AC

Power Unit

LCD display

Control Processor

HD-SDI

Cable interface

Video Buffer

Table 1. Speciﬁcation of SHR encoder system Communication Module

CODEC LSI

video format Cable interface

DVB-ASI

video encoder output stream

module#1 Control Processor

HD-SDI

Cable interface

Video Buffer

Communication Module

CODEC LSI

Cable interface

maximum bitrate

DVB-ASI

module#2 Control Processor

HD-SDI

Cable interface

Video Buffer

input interface output interface control interface power size(mm)

Communication Module

CODEC LSI

Cable interface

DVB-ASI

module#3 Control Processor

HD-SDI

Cable interface

Video Buffer

Communication Module

CODEC LSI

¢ ¢

1080i (60/59.94i) 4ch, 720p (60/59.94p) 4ch MPEG-2 422P@HL CODEC LSI 4 or 8( ) CBR-TS 4, or CBR-TS 1 (VBR-ES 4)( ) 80Mbps 4, or 160Mbps 1( ) HD-SDI 4ch, AES3id 4 DVB-ASI 1ch or 4ch( ) 100BASE-TX AC-100-240V 460(w) 440(D) 44(H) 1 or 2( )

¢ ¢ ¢ ¢ Ý ¢ ¢

¢

¢

Ý

¢ Ý ¢ Ý

¢

¢

Ý

Ýmultiplexing mode Cable interface

DVB-ASI

module#4 100 Base-TX

a video shift padding function and adaptive bit allocation using a pre-encoder for high-efﬁciency video coding. The hardware of this system comprises several channel modules that are connected to each other by high-speed serial cable and that cooperatively carry out the encoding process. In the future, we will increase the resolution of the video that the CODEC can handle.

Serial Cable

Fig. 7. Block diagram of encoder unit.

8. REFERENCES [1] T.Fujii, ”Super High Deﬁnition Digital Movie System”, SPIE VCIP’99, Vol.3653, pp.1412-1319, 1999. [2] T.Yoshitome, ”A scalable architecture for use in an over-HDTV real-time codec system for multiresolution video”, Visual Communications and Image Processing (VCIP) 2003, pp.1752-1759 Fig. 8. Encoder unit.

tions between the encoder units. The SHR decoder is comprised of four channel HDTV decoders, and it embodies the STC sharing and sync marker synchronization schemes described in Section 4. The decoder hardware unit has a similar conﬁguration to the encoder hardware. These hardwares have extensibility with respect to the number of channels for encoding and decoding. Thus, video CODEC’s with various resolutions are conﬁgurable simply by adding channel modules and changing ﬁrmware. 7. CONCLUSION We developed a new SHR video CODEC system using multiple MPEG-2 HDTV CODEC LSI’s. The system has a non-multiplexing mode and a multiplexing mode, and it has

[3] T.Yoshitome, et al, ”Development of an HDTV MPEG2 encoder based on multiple enhanced SDTV encoding LSIs”, International Conference on Consumer Electronics(ICCE) 2001, pp. 160-161 [4] M.Ikeda, ”An MPEG-2 Video Encoder LSI with Scalability for HDTV based on Three-layer Cooperative Architecture”, Design, Automation and Test in Europe Conference (DATE) 1999, pp.44-50 [5] J.Naganuma, et al, ”Vasa: Single-chip MPEG-2 422P@HL CODEC LSI with Multi-chip Conﬁguration for Large Scale Processing beyond HDTC Level”, HOT Chips 14, August 2002 [6] H.Iwasaki, et al, ”Single-chip MPEG-2 422P@HL CODEC LSI with Multi-chip Conﬁguration for Large Scale Processing beyond HDTV Level”, Design, Automation and Test in Europe Conference (DATE) 2003, Designers’ Forum, March 2003, pp.2-7

,,,

Super-resolution of Video Sequences Using Local ...