JPH05344491A - Inter-frame prediction coding system - Google Patents

Abstract

(57) [Summary] [Purpose] Judge scene changes and parallel movements with a small amount of calculation. [Structure] The differentiator has four corner macroblocks.
The difference between the previous frame and the previous frame is calculated. If the difference between all four macroblocks is not 0, step 102
If the difference is 0, normal encoding is performed. In step 102, the frame memory & predictor searches for motion vectors for the macroblocks at the four corners, and if motion compensation prediction based on the motion vector is possible in any of the macroblocks, the frame is not bidirectionally predictable. If it is set to an independent frame and motion compensation prediction is not possible, it is determined that the scene has changed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a high efficiency coding system for efficiently coding a moving image signal with a smaller code amount, and more particularly to an interframe predictive coding system.

[0002]

2. Description of the Related Art MPEG1 is used for encoding moving images.
However, the standard method is not specified, but at what intervals independent frames (referred to as I-pictures (Intra coded pictures)) are set.

In general, I pictures are set at regular intervals, and no consideration is given to correlation between frames, rapid change between frames, and magnitude of prediction error. The position of the P picture in the non-independent frame is also not specified, and is set without considering whether backward prediction is necessary.

Further, when determining the motion compensation mode in non-independent frame coding, the motion vector search is performed independently for each macroblock, and even in the case of parallel movement of the entire screen or the scene. In the case of a change, the search method is the same.

As related prior arts of this kind, for example, JP-A-2-285816 and 2-19.
No. 2378, etc. are mentioned.

[0006]

As described above, when the interval between I pictures is constant every n frames, the quality of the reproduced image is large in a section where the image does not change significantly and a section where the image changes significantly. It makes a difference and is very unbalanced in terms of coding efficiency.

Further, for example, when there is a scene change, the frame correlation naturally disappears, and the prediction error of the subsequent non-independent frame (particularly P picture) becomes large, which adversely affects the quality of the reproduced image.

In general, in the case of a scene change, it is better to set it in an independent frame so that the coding efficiency is higher. Further, when the entire screen is moved in parallel, if the frame happens to be a P picture, backward prediction cannot be performed, so that there is no choice but to intra-code a certain area. In such a case, it is better to set the B picture.

However, as a method of determining such a scene change or parallel movement, a method of determining a motion vector, a non-intra / intra ratio, etc. for all macroblocks is adopted. There was a problem that the amount of calculation was huge.

Further, conventionally, motion compensation of non-independent frames /
In the non-motion compensation mode selection, a motion vector search is performed for each macroblock. Therefore, on the translation screen, the motion vector is
Almost the same for all macroblocks. Therefore, the amount of calculation is smaller when the motion vector is fixed and encoded in the motion compensation prediction mode. However, there was no criterion for deciding which macroblock motion vector to refer to.

The present invention has been made in consideration of the above points, and an object of the present invention is to detect macroblocks at any of the four corners as long as the screen is moving on the same scene (background). If there is no change between frames and if there is a large parallel movement, there is a difference from the previous frame at all four corners, and one of these motion vectors matches the motion vector of the central macroblock. Focusing on this, it is an object to provide an inter-frame predictive coding method that discriminates a scene change or a parallel movement with a small amount of calculation, has little deterioration in reproduced image quality, and has high coding efficiency.

Still another object of the present invention is to determine a frame of parallel movement by the method of the present invention, and for a frame determined to be parallel movement, the motion vector search is limited to the minimum necessary macroblocks. An object of the present invention is to provide an interframe predictive coding method with a smaller amount of calculation and high coding efficiency by performing coding by motion compensation prediction for all macroblocks on the basis of a motion vector.

[0013]

In order to achieve the above object, according to the invention of claim 1, independent frames are set at arbitrary intervals from consecutive frames of an input image signal, and the independent frames are set within the frame. In an inter-frame predictive coding method for independently coding and predicting and coding a signal of a non-independent frame between the independent frames based on signals of preceding and following independent frames, four corners or four corners of the non-independent frame For the neighboring blocks, calculate the difference from the previous frame,
When the difference is not zero in any of the blocks, it is characterized in that it is determined that the entire screen is moving in parallel with respect to the previous frame, or that a scene change has occurred and the encoding is performed.

According to the second aspect of the present invention, a motion vector search is performed for blocks at four corners or near four corners of the non-independent frame, and when motion-compensated prediction is possible in any block, the frame is bidirectionally predicted. It is characterized by setting to possible non-independent frames.

According to a third aspect of the present invention, when the motion vector searched for the four corners or the blocks near the four corners of the non-independent frame and the motion vector searched for the block at the center or near the center of the frame match. It is characterized in that it is determined that the entire screen is moving in parallel with respect to the previous frame, and that encoding is performed.

The invention according to claim 4 is characterized in that motion compensation prediction is performed for other blocks of the entire frame by using the matched motion vectors.

According to the fifth aspect of the invention, motion vector search is performed for blocks at four corners or in the vicinity of four corners of the non-independent frame, and when there is no motion vector in any block, a scene change is performed for the previous frame. It is characterized in that it is judged to be one and encoded.

According to a sixth aspect of the invention, when the scene is changed, the non-independent frame is reset to an independent frame.

[0019]

The differentiator calculates the difference between the previous frame and the previous frame for the macro blocks at the four corners. If the difference between all four macroblocks is 0, normal encoding is performed. When the difference between all four macroblocks is not 0,
The frame memory & predictor searches for motion vectors for macro blocks at the four corners, and if motion compensation prediction based on motion vectors is possible in any of the macro blocks,
If the frame is set as a non-independent frame capable of bidirectional prediction, and if motion compensation prediction is not possible, it is determined that the scene has changed. Furthermore, the motion vector is searched for the macroblock in the center of the screen, and if the motion vector of the macroblock in the center and the motion vectors of the macroblocks in the four corners match, it is determined that the screen is a parallel movement, and All the macroblocks are encoded in the motion compensation mode using the same motion vector as that of the macroblock.

[0020]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be specifically described below with reference to the drawings. First, encoding and decoding of a moving image, which is a premise of the present invention, will be described. FIG. 3 shows a moving picture encoding,
It is a block block diagram of the system which performs a decoding. The input device 1 inputs moving image signals of various formats such as Y, Cb, and Cr. The preprocessor 2 converts into a format required by the encoder. The encoder 3 is
A bit stream is generated by reducing the data amount without degrading the input moving image as much as possible.

The storage device 4 is composed of a CD, a DAT, a hard disk, etc., and stores the generated bit stream. The decoder 5 receives the bitstream and creates a reproduced image. The post-processor 6 performs processing such as line interpolation, pixel interpolation, rate conversion, frame field conversion, and pixel aspect ratio conversion according to the specifications of the output display. The output device 7 displays and outputs the reproduced moving image.

FIG. 4 shows a block diagram of the encoder 3. Before describing the encoder 3, the encoding mode will be briefly described. The coding modes are roughly classified into two modes: (1) intra coding mode and (2) non-intra coding mode.

In the intra coding mode, the input image is coded as it is. On the other hand, in the non-intra coding mode, that is, the predictive coding mode, the predicted image is referred to by referring to the image already coded in any of the three prediction modes of forward, backward, or their interpolation. A generated image is generated, and a difference image from the predicted image is encoded.
At this time, in the case of the prediction mode and the motion compensation prediction, the motion vector is also coded at the same time.

Explaining the encoder 3, the input image data (ID) is subjected to the DCT without passing through the differentiator 301 according to the encoding mode and in the intra encoding mode.
In the case of the non-intra coding mode, that is, the mode of performing coding by performing some prediction, the prediction image (PID) according to the prediction mode and the motion vector
The difference with the difference image (S
ID) is input to the DCT unit 302. DCT device 302
Transformation coefficient (C) of the result of the discrete cosine transformation in
Is quantized by the quantizer 303, variable-length coded by the VLC unit 304, and stored in the buffer 305. The encoded data stored in the buffer 305 is read at a constant rate and sent to DSM (digital storage medium) or the like.

The predictive image in the non-intra coding mode is generated by the predictor from the locally decoded image stored in the frame memory of the frame memory & predictor 309. The locally decoded image is encoded by the encoder at the same time as
It is an image locally decoded by performing reverse decoding processing. This is because at the time of encoding, the data (QC) quantized by the quantizer 303 is inversely quantized by the inverse quantizer 306 and inversely DCT by the inverse DCT unit 307, which is already stored in the frame memory 309. The prediction image generated by the predictor 309 is added to the predicted image generated by the predictor 309 from another locally decoded image that has been generated, is generated, and is stored in the frame memory 309.

The two switches in FIG. 4, that is, the switch 311 between the difference unit 301 and the DCT unit 302.
And the output of the frame memory & predictor 309 and the adder 30
The switch 312 during the input to 8 operates as follows.

In the intra coding mode, the switch 3
11 is connected to the lower side, that is, the ID side, and the switch 312 is connected to the upper side, that is, "0". In the non-intra coding mode, the switch 311 is connected to the upper side, that is, the SID side, and the switch 312 is connected to the lower side, that is, the PID.

FIG. 5 is a diagram showing the configuration of the decoder 5.
The processing of the decoder is basically the same as the local decoder (310 in FIG. 4) in the encoder described above. The encoded data is input to the buffer 501 at a constant rate. The data read from the buffer 501 is the inverse VLC unit 50.
2 and is inversely quantized by the inverse quantizer 503 and inverse D
Inverse DCT is performed by the CT device 504. After that, in the intra coding mode, it is output as it is (“0” is added) through the adder 505, and in the non-intra coding mode, the coding mode and the motion simultaneously decoded in the frame memory & predictor 506 are used. According to the vector, the predicted image generated from the already decoded image on the frame memory is added by the adder 505 and output.

The switch 507 provided between the output of the frame memory & predictor 506 and the input to the adder 505 is connected and operated as follows.

In the intra coding mode, the switch 5
07 is connected to the left side or "0", and in non-intra coding mode, the switch 507 is on the right side or PI
Connected to D.

The structure of the I / P / B picture will be described. FIG. 6A is a diagram showing some pictures in display order, and the arrows indicate the dependency relationship between P pictures and B pictures.

There are four types of pictures. That is, the I (intra) picture is encoded without referring to other pictures. P (predicted)
Pictures are coded with motion compensation from previous I or P pictures. B (bidiritiona
Lly predicted) pictures are encoded using motion compensation from previous or subsequent I or P pictures. Then, the D (DC) picture is used in the fast forward reproduction mode (not shown in FIG. 6A). In a typical encoding mechanism, I, P and B pictures are mixed.

Also, because of the picture dependencies, the order of the bitstreams, that is, the order in which the pictures are transmitted, stored, and received, is not the order in which they are displayed, but the order required by the decoder to decode the bitstream. Become.

FIG. 6B shows a picture sequence in a typical display order, and FIG. 6C shows a bitstream sequence for the picture sequence. Since the B picture depends on the subsequent P picture in the display order, the P picture is transmitted and decoded before the dependent B picture as shown in FIG. 6C.

The MPEG video coding technique has a layer structure corresponding to a hierarchical structure. FIG. 7 shows the data structure of encoded data. The video sequence layer 11 is
It is the top layer of the coding structure and consists of a header and several layers of pictures (GOPs) layer 12. The sequence header initializes the state of the decoder, which allows the decoder to decode any sequence regardless of past decoding history.

The GOP layer 12 is the smallest coding unit that can be individually decoded in the sequence, and is composed of a header and several picture layers 13, and contains at least one I picture. The GOP header contains time and edit information.

The picture layer 13 comprises a header and one or more slice layers 14. If the bitstream becomes unreadable in the middle of a picture, the decoder can wait for the next slice layer 14 to recover and not lose the entire picture.

The slice layer 14 comprises a header and one or more macroblock layers 15. The slice header contains information on the position and the quantization scale. This is sufficient to allow recovery from local errors.

The macroblock layer 15 is a basic unit for motion compensation and quantization scale change. Each macroblock layer 15 comprises a header and a 6-component block layer 16. That is, 4 blocks of luminance and 1 block of Cb
Color difference is one block of Cr color difference. The macroblock header contains quantization scale and motion compensation information.

The block layer 16 is a basic coding unit, and DCT is applied to this block level. Each block consists of 64 pixels in an 8 × 8 matrix. The pixel values are not individually coded, but are elements of a coded block. Each luminance pixel corresponds to one picture pixel, but since the chrominance information is subsampled 2: 1 in both horizontal and vertical directions, each chrominance pixel corresponds to four picture pixels.

Next, a technique used in moving picture coding / decoding will be described. The compression of a moving image is configured by the technique described below. That is, (1) human vis
chrominance signal sub-samples adapted to the sensitivity of the dual system (HVS), (2) quantization, (3) predictive coding, (4) motion compensation (MC) using temporal redundancy,
(5) Discrete cosine transform (DC
T) frequency conversion, (6) variable length coding (V
LC) and (7) image interpolation.

(1) The subsample HVS of the color difference signal has the highest sensitivity to the resolution of the luminance component of the image, so that the Y component of the pixel is encoded with perfect resolution. HV
Since S has poor sensitivity to color difference components, C of the pixel
The pixels of the r and Cb components can be discarded by sub-sampling to reduce the amount of information. The sub-sample ratio (= 1 color difference pixel for the surrounding 2 × 2 luminance pixels)
The relationship is 4: 1: 1).

(2) Quantization Quantization indicates a value in a certain range with one value within the range. For example, the process of rounding a real number to the nearest integer is also one quantization. The quantization range can be simply indicated by an integer code, which is used in decoding the quantized value during decoding. The difference between the true value and the quantized value is called quantization noise, and under certain conditions, HVS is insensitive to quantization noise, so noise can be made large, and therefore the compression effect can be enhanced. ..

(3) Predictive coding Predictive coding is a method of compressing using statistical redundancy. That is, both the encoder and the decoder can estimate or predict the value of pixels that have not yet been encoded or decoded based on the previously decoded value. The difference between the predicted value and the true value is encoded. This difference is the prediction error that the decoder uses to correct the prediction. Usually, the values of pixels are not so different in spatially close regions, so many error values are small and concentrated around 0. The distribution of prediction error is biased to a certain value.

(4) Motion Compensation Motion compensation (MC) searches blocks having strong correlation from past and future frames, obtains a translation vector, and moves blocks corresponding to the preceding or subsequent frames to correspond them. The difference between the frames is encoded. Motion is indicated by a two-dimensional motion vector that transforms the block into a new position. The simplest example is that the camera is not moving,
This is when no object in the scene is moving. The pixel value at the position of each image does not change, and the motion vector of each block is also 0. Generally, the encoder must send a motion vector for each block. This technique is generally based on the fact that many objects do not move in the short sequence of images of the same scene, others move slightly.

(5) Frequency (DCT) Transform The discrete cosine transform (DCT) transforms pixel values of an 8 × 8 block into a matrix of 8 × 8 horizontal / vertical spatial frequency coefficients. The pixel values of the 8x8 block are reconstructed by applying the inverse discrete cosine transform (IDCT) to the spatial frequency coefficients. In general, most of the energy is concentrated in the low frequency coefficients located in the upper left corner of the transformation matrix.

DCT at the upper left position (0,0) of the block
The coefficient indicates zero horizontal and zero vertical frequency and is called a DC coefficient. The DC coefficient is proportional to the average value of the pixel values of the 8 × 8 block, and the difference from the average value of the pixel values of the neighboring 8 × 8 blocks is likely to be relatively small. Therefore, the DC coefficient is further compressed using the predictive coding. To be done. The other coefficient represents a horizontal / vertical frequency in which one of them is not zero, and is called an AC coefficient.

The order of the coefficients is zigzag in order to concentrate the non-zero coefficients at the beginning of the sequence and to maximize the number of consecutive zero coefficients at the end of the order. This order will concentrate the highest spatial frequencies at the end of the sequence.

DCT is one of the transformations called orthogonal transformation, and transforms an image signal from the spatial domain to the frequency domain. Image encoding by orthogonal transformation usually involves converting an original image into N ×
It is divided into N pixel blocks and processed in units of divided blocks. By the orthogonal transform, N × N two-dimensional frequency components (transform coefficients) equal to the number of pixels of the original block are obtained, and these are quantized and variable-length coded.

In image coding, the advantage of converting an image signal into the frequency domain is due to the following properties of the image signal. That is, since the image signal generally has a very high degree of spatial correlation, most of the signal components in the frequency domain are concentrated in the low frequency components, and many high frequency components are close to zero. Therefore, by encoding each frequency component with bit allocation according to the degree of power concentration, it is possible to achieve highly efficient encoding of an image signal.

Specifically, the quantization precision of the transform coefficient is made finer in the low frequency part and coarser as it becomes a high frequency component, and the coefficients which have become zero as a result of the quantization are collectively expressed by zero run length. , Encode. The DCT transform is equivalent to expanding the original image into a linear combination of basis functions composed of cosine waves with different frequencies.

(6) Variable Length Coding Variable length coding (VLC) is a statistical coding technique that assigns a code word to a value to be coded. Short codewords are assigned to high-frequency values, and long codewords are assigned to low-frequency values.

(7) Image Interpolation If the decoder can reconstruct the current image from the past and future images, it is possible to construct an intermediate image using the techniques of interpolation and bidirectional prediction. The macroblock of the intermediate image is
Predicted forward / backward and transformed using motion vectors. The decoder can reconstruct the pixel value of a given macroblock as an average value from past / future images.

FIG. 1 is a flowchart of the interframe predictive coding method of the present invention, which is processed by the encoder. In step 101, for the macro blocks A, B, C, and D at the four corners shown in FIG. 2, the differentiator takes the difference between the previous frame and the previous frame, and if the difference is not zero in all four macro blocks, Proceed to 102. If the difference is 0, the process proceeds to step 109, and normal encoding is performed according to a predetermined picture type.

In step 102, the frame memory & predictor searches for a motion vector for macroblocks at the four corners. If the motion compensation mode can be selected by the motion vector in any macroblock, the process proceeds to step 103. If not, the process proceeds to step 107. In step 103, this frame is set as a B picture (a bidirectionally predictable non-independent frame).

In step 104, the motion vector is searched for the macro block E at the center of the screen. It is possible to select the motion compensation mode according to the motion vector of the central macroblock, and the motion vector and step 1
If the motion vector selected in 02 matches, the process proceeds to step 105. If not, the process proceeds to step 106.

In step 105, it is determined that the entire screen is moved in parallel, and the same motion vector as that of the central macroblock is used to encode all macroblocks in the motion compensation mode.

In step 106, since it is determined that the translation is not performed, normal non-intra coding is performed. In step 107, it is determined that there is a scene change, this frame (the relevant frame) is set as an independent frame, and the process proceeds to step 108 to perform normal intra coding.

The present embodiment is not limited to the above, and for example, macro blocks near the four corners and macro blocks near the center of the screen may be used.

[0060]

As described above, as described above,
According to the inventions described in 5 and 6, the parallel movement or the scene change of the entire screen is determined by calculating the difference between the four corners of the non-independent frame or the blocks near the four corners and the previous frame. It is possible to determine the scene change or the parallel movement of the entire screen with a small amount of calculation, and it is possible to automatically select the optimum coding mode for each.

According to the third and fourth aspects of the invention, the motion vector searched for the four corners or the blocks near the four corners of the non-independent frame and the motion vector searched for the block at the center or near the center of the frame match. Then, it is determined that the entire screen is moving in parallel to the previous frame, the motion vector search is suppressed to the minimum necessary macroblock, and motion compensation is performed for all macroblocks based on the determined motion vector. Since coding is performed by prediction, the coding efficiency improves and
By reducing the amount of calculation, it becomes possible to perform encoding processing at high speed.

[Brief description of drawings]

FIG. 1 is a flowchart of an interframe predictive coding system according to the present invention.

FIG. 2 is a diagram showing macro blocks at four corners and a center.

FIG. 3 is a block configuration diagram of a system for encoding and decoding a moving image.

FIG. 4 is a block configuration diagram of an encoder.

FIG. 5 is a diagram showing a configuration of a decoder.

FIG. 6A is a diagram showing some pictures in display order, and FIG. 6B shows a picture sequence in a typical display order;
(C) shows a bitstream sequence for it.

FIG. 7 is a diagram showing a data structure of encoded data.

[Explanation of symbols]

 1 Input Device 2 Preprocessor 3 Encoder 4 Storage Device 5 Decoder 6 Postprocessor 7 Output Device

Claims (6)

[Claims] 1. An independent frame is set at an arbitrary interval from continuous frames of an input image signal, the independent frame is independently coded within the frame, and signals of non-independent frames between the independent frames are output to the front and back. In an inter-frame predictive coding method of predicting and coding based on a signal of an independent frame, a difference between a block at four corners or a neighborhood of four corners of a non-independent frame with respect to a previous frame is calculated, and in any block, the difference is calculated. An inter-frame predictive coding method characterized in that when the difference is not zero, the whole screen is translated in relation to the previous frame or the scene is changed and the coding is performed. 2. A motion vector search is performed for blocks at four corners or near four corners of the non-independent frame, and when motion-compensated prediction is possible in any block, the frame is changed to a bi-directionally predictable non-independent frame. The interframe predictive coding method according to claim 1, wherein the interframe predictive coding method is set. 3. When a motion vector searched for a block at four corners of the non-independent frame or a block near the four corners and a motion vector searched for a block at the center of the frame or a block near the center of the frame coincide with each other, the motion vector for the previous frame is compared. The inter-frame predictive coding method according to claim 1, wherein coding is performed by determining that the entire screen is moving in parallel. 4. For other blocks of the entire frame,
The inter-frame predictive coding method according to claim 3, wherein motion-compensated prediction is performed using the matched motion vector. 5. A motion vector search is performed for blocks at four corners or in the vicinity of four corners of the non-independent frame, and when there is no motion vector in any block, it is determined that a scene change has been performed with respect to the previous frame, and the code is coded. The interframe predictive coding system according to claim 1, wherein 6. The interframe predictive coding method according to claim 5, wherein when the scene is changed, the non-independent frame is reset to an independent frame.