[go: up one dir, main page]

WO2013077304A1 - Dispositif de codage d'image, dispositif de décodage d'image et procédés et programmes correspondants - Google Patents

Dispositif de codage d'image, dispositif de décodage d'image et procédés et programmes correspondants Download PDF

Info

Publication number
WO2013077304A1
WO2013077304A1 PCT/JP2012/080019 JP2012080019W WO2013077304A1 WO 2013077304 A1 WO2013077304 A1 WO 2013077304A1 JP 2012080019 W JP2012080019 W JP 2012080019W WO 2013077304 A1 WO2013077304 A1 WO 2013077304A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
block
unit
encoding
prediction
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2012/080019
Other languages
English (en)
Japanese (ja)
Inventor
大津　誠
内海　端
貴也 山本
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sharp Corp
Original Assignee
Sharp Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sharp Corp filed Critical Sharp Corp
Publication of WO2013077304A1 publication Critical patent/WO2013077304A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/593—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial prediction techniques
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding

Definitions

  • luminance compensation is performed on the reference block specified by the ID indicating the reference image and the disparity vector using Expression (1). Further, the residual component is added to the luminance compensated image to reproduce the original image. Color compensation can also be realized by a method similar to luminance compensation.
  • the offset parameter in color compensation will be described as C CC
  • the scale parameter will be described as S CC .
  • MPEG-3DV an MPEG ad hoc group
  • the depth image is information representing the distance from the camera to the subject, and as a generation method, for example, there is a method of obtaining from a device that measures the distance installed in the vicinity of the camera.
  • a depth image can be generated by analyzing an image taken from a multi-viewpoint camera.
  • the decoder 907 receives the output result of the encoder 906 that is transmitted via the network N or directly, decodes it, and outputs a decoded image and a decoded depth image.
  • the display unit 908 receives the decoded image and the decoded depth image from the decoder 907 as input, and displays the decoded image, or displays the decoded image after performing processing using the depth image.
  • the present invention has been made in view of the above-described circumstances, and an object of the present invention is to always provide a highly accurate image without requiring additional information other than information necessary for parallax compensation prediction in the parallax compensation prediction method. It is to be able to carry out inter-characteristic difference compensation processing.
  • another object of the present invention is to perform inter-image characteristic difference compensation processing with high accuracy at all times in the multi-view video and the corresponding viewpoint depth video coding system being developed by MPEG-3DV. This is to perform the parallax compensation prediction with high efficiency.
  • the first technical means of the present invention corrects a difference in characteristics between viewpoint images and encodes a parallax when encoding each viewpoint image taken from at least two viewpoints.
  • a corresponding block extraction unit that extracts a reference block to be referred to when encoding an encoding target block, a block around the encoding target block, and a block around the reference block
  • a correction processing unit that corrects a difference in characteristics between the viewpoint images based on the similarity to.
  • the correction processing unit performs correction by excluding peripheral blocks in which a subject different from the subject in the encoding target block is captured. It is characterized by.
  • the correction processing unit selects one peripheral block most likely to show the same subject as the subject shown in the encoding target block, The correction is performed.
  • the sixth technical means is characterized in that, in the fifth technical means, the depth information is information based on a representative value of a block obtained by dividing a depth image.
  • the correction processing unit executes correction by excluding a peripheral block in which a subject different from the subject in the decoding target block is shown. It is a feature.
  • the correction processing section selects one peripheral block most likely to show the same subject as the subject shown in the decoding target block, and corrects it. It is characterized by executing.
  • the eleventh technical means is characterized in that, in the ninth or tenth technical means, depth information corresponding to each viewpoint image is used for determination of the subject.
  • the twelfth technical means is characterized in that, in the eleventh technical means, the depth information is information based on a representative value of a block obtained by dividing a depth image.
  • the fifteenth technical means causes a computer to execute an image encoding process for correcting parallax compensation by correcting a characteristic difference between viewpoint images when encoding each viewpoint image taken from at least two viewpoints.
  • An image encoding program for extracting a reference block to be referred to when encoding an encoding target block, a block around the encoding target block, and the reference block And a step of correcting a characteristic difference between the viewpoint images based on the similarity to the peripheral blocks.
  • FIG. 3 is a block diagram illustrating a configuration example of a motion / disparity compensation unit in the image encoding unit of FIG. 2.
  • FIG. 4 is a diagram for conceptually explaining a block position extracted by a corresponding block extraction unit in the motion / disparity compensation unit in FIG. 3, and is a conceptual diagram showing a block position handled in an inter-image characteristic difference compensation process. It is a conceptual diagram of the determination process of a representative depth value. It is a flowchart for demonstrating the image coding process which the image coding apparatus of FIG. 1 performs.
  • the viewpoint image is first input to the image encoding unit 101.
  • the reference viewpoint encoding processing unit 102 compresses and encodes the viewpoint image of the reference viewpoint using the intra-view prediction encoding method.
  • intra-view prediction encoding intra-screen prediction or motion compensation is performed within the same viewpoint, and image data is compression-encoded based only on image data within the viewpoint.
  • reverse processing that is, internal decoding is performed for reference when encoding a viewpoint image of a non-reference viewpoint (to be described later), and the image signal is restored.
  • the non-reference viewpoint encoding processing unit 103 compresses and encodes the viewpoint image of the non-reference viewpoint using an inter-view prediction encoding method.
  • the inter-view prediction encoding method parallax compensation is performed using an image of a viewpoint different from the encoding target image, and the image data is compressed and encoded.
  • the non-reference viewpoint encoding processing unit 103 can also select an intra-view prediction encoding method that uses only image data within the viewpoint based on the overall encoding efficiency.
  • the encoded data encoded by the reference viewpoint encoding processing unit 102 and the encoded data encoded by the non-reference viewpoint encoding processing unit 103 are output from the image encoding unit 101.
  • the code construction unit 104 receives the encoded data from the image encoding unit 101, is connected and rearranged, and outputs the encoded stream to the outside of the image encoding device 100 (for example, an image decoding device 700 described later).
  • FIG. 2 is a block diagram illustrating a configuration example of the image encoding unit 101.
  • the image coding unit 101 includes an image input unit 201, a subtraction unit 202, an orthogonal transformation unit 203, a quantization unit 204, an entropy coding unit 205, an inverse quantization unit 206, an inverse orthogonal transformation unit 207, an addition unit 208, and a prediction method.
  • the control unit 209, the image selection unit 210, the deblocking filter unit 211, the frame memory 212, the motion / disparity compensation unit 213, the motion / disparity vector detection unit 214, and the intra prediction unit 215 are configured.
  • the reference viewpoint encoding process and the other non-reference viewpoint encoding processes are explicitly divided into the reference viewpoint encoding processing unit 102 and the non-reference viewpoint encoding process.
  • the reference viewpoint encoding processing unit 103 is used, since there are many processes that are common to each other in practice, a mode in which the reference viewpoint encoding process and the non-reference viewpoint encoding process are integrated will be described below.
  • processing for referencing an image with the same viewpoint as the encoding target viewpoint (motion compensation) and processing for referring to an image with a different viewpoint (parallax compensation) performed in the inter-screen prediction unit 218 are also referred to during encoding.
  • the processing can be made common by using ID information (reference viewpoint number, reference frame number) indicating a reference image, only with different images.
  • ID information reference viewpoint number, reference frame number
  • the method of encoding the residual component between the image predicted by each prediction unit 217 and 218 and the input viewpoint image can be performed in common for both the reference viewpoint and the non-reference viewpoint. Details will be described later.
  • the image input unit 201 outputs the divided image block signal to the subtraction unit 202, the intra prediction unit 215 in the intra-screen prediction unit 217, and the motion / disparity vector detection unit 214 in the inter-screen prediction unit 218.
  • the intra-screen prediction unit 217 is a processing unit that performs encoding using only information in the same screen that has been processed before the encoding target block, and details of the processing will be described later.
  • the inter-screen prediction unit 218 is a processing unit that performs encoding using information on a viewpoint image of the same viewpoint processed in the past or a viewpoint image of a different viewpoint, which is different from the encoding target image. Will be described later.
  • the image input unit 201 repeatedly outputs the blocks until all the blocks in the image frame are completed and all the input images are completed while sequentially changing the block positions.
  • the block size when the image input unit 201 divides the image signal is not limited to the 16 ⁇ 16 size described above, and may be 8 ⁇ 8, 4 ⁇ 4, or the like.
  • the number of vertical and horizontal pixels may not be the same, and may be 16 ⁇ 8, 8 ⁇ 16, 8 ⁇ 4, 4 ⁇ 8, or the like. Examples of these sizes are described in H.C.
  • the block size is not limited to the above size, and can correspond to an arbitrary block size that will be employed in a future encoding scheme.
  • the orthogonal transform unit 203 performs orthogonal transform on the difference image block signal input from the subtracting unit 202 to generate signals indicating the strengths of various frequency characteristics.
  • the difference image block signal is subjected to, for example, DCT transform (discrete cosine transform; Discrete Cosine Transform) and frequency domain signal (for example, DCT transform). Produces a DCT coefficient).
  • DCT transform discrete cosine transform
  • frequency domain signal for example, DCT transform
  • FFT Fast Fourier Transform
  • the orthogonal transform unit 203 outputs the coefficient value included in the generated frequency domain signal to the quantization unit 204.
  • the inverse quantization unit 206 decodes the difference image code input from the quantization unit 204 by performing processing (inverse quantization) opposite to the quantization performed by the quantization unit 204 using the quantization coefficient.
  • a frequency domain signal is generated and output to the inverse orthogonal transform unit 207.
  • the inverse orthogonal transform unit 207 generates a decoded differential image block signal that is a spatial domain signal by performing a process reverse to the orthogonal transform unit 203, for example, inverse DCT transform, on the input decoded frequency domain signal.
  • the inverse orthogonal transform unit 207 can generate a spatial domain signal based on the decoded frequency domain signal, the inverse orthogonal transform unit 207 is not limited to the inverse DCT transform, and other methods (for example, IFFT (Inverse Fast Fourier Transform)) are used. It may be used.
  • IFFT Inverse Fast Fourier Transform
  • the addition unit 208 receives the predicted image block signal from the image selection unit 210 and also receives the decoded difference image block signal from the inverse orthogonal transform unit 207.
  • the adder 208 adds the decoded differential image block signal to the predicted image block signal, and generates a reference image block signal obtained by encoding / decoding the input image (internal decoding).
  • the reference image block signal is output to the intra-screen prediction unit 217 and the inter-screen prediction unit 218.
  • the intra prediction unit 217 receives the reference image block signal from the adder 208 and also receives the image block signal of the encoding target image from the image input unit 201, and predicts the intra prediction in a predetermined direction.
  • the signal is output to the prediction method control unit 209 and the image selection unit 210.
  • the intra-screen prediction unit 217 generates information indicating the direction of prediction necessary for generating the intra-screen prediction image block signal as intra-screen prediction coding information, and outputs the information to the prediction method control unit 209.
  • the intra-screen prediction is performed according to the intra-screen prediction method of the conventional method (for example, H.264 Reference Software Software JM Ver. 13.2 Encoder, http://iphome.hhi.de/suehring/tml/, 2008). Note that these processes in the in-screen prediction unit 217 are executed by the intra prediction unit 215 as shown in the configuration described above.
  • the inter-screen prediction unit 218 receives the reference image block signal from the addition unit 208 and also receives the image block signal of the encoding target image from the image input unit 201, and receives the inter-screen prediction image block signal generated by the inter-screen prediction.
  • the data is output to the prediction method control unit 209 and the image selection unit 210.
  • the inter-screen prediction unit 218 includes information necessary for generating an inter-screen prediction image block signal (reference image information including a reference viewpoint image number and a reference frame number, and a difference vector between a motion / disparity vector and a prediction vector. ) Is generated as inter-screen prediction encoding information, and the generated inter-screen prediction encoding information is output to the prediction scheme control unit 209. Details of the inter-screen prediction unit 218 will be described later.
  • the prediction scheme control unit 209 based on the picture type and coding efficiency of the input image, the intra-screen prediction image block signal input from the intra-screen prediction unit 217, the intra-screen prediction encoding information, and the screen
  • the prediction method for each block is determined based on the inter-screen prediction image block signal input from the inter-prediction unit 218 and the inter-screen coding information, and information on the prediction method is output to the image selection unit 210.
  • the picture type of the input image is information for identifying an image that can be referred to as a prediction image by the encoding target image, and includes an I picture, a P picture, a B picture, and the like. Similar to the coefficient, it is determined by a parameter given from the outside, and the same method as the conventional MVC can be used.
  • the prediction method control unit 209 monitors the picture type of the input image, and if the input encoding target image is an I picture that can only refer to information within the screen, it selects the intra prediction method deterministically. P-pictures that can refer to past frames that have already been encoded or images from different viewpoints, past frames that have been encoded, and future frames that have already been encoded (although they are future frames in display order, they have been processed in the past In the case of a B picture that can refer to an image of a different viewpoint from the frame), the prediction method control unit 209 determines the residual between the number of bits generated by the encoding performed by the entropy encoding unit 205 and the original image of the subtraction unit 202 To calculate the Lagrangian cost using the conventional method (eg, H.264.Reference.Software JM ver. 13.2 Encoder, http://iphome.hhi.de/suehring/tml/, 2008) Alternatively, an inter-screen prediction method is determined.
  • the conventional method
  • the prediction method control unit 209 adds information that can specify the prediction method to the coding information corresponding to the prediction method selected by the above method from the intra-screen prediction coding information or the inter-screen prediction coding information. It outputs to the entropy encoding part 205 as prediction encoding information.
  • the image selection unit 210 performs the intra-screen prediction image block signal input from the intra-screen prediction unit 217 or the inter-screen prediction input from the inter-screen prediction unit 218 according to the prediction method information input from the prediction method control unit 209.
  • the image block signal is selected, and the predicted image block signal is output to the subtraction unit 202 and the addition unit 208.
  • the prediction method input from the prediction method control unit 209 is intra-screen prediction
  • the image selection unit 210 selects and outputs the intra-screen prediction image block signal input from the intra-screen prediction unit 217, and performs prediction.
  • the prediction method input from the method control unit 209 is inter-screen prediction
  • the inter-screen prediction image block signal input from the inter-screen prediction unit 218 is selected and output.
  • the entropy encoding unit 205 packs the differential image code input from the quantization unit 204, the quantization coefficient used by the quantization unit 204, and the prediction encoding information input from the prediction scheme control unit 209. And encoding using variable length coding (entropy coding), for example, to generate encoded data in which the amount of information is further compressed.
  • the entropy encoding unit 205 outputs the generated encoded data to the code configuration unit 104, and then the code configuration unit 104 outputs the encoded stream to the outside of the image encoding device 100 (for example, the image decoding device 700). .
  • the deblocking filter unit 211 receives the reference image block signal from the adder unit 208 and reduces the block distortion that occurs during image coding (for example, H.264 Reference Software JM ver. 13.2). Encoder, http://iphome.hhi.de/suehring/tml/, 2008).
  • the deblocking filter unit 211 outputs the processing result (correction block signal) to the frame memory 212.
  • the frame memory 212 receives the correction block signal from the deblocking filter unit 211, and holds the correction block signal as a part of the image together with information that can identify the viewpoint number and the frame number.
  • the frame memory 212 manages the picture type or the image order of the input image by a memory management unit (not shown), and stores or discards the image according to the instruction. For image management, a conventional MVC image management method can also be used.
  • the motion / disparity vector detection unit 214 searches for a block similar to the image block signal input from the image input unit 201 by the block matching described later from the image group stored in the frame memory 212, and vector information indicating the searched block , Generating a viewpoint number and a frame number.
  • the vector information is referred to as a motion vector when the referenced image is the same viewpoint as the encoding target image, and is referred to as a disparity vector when the referenced image is different from the encoding target image.
  • the motion / disparity vector detecting unit 214 calculates an index value between the divided blocks for each area, and searches for an area where the calculated index value is minimum. The index value only needs to indicate the correlation or similarity between the image signals.
  • the motion / disparity vector detection unit 214 uses, for example, the absolute value sum (SAD) of the difference between the luminance value of the pixel included in the divided block and the luminance value in a certain region of the reference image.
  • SAD absolute value sum
  • the SAD between a block (for example, a size of N ⁇ N pixels) divided from the input viewpoint image signal and the block of the reference image signal is expressed by the following equation (2).
  • I in (i 0 + i, j 0 + j) is the luminance value at the coordinates (i 0 + i, j 0 + j) of the input image, and (i 0 , j 0 ) is the upper left of the divided block
  • I ref (i 0 + i + p, j 0 + j + q) represents the luminance value in the reference image coordinates (i 0 + i + p, j 0 + j + q)
  • (p, q) is shifted relative to the coordinates of the upper left corner of the divided blocks It is a quantity (motion vector).
  • the motion / disparity compensation unit 213 When the reference vector input from the motion / disparity vector detection unit 214 is a disparity vector, the motion / disparity compensation unit 213 simultaneously receives an image block signal around an encoding target block that has already been processed from the frame memory 212. And a corresponding image block signal around the reference image block (the position of the image block will be described later). When the motion / disparity compensation unit 213 calculates an index value indicating the correlation between corresponding neighboring blocks in the encoding target image and the reference image, and determines that the image blocks indicate the same subject to each other, The inter-image characteristic difference compensation is performed (details will be described later). The motion / disparity compensation unit 213 outputs the processed image block to the prediction method control unit 209 and the image selection unit 210 as an inter-screen prediction image block signal.
  • the motion / disparity compensation unit 213 does not perform the above-described peripheral block input and inter-image characteristic difference compensation processing.
  • the reference image block extracted from the frame memory 212 based on the motion vector, the reference image number, and the frame number is output as it is to the prediction method control unit 209 and the image selection unit 210 as an inter-screen prediction image block signal.
  • the motion / disparity compensation unit 213 further subtracts the prediction vector generated based on the motion / disparity vector adopted by the encoded block adjacent to the encoding target block from the motion / disparity vector calculated by the block matching. Then, a difference vector is calculated.
  • the prediction vector generation method is the median value of the horizontal and vertical components of the vector adjacent to the block to be encoded, the block adjacent to the upper right, and the block adjacent to the left. To obtain a prediction vector.
  • a method for calculating a prediction vector a method adopted in MVC can be used.
  • the motion / disparity compensation unit 213 outputs the difference vector and the reference image information (reference view number, reference frame number) to the prediction scheme control unit 209 as inter-frame coding information. Note that it is necessary that at least the reference viewpoint image number and the reference frame number match the region most similar to the input image block detected by block matching and the region indicated by the prediction vector.
  • FIG. 3 is a block diagram illustrating a configuration example inside the motion / disparity compensation unit 213.
  • the motion / disparity compensation unit 213 includes a corresponding block extraction unit 301, a correction coefficient calculation unit 302, and a correction processing unit 303.
  • the motion / disparity vector, reference viewpoint number, and reference frame number input from the motion / disparity vector detection unit 214 are input to the corresponding block extraction unit 301.
  • the corresponding block extraction unit 301 extracts a corresponding image block (reference image block) from the frame memory 212 based on the input motion / disparity vector, reference viewpoint number, and reference frame number.
  • FIG. 4 is a diagram for conceptually explaining the block positions extracted by the corresponding block extraction unit 301.
  • an encoding target image 401 indicates an encoding target viewpoint image
  • a reference image 402 indicates an image referred to by the encoding target image 401.
  • the reference image 402 is an image in which the entire image has already been encoded and decoded and stored in the frame memory 212, and is a viewpoint image with a different viewpoint from the encoding target image 401.
  • encoding / decoding has been completed up to the block immediately before the encoding target block position.
  • the block on which the encoding process is being performed is the encoding target block 403, and the vector 407 extending therefrom is a disparity vector input from the motion / disparity vector detection unit 214.
  • a specific block (reference block 405) of the reference image is specified by the disparity vector, the reference image number, and the reference frame number.
  • the encoding target block peripheral image block 404 is a block above the encoding target block 403 (adjacent block A), an upper right block (adjacent block B), and a left block (adjacent block C).
  • the reference block peripheral image block 406 is a block above the reference block 405 (adjacent block A ′), an upper right block (adjacent block B ′), and a left block (adjacent block C ′).
  • the relative position of the block 404 around the encoding target block 403 viewed from the encoding target block 403 is the reference block 405 extracted by the corresponding block extraction unit 301 as the reference source of the encoding target block 403. This corresponds to the relative position of the block 406 around the reference block 405 as seen from FIG.
  • this correspondence relationship is based on the disparity vectors in the encoding target block 403, and adjacent blocks A and A ′, B and B ′, and C It may be uniform in each relationship of C ′.
  • the correspondence relationship described above is based on the disparity vectors of the encoding target block 403 and its adjacent blocks A, B, and C, for example, the relative position of the adjacent block A with respect to the encoding target block 403 and the reference block of the adjacent block A ′
  • the correspondence relationship between the relative position of 405 and the relative position of the adjacent block B with respect to the encoding target block 403 and the relative position of the adjacent block B ′ with respect to the reference block 405 may be different.
  • the corresponding block extraction unit 301 when the reference vector input from the motion / disparity vector detection unit 214 is a disparity vector, the corresponding block extraction unit 301 includes the peripheral block (encoding target block peripheral image block) 404 of the encoding target block 403.
  • the image signal and the image signal of the peripheral block (reference block peripheral image block) 406 of the reference block 405 are extracted and output to the correction coefficient calculation unit 302.
  • an index value is calculated between the divided adjacent blocks, and determination is performed based on the obtained index value. Similar to the block matching described above, the sum of absolute values of differences in luminance values as shown in Expression (3) can be introduced into the index value.
  • I target and I ref indicate the luminance value of the encoding target image and the luminance value of the reference image, respectively.
  • the coordinates of the upper left corner of adjacent blocks A, B, and C of the encoding target image and the reference image on the right side are represented by (i block , j block ).
  • the above index value is calculated for each adjacent block of the encoding target block and the corresponding adjacent block of the reference image, and similarity determination is executed.
  • the value “20” can be used as the threshold value.
  • the index value expressed by the equation (3) is smaller than this threshold, it is determined that the similarity of individual blocks is high.
  • the similarity is calculated for each of the blocks A, B, and C, and it is determined that the similarity is high in all the blocks, calculation of a correction coefficient and correction processing described later are performed.
  • the threshold value used here is transmitted to the image decoding device (if a fixed threshold value is used for the entire image, it is only necessary to transmit the threshold value once at the beginning of processing), and the same similarity determination result is obtained at the time of decoding. There is a need.
  • the threshold value may be common to the entire image, or a method of adaptively switching in consideration of the encoding efficiency of the image may be employed. In the adaptive switching method, it is necessary to transmit all the information to the image decoding apparatus.
  • correction processing is performed when it is determined that the similarity is high in all the blocks A, B, and C.
  • the present invention is not particularly limited. For example, correction processing is performed when the similarity is high in any one or two blocks (or one or two blocks at a predetermined position), and the number of blocks to be determined (or the position of the block to be determined) May be transmitted to the image decoding apparatus together with a threshold value.
  • the absolute value sum of the differences in luminance values is used as an index for determining similarity, any method can be used as long as it is an index that can determine the similarity of corresponding blocks.
  • correction coefficient calculation When it is determined that correction processing is performed in the similarity determination described above, correction using the formula (1) employed in the above-described conventional technique may be executed.
  • a correction coefficient is used.
  • the calculation unit 302 calculates a correction coefficient as shown in Equation (4).
  • an offset of an average value of luminance values (offset Y ) and an offset of an average value of chrominance components (offset U , offset V ) are introduced as correction coefficients.
  • the luminance I used in the above equation (3) and Y used in the equation (4) are the same.
  • the correction coefficient calculation unit 302 outputs the offset value calculated in this way to the correction processing unit 303.
  • the correction processing unit 303 receives the reference image block signal from the corresponding block extraction unit 301 and performs correction processing based on the correction coefficient input from the correction coefficient calculation unit 302.
  • the correction processing unit 303 does not perform the correction process and does not perform the correction process.
  • the reference block signal input from the extraction unit 301 is output as it is.
  • the correction process is performed as shown in the following equation (5).
  • Y (x, y), U (x, y), and V (x, y) are pixel values of the reference image block input from the corresponding block extraction unit 301.
  • the motion / disparity compensation unit 213 outputs the image block signal output from the correction processing unit 303 as an inter-screen prediction image block signal to the prediction method control unit 209 and the image selection unit 210, and at the same time,
  • the difference vector calculated from the reference vector input from the vector detection unit 214 and the reference image information are also output to the prediction method control unit 209.
  • a characteristic difference between viewpoint images (that is, a characteristic difference between captured images between different viewpoints) is calculated. Correct and perform parallax compensation.
  • the motion / parallax compensation unit 213 performs this parallax compensation processing.
  • the parallax compensation itself is a process that takes into account the redundancy between the viewpoints of different viewpoint images.
  • the correction processing unit 303 of the motion / disparity compensation unit 213 determines whether each viewpoint image (between viewpoints) is based on the similarity between the block around the encoding target block and the block around the reference block indicated by the disparity vector. Correct the difference in image characteristics.
  • correction coefficient calculation method there is a method of using a depth information corresponding to an image and excluding from the correction coefficient calculation a block that is supposed to include a subject different from the encoding target block.
  • the correction processing unit 303 performs the correction by excluding peripheral blocks in which a subject different from the subject in the encoding target block is captured.
  • the depth information is image information indicating the distance to the subject shown in the image.
  • the depth information to be used is depth information about the encoding target block and its surrounding blocks, and the reference block and its surrounding blocks.
  • the depth image handled by the above-mentioned MPEG-3DV can be used in this example.
  • a depth image corresponding to the encoding target image is input from the outside of the image encoding apparatus 100, and representative values of depth values corresponding to the encoding target block and the encoding target block peripheral image block positions are calculated. Based on the calculated depth value, if the difference between the representative depth value of the encoding target block and the representative value of the depth value of the surrounding block at each position is larger than a predetermined value, it is excluded from the correction coefficient calculation processing. . Specifically, processing is performed using the following equation (6).
  • FLG (•) is a flag that determines whether or not to exclude from the correction coefficient calculation processing at a predetermined block position, and is given as follows.
  • BK indicates the position of the adjacent peripheral block.
  • D cur is a representative value of the depth value of the encoding target block
  • D (BK) is a representative value of the depth value in the peripheral block BK of the encoding target block.
  • a TH D is the determination threshold (the predetermined value).
  • the representative value of the depth value for each divided block can be calculated by the following method, for example. Specifically, a frequency distribution (histogram) of depth values in the block is created, and the depth value having the highest appearance frequency is extracted and determined as a representative value.
  • FIG. 5 shows a conceptual diagram of the representative depth value determination process.
  • the depth image 502 illustrated in FIG. 5B is given as the depth image corresponding to the viewpoint image 501 illustrated in FIG.
  • the depth image is represented as a monochrome image with luminance only.
  • the depth value 505 having the highest appearance frequency is represented by the block 503. Determined as the depth value.
  • the representative value of the depth value in addition to the method based on the histogram as described above, it may be determined according to the following method. For example, (a) an intermediate value of the in-block depth value, (b) an average value in consideration of the appearance frequency of the in-block depth value, and (c) a value closest to the camera among the in-block depth values (intra-block depth) (D) (maximum value), (d) the most distant value from the camera among the in-block depth values (minimum value of the in-block depth value), or (e) the depth value at the center position of the block among the in-block depth values. Either of these may be extracted and determined as a representative value.
  • the criteria for selecting which method is, for example, a method in which the most efficient method is fixed to a common method for encoding and decoding, and a parallax prediction using a depth representative value obtained based on each method
  • it is not necessary to prepare the above-described most frequently occurring depth value 505 and all of (a) to (e), and at least two of them may be selected.
  • the block size for dividing the depth image may be matched to the divided block size of the image.
  • the size is not limited to 16 ⁇ 16 size, and may be 8 ⁇ 8, 4 ⁇ 4, or the like.
  • the number of vertical and horizontal pixels may not be the same, and may be 16 ⁇ 8, 8 ⁇ 16, 8 ⁇ 4, 4 ⁇ 8, or the like.
  • a method of selecting an optimum size according to the size of a subject included in a depth image or a corresponding viewpoint image, a required compression rate, or the like is also possible.
  • the example which adopted the information based on the representative value of the block which divided the depth image is given as depth information, it is not limited to this. For example, if it is acceptable to slightly reduce the accuracy, it is prepared for each slice. Deformed depth information can also be used.
  • the example which used the depth information corresponding to each viewpoint image (encoding object image) was given for determination of a subject, other information is used, such as performing color determination, edge detection, etc., and using that information.
  • the amount of parallax can be obtained from a plurality of different viewpoint images, and the determination can be made using the difference in the amount of parallax for each region.
  • the depth information is information that can be calculated from the parallax
  • this determination method is almost equivalent to using the depth information, but there is an advantage that it is not necessary to use the depth information separately.
  • only one depth information (or other information) is prepared for each block (or only one is acquired), and processing is performed, so that one periphery of the encoding target block is executed. Since it is not determined that a plurality of subjects appear in the block, that is, it is determined that both the same subject and the different subject are included in the surrounding blocks in the encoding target block. Therefore, only the peripheral blocks where different subjects are captured can be excluded.
  • the select ( ⁇ ) function outputs the value of the corresponding argument based on the adjacent block position having the depth value closest to the representative depth value of the encoding target block shown in the following equation (9). To do. For example, the value input as the first argument when adjacent block A is closest, the second argument when adjacent block B is closest, and the third argument when adjacent block C is closest Is output.
  • FIG. 6 is a flowchart showing an image encoding process performed by the image encoding apparatus 100. This will be described with reference to FIG.
  • step S101 the image encoding device 100 inputs a viewpoint image (reference viewpoint, non-reference viewpoint) from the outside. Thereafter, the process proceeds to step S102.
  • viewpoint image reference viewpoint, non-reference viewpoint
  • step S102 the image encoding unit 101 encodes a viewpoint image input from the outside.
  • the image encoding unit 101 outputs the encoded data to the code configuration unit 104. Thereafter, the process proceeds to step S103.
  • step S103 if the input of the viewpoint image from the outside is completed, the image encoding device 100 proceeds to step S104. If the input of the viewpoint image from the outside is not completed, the image encoding device 100 returns to step S101 and repeats the process.
  • step S104 the code configuration unit 104 receives encoded data of images from a plurality of viewpoints from the image encoding unit 101, concatenates and rearranges the encoded data, and outputs the encoded data as an encoded stream to the outside of the image encoding device 100. Output to.
  • the image encoding unit 101 repeats the processing from step S202 to step S210 for each image block in the frame. Next, the process proceeds to step S203 and step S204.
  • the intra prediction unit 217 receives the image block signal of the viewpoint image from the image input unit 201 and also receives the reference image block signal decoded (internally decoded) from the addition unit 208, and performs intra prediction. To do.
  • the intra-screen prediction unit 217 outputs the generated intra-screen prediction image block signal to the prediction method control unit 209 and the image selection unit 210, and outputs the intra-screen prediction encoding information to the prediction method control unit 209.
  • a reset image block an image block in which all pixel values are 0
  • the process of the in-screen prediction unit is completed, the process proceeds to step S205.
  • step S204 the inter-screen prediction unit 218 inputs the image block signal of the viewpoint image from the image input unit 201 and also receives the reference image block signal decoded (internally decoded) from the addition unit 208, and performs inter-screen prediction. To do.
  • the inter-screen prediction unit 218 outputs the generated inter-screen prediction image block signal to the prediction method control unit 209 and the image selection unit 210, and outputs the inter-screen prediction coding information to the prediction method control unit 209.
  • a reset image block an image block signal in which all pixel values are 0
  • the process of the inter-screen prediction unit 218 is completed, the process proceeds to step S205.
  • step S ⁇ b> 205 the prediction method control unit 209 receives the intra-screen prediction image block signal and the intra-screen prediction encoding information from the intra-screen prediction unit 217, and the inter-screen prediction image block signal and the inter-screen prediction from the inter-screen prediction unit 218. Encoding information is received, and a prediction mode with good encoding efficiency is selected based on the aforementioned Lagrangian cost.
  • the prediction method control unit 209 outputs information on the selected prediction mode to the image selection unit 210.
  • the prediction scheme control unit 209 adds information for identifying the selected prediction mode to the prediction coding information corresponding to the selected prediction mode, and outputs the information to the entropy coding unit 205.
  • the image selection unit 210 performs an intra-screen prediction image block signal input from the intra-screen prediction unit 217 or an inter-screen prediction image input from the inter-screen prediction unit 218 according to the prediction mode information input from the prediction method control unit 209. A block signal is selected and output to the subtraction unit 202 and the addition unit 208. Thereafter, the process proceeds to step S206.
  • step S206 the subtraction unit 202 subtracts the predicted image block signal input from the image selection unit 210 from the image block signal input from the image input unit 201 to generate a difference image block signal.
  • the subtraction unit 202 outputs the difference image block signal to the orthogonal transformation unit 203. Thereafter, the process proceeds to step S207.
  • the orthogonal transformation unit 203 receives the difference image block signal from the subtraction unit 202 and performs the above-described orthogonal transformation.
  • the orthogonal transform unit 203 outputs the signal after the orthogonal transform to the quantization unit 204.
  • the quantization unit 204 performs the above-described quantization processing on the signal input from the orthogonal transform unit 203 to generate a difference image code.
  • the quantization unit 204 outputs the difference image code and the quantization coefficient to the entropy coding unit 205 and the inverse quantization unit 206.
  • the entropy encoding unit 205 packs the differential image code input from the quantization unit 204, the quantization coefficient, and the prediction encoding information input from the prediction scheme control unit 209, and performs variable length encoding. (Entropy coding) is performed to generate encoded data in which the amount of information is further compressed. The entropy encoding unit 205 outputs the encoded data to the external code configuration unit 104. Thereafter, the process proceeds to step S208.
  • step S208 the inverse quantization unit 206 receives the difference image code and the quantization coefficient from the quantization unit 204, and performs the inverse process of the quantization performed by the quantization unit 204.
  • the inverse quantization unit 206 outputs the generated signal to the inverse orthogonal transform unit 207.
  • the inverse orthogonal transform unit 207 receives the inversely quantized signal from the inverse quantization unit 206, performs the inverse orthogonal transform process of the orthogonal transform process performed by the orthogonal transform unit 203, and performs a difference image (decoded difference image block signal). ).
  • the inverse orthogonal transform unit 207 outputs the decoded difference image block signal to the addition unit 208. Thereafter, the process proceeds to step S209.
  • step S209 the addition unit 208 decodes the input image by adding the predicted image block signal input from the image selection unit 210 to the decoded differential image block signal input from the inverse orthogonal transform unit 207 ( Reference image block signal).
  • the adding unit 208 outputs the reference image block signal to the intra-screen prediction unit 217 and the inter-screen prediction unit 218. Thereafter, the process proceeds to step S210.
  • step S210 when the image encoding unit 101 has not completed the processes in steps S202 to S210 for all blocks and all viewpoint images in the frame, the block to be processed is changed and the process returns to step S202. When all the processes are completed, the process ends.
  • the processing flow of intra prediction performed in step S203 described above is the conventional method H.264. It may be the same as the processing step of H.264 or MVC intra-screen prediction.
  • step S301 the deblocking filter unit 211 inputs a reference image block signal from the adder unit 208 outside the inter-screen prediction unit 218, and performs the above-described FIR filter processing.
  • the deblocking filter unit 211 outputs the corrected block signal after the filtering process to the frame memory 212. Thereafter, the process proceeds to step S302.
  • step S302 the frame memory 212 receives the correction block signal of the deblocking filter unit 211, and holds the correction block signal as a part of the image together with information that can identify the viewpoint number and the frame number. Thereafter, the process proceeds to step S303.
  • step S303 upon receiving the image block signal from the image input unit 201, the motion / disparity vector detection unit 214 searches for a block similar to the image block from the reference image group stored in the frame memory 212 (block matching). Then, vector information (motion vector / disparity vector) representing the found block is generated. The motion / disparity vector detection unit 214 outputs information (reference viewpoint image number, reference frame number) necessary for encoding including the detected vector information to the motion / disparity compensation unit 213. Thereafter, the process proceeds to step S304.
  • information reference viewpoint image number, reference frame number
  • step S304 the motion / disparity compensation unit 213 inputs information necessary for encoding from the motion / disparity vector detection unit 214, and extracts a corresponding prediction block from the frame memory 212.
  • the motion / disparity compensation unit 213 performs the inter-image characteristic difference compensation process when the correlation between the input peripheral image block signals is high.
  • the motion / disparity compensation unit 213 outputs the finally generated predicted image block signal to the prediction method control unit 209 and the image selection unit 210 as an inter-screen predicted image block signal.
  • the motion / disparity compensation unit 213 uses the reference image block signal as it is without performing inter-image characteristic difference compensation processing. It outputs to the prediction system control part 209 and the image selection part 210 as a block signal.
  • the motion / disparity compensation unit 213 calculates a difference vector between the prediction vector generated based on the vector information of the adjacent block of the encoding target block and the motion / disparity vector input from the motion / disparity vector detection unit 214.
  • the motion / disparity compensation unit 213 outputs the calculated difference vector and information necessary for prediction (reference viewpoint image number and reference frame number) to the prediction method control unit 209. Thereafter, the inter-screen prediction is terminated.
  • the image encoding device 100 corresponds to at least one or more neighboring blocks of the previous reference block indicated by the disparity vector of the encoding target block in the disparity compensation prediction method.
  • parallax compensation prediction no implicit additional information (additional information for inter-image characteristic difference compensation) is required and, of course, there is no need to transmit to the decoding side, and inter-image characteristics during parallax compensation prediction Difference compensation processing can be performed.
  • An encoding method is provided that enables accurate estimation of information necessary for the inter-image characteristic difference compensation process even when different subjects are reflected in the encoding target block and its periphery. Therefore, it is possible to dramatically improve the encoding efficiency.
  • FIG. 9 is a block diagram illustrating a configuration example of an image decoding apparatus according to an embodiment of the present invention.
  • An image decoding device 700 illustrated in FIG. 9 includes a code separation unit 701 and an image decoding unit 702. Note that the blocks and arrows indicated by dotted lines described in the image decoding unit 702 are used for conceptually explaining the operation of the image decoding unit 702.
  • the image decoding apparatus 700 When receiving the transmitted encoded stream, the image decoding apparatus 700 passes the encoded stream to the code separation unit 701.
  • the code separation unit 701 separates the reference viewpoint image encoded data and the non-reference viewpoint image encoded data.
  • the code separation unit 701 outputs the separated reference viewpoint image encoded data and non-reference viewpoint image encoded data to the image decoding unit 702.
  • the reference viewpoint decoding processing unit 703 decodes the encoded data that has been compression-encoded by a method according to intra-view prediction encoding, and restores the viewpoint image of the reference viewpoint.
  • FIG. 10 is a block diagram illustrating a configuration example of the image decoding unit 702.
  • the image decoding unit 702 includes an encoded data input unit 813, an entropy decoding unit 801, an inverse quantization unit 802, an inverse orthogonal transform unit 803, an addition unit 804, a prediction scheme control unit 805, an image selection unit 806, and a deblocking filter unit. 807, a frame memory 808, a motion / disparity compensation unit 809, an intra prediction unit 810, and an image output unit 812.
  • the intra-screen prediction unit 816 and the inter-screen prediction unit 815 are illustrated by dotted lines, the intra-screen prediction unit 816 includes an intra prediction unit 810, and the inter-screen prediction unit 815 includes the deblocking filter unit 807, It is assumed that a frame memory 808 and a motion / disparity compensation unit 809 are included.
  • the reference viewpoint decoding process and the non-reference viewpoint decoding process are explicitly divided into the reference viewpoint decoding process and the non-reference viewpoint decoding process.
  • the reference viewpoint decoding process and the non-reference viewpoint decoding process are integrated.
  • the intra-view prediction decoding method performed as the reference viewpoint decoding processing unit 703 described above is a part of the processing performed by the intra-screen prediction unit 816 and the inter-screen prediction unit 815 of FIG. This is a combination of processing (motion compensation) for referring to an image of the same viewpoint.
  • the inter-view prediction decoding method performed as the non-reference viewpoint decoding processing unit 704 is a process performed by the intra-screen prediction unit 816 and a process referring to the same viewpoint image performed by the inter-screen prediction unit 815 (motion compensation). And a process of referring to images from different viewpoints (parallax compensation).
  • the processing (motion compensation) for referring to an image of the same viewpoint as the processing target viewpoint performed by the inter-screen prediction unit 815 motion compensation
  • ID information reference viewpoint number, reference frame number
  • the processing for restoring the image by adding the residual component obtained by decoding the encoded image data and the image predicted by each of the prediction units 815 and 816 is common to both the reference viewpoint and the non-reference viewpoint. Yes. Details will be described later.
  • the encoded data input unit 813 divides the image encoded data input from the code separation unit 701 into processing block units (for example, 16 pixels ⁇ 16 pixels), and outputs the result to the entropy decoding unit 801.
  • the encoded data input unit 813 repeatedly outputs the blocks until the blocks are sequentially changed and all the blocks in the frame are completed and the input encoded data is completed.
  • the entropy decoding unit 801 processes the encoded data input from the encoded data input unit 813 in a process reverse to the encoding method (for example, variable length encoding) performed by the entropy encoding unit 205 in FIG. Variable length decoding), and a differential image code, a quantization coefficient, and predictive coding information are extracted.
  • the entropy decoding unit 801 outputs the difference image code and the quantization coefficient to the inverse quantization unit 802 and the prediction coding information to the prediction scheme control unit 805.
  • the inverse quantization unit 802 dequantizes the difference image code input from the entropy decoding unit 801 using the extracted quantization coefficient to generate a decoded frequency domain signal, and outputs the decoded frequency domain signal to the inverse orthogonal transform unit 803.
  • the inverse orthogonal transform unit 803 generates a decoded differential image block signal that is a spatial domain signal by performing, for example, inverse DCT transform on the input decoded frequency domain signal.
  • the inverse orthogonal transform unit 803 can generate a spatial domain signal based on the decoded frequency domain signal, the inverse orthogonal transform unit 803 is not limited to the inverse DCT transform, and other methods (for example, IFFT (Inverse Fast Fourier Transform)) are used. It may be used.
  • IFFT Inverse Fast Fourier Transform
  • the prediction scheme control unit 805 takes out the block unit prediction scheme employed in the image coding apparatus 100 shown in FIGS. 1 and 2 from the prediction coding information input from the entropy decoding unit 801.
  • the prediction method is intra-screen prediction or inter-screen prediction.
  • the prediction method control unit 805 outputs information regarding the extracted prediction method to the image selection unit 806.
  • the prediction method control unit 805 extracts the encoded information from the prediction encoded information input from the entropy decoding unit 801, and outputs the encoded information to the processing unit corresponding to the extracted prediction method.
  • the prediction method control unit 805 outputs the encoded information to the intra prediction unit 816 as intra prediction prediction information.
  • the prediction method control unit 805 outputs the encoded information to the inter-screen prediction unit 815 as inter-screen prediction encoded information.
  • the image selection unit 806, based on the prediction method input from the prediction method control unit 805, the intra-screen prediction image block signal input from the intra-screen prediction unit 816 or the inter-screen prediction image block input from the inter-screen prediction unit 815. Select a signal.
  • the prediction method is intra prediction
  • an intra prediction image block signal is selected.
  • the prediction method is inter-screen prediction
  • an inter-screen prediction image block signal is selected.
  • the image selection unit 806 outputs the selected predicted image block signal to the addition unit 804.
  • the addition unit 804 adds the predicted image block signal input from the image selection unit 806 to the decoded difference image block signal input from the inverse orthogonal transform unit 803 to generate a decoded image block signal.
  • the adding unit 804 outputs the decoded decoded image block signal to the intra-screen prediction unit 816, the inter-screen prediction unit 815, and the image output unit 812.
  • the image output unit 812 receives the decoded image block signal from the adder 804 and temporarily holds it as a part of the image in a frame memory (not shown).
  • the image output unit 812 outputs the image to the outside of the image decoding apparatus 700 when all the viewpoint images are prepared after rearranging the frames in the display order.
  • the intra prediction unit 810 in the intra prediction unit 816 receives the decoded image block signal from the addition unit 804 and the intra prediction encoding information from the prediction scheme control unit 805.
  • the intra prediction unit 810 reproduces the intra prediction performed at the time of encoding based on the intra prediction encoding information. Note that intra prediction can be performed according to the conventional method described above.
  • the intra prediction unit 810 outputs the generated prediction image to the image selection unit 806 as an intra-screen prediction image block signal.
  • the deblocking filter unit 807 performs the same processing as the FIR filter performed by the deblocking filter unit 211 on the decoded image block signal input from the addition unit 804, and the processing result (correction block signal) is stored in the frame memory. Output to 808.
  • the frame memory 808 receives the correction block signal from the deblocking filter unit 807 and holds the correction block signal as a part of the image together with information that can identify the viewpoint number and the frame number.
  • the frame memory 808 manages the picture type or image order of the input image by a memory management unit (not shown), and stores or discards the image according to the instruction. For image management, a conventional MVC image management method can also be used.
  • the motion / disparity compensation unit 809 receives inter-frame prediction encoding information from the prediction scheme control unit 805, and from among these, reference image information (reference viewpoint image number and reference frame number) and a difference vector (motion / disparity vector and prediction). Vector difference vector).
  • the motion / disparity compensation unit 809 generates a prediction vector by the same method as the prediction vector generation method performed by the motion / disparity compensation unit 213 described above.
  • the motion / disparity compensation unit 809 reproduces the motion / disparity vector by adding the difference vector to the calculated prediction vector.
  • the motion / disparity compensation unit 809 extracts a target image block signal (predicted image block signal) from the images stored in the frame memory 808 based on the reference image information and the motion / disparity vector.
  • the motion / disparity compensation unit 809 simultaneously performs an image block signal around the decoding target block (decoding target block surrounding image block signal) and an image signal block signal around the prediction image block (prediction image block surrounding image).
  • the block signal) is extracted, and the similarity is determined under the same conditions as in the encoding using the above-described method for determining the similarity used at the time of encoding. Further, according to the determination result, the same inter-image characteristic difference correction processing as that at the time of encoding is performed.
  • the reference image block signal is output as it is without performing the inter-image characteristic difference compensation process.
  • the motion / parallax compensation unit 809 outputs the image block signal after the inter-image characteristic difference correction or the reference image block signal to the image selection unit 806 as an inter-screen prediction image block signal.
  • the image decoding apparatus 700 corrects the characteristic difference between the viewpoint images and performs parallax compensation.
  • the motion / parallax compensation unit 809 performs this parallax compensation processing in the same manner as the motion / parallax compensation unit 213 described above. That is, the motion / disparity compensation unit 809 includes a corresponding block extraction unit that extracts a reference block to be referred to when decoding the decoding target block, a block around the decoding target block, and a block around the reference block indicated by the disparity vector. And a correction processing unit that corrects a characteristic difference between the viewpoint images based on the similarity.
  • the relative positions of the blocks around the decoding target block viewed from the decoding target block are determined from the reference blocks extracted as reference sources of the decoding target block by the corresponding block extraction unit. Preferably, it corresponds to the relative position of the block around the reference block as seen.
  • the correction processing unit performs correction by excluding a peripheral block in which a subject different from the subject in the decoding target block is captured, or a peripheral in which the same subject as the subject in the decoding target block is most likely to be captured It is preferable to perform correction by selecting one block.
  • the depth information corresponding to each viewpoint image (decoding target image) can be used for subject determination.
  • the depth information to be used is depth information about the decoding target block and its peripheral blocks, and the reference block and its peripheral blocks.
  • the depth information is preferably information based on representative values of blocks obtained by dividing the depth image.
  • FIG. 11 is a flowchart showing an image decoding process performed by the image decoding apparatus 700. This will be described with reference to FIG.
  • step S501 the image decoding apparatus 700 receives an encoded stream from the outside (for example, the image encoding apparatus 100), and the code separation unit 701 separates and extracts the image encoded data of each viewpoint. Thereafter, the process proceeds to step S502.
  • step S502 the image decoding unit 702 decodes the encoded image data of the viewpoint image separated and extracted in step S501, and outputs the result to the outside of the image decoding unit 702. Thereafter, the process proceeds to step S503.
  • step S503 when the image decoding apparatus 700 determines that the processing of all viewpoint images has been completed, the image decoding apparatus 700 rearranges the viewpoint images in the time axis direction and arranges the order of the viewpoint directions. The image is output to the outside of the image decoding apparatus 700. If the image decoding apparatus 700 determines that the processing of all the viewpoints and time images has not been completed, the image decoding apparatus 700 returns to step S502 and continues the processing.
  • step S603 the entropy decoding unit 801 performs entropy decoding on the encoded image data input from the encoded data input unit 813, and generates a differential image code, a quantization coefficient, and predictive encoding information.
  • the entropy decoding unit 801 outputs the difference image code and the quantization coefficient to the inverse quantization unit 802, and outputs the prediction coding information to the prediction scheme control unit 805.
  • the prediction scheme control unit 805 receives prediction coding information from the entropy decoding unit 801, and extracts information regarding the prediction scheme and coding information corresponding to the prediction scheme.
  • the prediction method is intra prediction
  • the encoding information is output to the intra prediction unit 816 as intra prediction encoding information.
  • the prediction method is inter-screen prediction
  • the encoding information is output to the inter-screen prediction unit 815 as inter-screen prediction encoding information. Then, it progresses to step S604 and step S605.
  • step S604 the intra prediction unit 810 in the intra prediction unit 816 receives the intra prediction encoding information input from the prediction scheme control unit 805 and the decoded image block signal input from the addition unit 804, and the screen Intra prediction processing is performed.
  • the intra prediction unit 810 outputs the generated intra-screen prediction image block signal to the image selection unit 806.
  • a reset image block signal an image block signal in which all pixel values are 0
  • the process proceeds to step S606.
  • step S605 the inter-screen prediction unit 815 performs inter-screen prediction based on the inter-screen prediction encoding information input from the prediction method control unit 805 and the decoded image block signal input from the addition unit 804.
  • the inter-screen prediction unit 815 outputs the generated inter-screen prediction image block signal to the image selection unit 806.
  • the inter-screen prediction process will be described later.
  • a reset image block signal an image block signal in which all pixel values are 0
  • the process proceeds to step S606.
  • step S ⁇ b> 606 the image selection unit 806 receives the intra-screen prediction image block signal input from the intra-screen prediction unit 816 or the inter-screen prediction unit 815 based on the information related to the prediction method input from the prediction method control unit 805. The input inter-screen prediction image signal is selected and output to the adding unit 804. Thereafter, the process proceeds to step S607.
  • step S607 the inverse quantization unit 802 uses the difference image code and the quantization coefficient input from the entropy decoding unit 801, and performs the inverse of the quantization performed by the quantization unit 204 of the image encoding unit 101 in FIG. Process.
  • the inverse quantization unit 802 outputs the generated decoded frequency domain signal to the inverse orthogonal transform unit 803.
  • the inverse orthogonal transform unit 803 receives the inverse quantized decoded frequency domain signal from the inverse quantization unit 802 and performs the inverse orthogonal transform process of the orthogonal transform process performed by the orthogonal transform unit 203 of the image coding unit 101 in FIG. And the differential image (decoded differential image block signal) is decoded.
  • the inverse orthogonal transform unit 803 outputs the decoded decoded difference image block signal to the adding unit 804.
  • the adding unit 804 adds the predicted image block signal input from the image selection unit 806 to the decoded difference image block signal input from the inverse orthogonal transform unit 803, thereby generating a decoded image block signal.
  • the adding unit 804 outputs the decoded decoded image block signal to the image output unit 812, the intra-screen prediction unit 816, and the inter-screen prediction unit 815. Thereafter, the process proceeds to step S608.
  • step S608 the image output unit 812 places the decoded image block signal input from the adding unit 804 at a corresponding position in the image to generate an output image. If the processes in steps S602 to S608 have not been completed for all blocks in the frame, the block to be processed is changed and the process returns to step S602.
  • the image output unit 812 rearranges the images in the display order, aligns the viewpoint images of the same frame, and outputs them to the outside of the image decoding apparatus 700.
  • step S701 the deblocking / filtering unit 807 receives the decoded image block signal from the adding unit 804 outside the inter-screen prediction unit 815, and performs the FIR filter processing performed at the time of the encoding.
  • the deblocking filter unit 807 outputs the corrected corrected block signal to the frame memory 808. Thereafter, the process proceeds to step S702.
  • step S702 the frame memory 808 receives the correction block signal of the deblocking filter unit 807, and holds the correction block signal as a part of the image together with information that can identify the viewpoint number and the frame number. Thereafter, the process proceeds to step S703.
  • the motion / disparity compensation unit 809 receives the inter-frame predictive coding information from the prediction scheme control unit 805, and among them, the reference image information (reference viewpoint image number and reference frame number) and the difference vector (motion / motion). A difference vector between the disparity vector and the prediction vector) is extracted.
  • the motion / disparity compensation unit 809 generates a prediction vector by the same method as the prediction vector generation method performed by the motion / disparity compensation unit 213 described above.
  • the motion / disparity compensation unit 809 adds a difference vector to the calculated prediction vector to generate a motion / disparity vector.
  • the motion / disparity compensation unit 809 extracts a target image block signal (predicted image block signal) from the images stored in the frame memory 808 based on the reference image information and the motion / disparity vector.
  • the motion / disparity compensation unit 809 extracts the surrounding image block signals of the encoding target block and the reference image block, performs the above-described inter-image characteristic difference compensation processing, and the result Is output to the image selection unit 806 as an inter-screen prediction image block signal.
  • the motion / disparity compensation unit 809 When the reference vector is a motion vector, the motion / disparity compensation unit 809 outputs the predicted image block signal as it is as the inter-screen predicted image block signal to the image selection unit 806 without performing the above-described inter-image characteristic difference compensation processing. To do. Thereafter, the inter-screen prediction process ends.
  • the image decoding apparatus 700 performs the parallax compensation prediction in which the inter-image characteristic difference compensation is performed without receiving additional information for explicit inter-image characteristic difference compensation. Can do. That is, according to the present embodiment, it is possible to decode the encoded data with improved encoding efficiency as in the image encoding apparatus 100 of FIG.
  • (Embodiment 3) Part of the image encoding device 100 and the image decoding device 700 in the above-described embodiment, for example, the code configuration unit 104, the subtraction unit 202 in the image encoding unit 101, the orthogonal transform unit 203, the quantization unit 204, and the entropy code Unit 205, inverse quantization unit 206, inverse orthogonal transform unit 207, addition unit 208, prediction method control unit 209, image selection unit 210, deblocking filter unit 211, motion / disparity compensation unit 213, motion / disparity vector detection Unit 214, intra prediction unit 215, code separation unit 701, entropy decoding unit 801 in image decoding unit 702, inverse quantization unit 802, inverse orthogonal transform unit 803, addition unit 804, prediction scheme control unit 805, image
  • the selection unit 806, the deblocking filter unit 807, the motion / disparity compensation unit 809, and the intra prediction unit 810 are performed by a computer. It may be present.
  • a program for realizing this control function is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system. , May be realized by executing.
  • the “computer system” here is a computer system built in the image encoding apparatus 100 or the image decoding apparatus 700, and includes an OS and hardware such as peripheral devices.
  • the “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, and a hard disk incorporated in a computer system.
  • the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line,
  • a volatile memory inside a computer system that serves as a server or a client may be included that holds a program for a certain period of time.
  • the program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. .
  • this program is not limited to being distributed via a portable recording medium or a network, but can also be distributed via a broadcast wave.
  • the image encoding program causes a computer to execute an image encoding process that corrects a characteristic difference between viewpoint images and performs parallax compensation when encoding each viewpoint image captured from at least two viewpoints. It is a program for.
  • the image encoding process is based on the similarity between the step of extracting a reference block to be referred to when encoding the encoding target block, and the block around the encoding target block and the block around the reference block. And correcting the characteristic difference between the viewpoint images.
  • Other application examples are as described for the image encoding device.
  • the image decoding program performs an image decoding process for correcting a characteristic difference between viewpoint images and performing parallax compensation when decoding an encoded stream of each viewpoint image captured from at least two viewpoints. It is a program for making it run.
  • the image decoding process is based on the step of extracting a reference block to be referred to when decoding the decoding target block, and based on the similarity between the block around the decoding target block and the block around the reference block. And correcting a characteristic difference between images.
  • Other application examples are as described for the image decoding apparatus.
  • This image decoding program can be implemented as part of multi-viewpoint image playback software.
  • part or all of the image encoding device 100 and the image decoding device 700 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration) or an IC (Integrated Circuit) chip set.
  • LSI Large Scale Integration
  • IC Integrated Circuit
  • Each functional block of the image encoding device 100 and the image decoding device 700 may be individually made into a processor, or a part or all of them may be integrated into a processor.
  • the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor.
  • an integrated circuit based on the technology may be used.
  • the present invention further includes an image encoding method and an image encoding method, as described as the processing of each step of the image decoding program, A form as an image decoding method can also be adopted.
  • the image encoding method is a method for performing parallax compensation by correcting a difference in characteristics between viewpoint images when encoding each viewpoint image taken from at least two viewpoints, wherein the encoding target block Extracting a reference block to be referred to when encoding the image, and correcting a characteristic difference between the viewpoint images based on similarity between a block around the encoding target block and a block around the reference block And.
  • Other application examples are as described for the image encoding device.
  • the image decoding method is a method of performing parallax compensation by correcting a characteristic difference between viewpoint images when decoding an encoded stream of each viewpoint image captured from at least two viewpoints.
  • Other application examples are as described for the image decoding apparatus.
  • DESCRIPTION OF SYMBOLS 100 ... Image coding apparatus, 101 ... Image coding part, 102 ... Reference viewpoint coding process part, 103 ... Non-reference viewpoint coding process part, 104 ... Code structure part, 201 ... Image input part, 202 ... Subtraction part, DESCRIPTION OF SYMBOLS 203 ... Orthogonal transformation part, 204 ... Quantization part, 205 ... Entropy encoding part, 206 ... Dequantization part, 207 ... Inverse orthogonal transformation part, 208 ... Addition part, 209 ... Prediction scheme control part, 210 ... Image selection part , 211 ... deblocking filter unit, 212 ... frame memory, 213 ...
  • motion / disparity compensation unit 214 ... parallax vector detection unit, 215 ... intra prediction unit, 217 ... intra prediction unit, 218 ... inter prediction unit, 301 ... corresponding block extraction unit 302 ... correction coefficient calculation unit 303 ... correction processing unit 401 ... encoding target image 402 ... reference image 403 ... encoding target block 404 ... Encoding target block peripheral image block, 405 ... reference block, 406 ... reference block peripheral image block, 407 ... disparity vector, 700 ... image decoding apparatus, 701 ... code separation unit, 702 ... image decoding unit, 703 ... reference viewpoint decoding process 704... Non-reference viewpoint decoding processing unit 801... Entropy decoding unit 802.
  • Deblocking filter unit 808 Frame memory 809
  • Motion / disparity compensation unit 810 Intra prediction unit 812
  • Image output unit 813 Encoded data input unit 815
  • Inter prediction unit 816 Screen Inner prediction unit, 901 ... subject, 902 ... camera, 903 ... sensor, 906 ... encoder, 907 ... decoder, 908 ... display unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

La présente invention, dans une technique de prédiction de compensation de disparité, a pour but de réaliser un traitement de compensation de différence de caractéristiques inter-images constamment de haute précision sans nécessiter d'informations supplémentaires à part des informations requises pour la prédiction de compensation de disparité. Lors du codage de chaque image de point de vue qui est photographiée à partir d'au moins deux points de vue ou plus, ce dispositif de codage d'image corrige des différences de caractéristiques entre chaque image de point de vue et réalise une compensation de disparité. Ce dispositif de codage d'image comprend : une unité d'extraction de bloc correspondant qui extrait un bloc de référence (405) auquel il est fait référence lors du codage d'un bloc à coder (403) ; et une unité de traitement de correction qui, sur la base de la similarité entre des blocs périphériques (404) du bloc à coder (403) et des blocs périphériques (406) du bloc de référence (405), corrige des différences de caractéristiques entre chaque image de point de vue.
PCT/JP2012/080019 2011-11-21 2012-11-20 Dispositif de codage d'image, dispositif de décodage d'image et procédés et programmes correspondants Ceased WO2013077304A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2011-253617 2011-11-21
JP2011253617A JP2013110555A (ja) 2011-11-21 2011-11-21 画像符号化装置、画像復号装置、並びにそれらの方法及びプログラム

Publications (1)

Publication Number Publication Date
WO2013077304A1 true WO2013077304A1 (fr) 2013-05-30

Family

ID=48469748

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2012/080019 Ceased WO2013077304A1 (fr) 2011-11-21 2012-11-20 Dispositif de codage d'image, dispositif de décodage d'image et procédés et programmes correspondants

Country Status (2)

Country Link
JP (1) JP2013110555A (fr)
WO (1) WO2013077304A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116320402A (zh) * 2018-04-04 2023-06-23 日本放送协会 预测图像校正装置、图像编码装置、图像解码装置以及程序
CN119815012A (zh) * 2023-10-09 2025-04-11 浙江大学 一种沉浸式媒体编码中处理图像块的方法与装置

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015009041A1 (fr) * 2013-07-15 2015-01-22 삼성전자 주식회사 Procédé de codage vidéo inter-couche destiné à une compensation adaptative de luminance et appareil correspondant, et procédé de décodage vidéo et appareil correspondant

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009005280A (ja) * 2007-06-25 2009-01-08 Nippon Telegr & Teleph Corp <Ntt> 画像符号化方法，画像復号方法，画像符号化装置，画像復号装置，画像符号化プログラム，画像復号プログラムおよびコンピュータ読み取り可能な記録媒体
JP2010507336A (ja) * 2006-10-18 2010-03-04 トムソン ライセンシング 明示的な信号伝達なしでの局所的な輝度および色の補償のための方法および装置
WO2010095471A1 (fr) * 2009-02-23 2010-08-26 日本電信電話株式会社 Procédé de codage d'images multi-vues, procédé de décodage d'images multi-vues, dispositif de codage d'images multi-vues, dispositif de décodage d'images multi-vues, programme de codage d'images multi-vues et programme de décodage d'images multi-vues
JP2012080242A (ja) * 2010-09-30 2012-04-19 Sharp Corp 予測ベクトル生成方法、画像符号化方法、画像復号方法、予測ベクトル生成装置、画像符号化装置、画像復号装置、予測ベクトル生成プログラム、画像符号化プログラムおよび画像復号プログラム

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010507336A (ja) * 2006-10-18 2010-03-04 トムソン ライセンシング 明示的な信号伝達なしでの局所的な輝度および色の補償のための方法および装置
JP2010507334A (ja) * 2006-10-18 2010-03-04 トムソン ライセンシング 明示的な信号伝達なしでの局所的な輝度および色の補償のための方法および装置
JP2009005280A (ja) * 2007-06-25 2009-01-08 Nippon Telegr & Teleph Corp <Ntt> 画像符号化方法，画像復号方法，画像符号化装置，画像復号装置，画像符号化プログラム，画像復号プログラムおよびコンピュータ読み取り可能な記録媒体
WO2010095471A1 (fr) * 2009-02-23 2010-08-26 日本電信電話株式会社 Procédé de codage d'images multi-vues, procédé de décodage d'images multi-vues, dispositif de codage d'images multi-vues, dispositif de décodage d'images multi-vues, programme de codage d'images multi-vues et programme de décodage d'images multi-vues
JP2012080242A (ja) * 2010-09-30 2012-04-19 Sharp Corp 予測ベクトル生成方法、画像符号化方法、画像復号方法、予測ベクトル生成装置、画像符号化装置、画像復号装置、予測ベクトル生成プログラム、画像符号化プログラムおよび画像復号プログラム

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116320402A (zh) * 2018-04-04 2023-06-23 日本放送协会 预测图像校正装置、图像编码装置、图像解码装置以及程序
CN119815012A (zh) * 2023-10-09 2025-04-11 浙江大学 一种沉浸式媒体编码中处理图像块的方法与装置

Also Published As

Publication number Publication date
JP2013110555A (ja) 2013-06-06

Similar Documents

Publication Publication Date Title
US9363535B2 (en) Coding motion depth maps with depth range variation
JP2024105604A (ja) 多視点信号コーデック
KR101773693B1 (ko) 스킵 및 직접 모드들에 대한 3ｄ 비디오 코딩에서의 디스패리티 벡터 유도
JP5575908B2 (ja) ２ｄビデオデータの３ｄビデオデータへの変換のための深度マップ生成技法
EP3100454B1 (fr) Procédé pour processus de compensation d&#39;éclairage à faible latence
KR102819129B1 (ko) 비디오 코딩을 위해 가상 참조 영상을 이용한 인터-픽처 예측 방법 및 장치
CN102918846B (zh) 多视点视频编码方法、多视点视频解码方法、多视点视频编码装置、多视点视频解码装置
CN111819853A (zh) 变换域中预测的信令残差符号
JP6307152B2 (ja) 画像符号化装置及び方法、画像復号装置及び方法、及び、それらのプログラム
JP6039178B2 (ja) 画像符号化装置、画像復号装置、並びにそれらの方法及びプログラム
TWI499277B (zh) 多視點畫像編碼方法、多視點畫像解碼方法、多視點畫像編碼裝置、多視點畫像解碼裝置及這些程式
KR20210158843A (ko) 다시점 영상을 위한 움직임 벡터를 부호화/복호화하는 방법 및 그 장치
JPWO2014168082A1 (ja) 画像符号化方法、画像復号方法、画像符号化装置、画像復号装置、画像符号化プログラム及び画像復号プログラム
RU2571511C2 (ru) Кодирование карт глубин движения с изменением диапазона глубины
KR20140124919A (ko) 객체 기반 적응적 밝기 보상 방법 및 장치
WO2013077304A1 (fr) Dispositif de codage d&#39;image, dispositif de décodage d&#39;image et procédés et programmes correspondants
US20160255370A1 (en) Moving image encoding method, moving image decoding method, moving image encoding apparatus, moving image decoding apparatus, moving image encoding program, and moving image decoding program
WO2012161318A1 (fr) Dispositif de codage d&#39;image, dispositif de décodage d&#39;image, procédé de codage d&#39;image, procédé de décodage d&#39;image et programme
JP5706291B2 (ja) 映像符号化方法，映像復号方法，映像符号化装置，映像復号装置およびそれらのプログラム
CN119384827B (zh) 对边界外条件下双向预测的自适应约束的方法及装置
JP6232117B2 (ja) 画像符号化方法、画像復号方法、及び記録媒体
KR20140124045A (ko) 객체 기반 적응적 밝기 보상 방법 및 장치
JP2013179554A (ja) 画像符号化装置、画像復号装置、画像符号化方法、画像復号方法およびプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12851953

Country of ref document: EP

Kind code of ref document: A1

DPE1 Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101)
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12851953

Country of ref document: EP

Kind code of ref document: A1