WO2023037575A1 - Image processing device and image processing method - Google Patents

Abstract

The purpose of the present invention is to provide an image processing device that uses a plurality of cameras, wherein high precision parallax is generated through a neural network by reducing the amount of computation. This image processing device comprises: an image reduction unit that reduces two input images and generates two reduced images; a first parallax image generation unit that determines the parallax of the two reduced images through neural network processing, and generates a reduced parallax image; and a second parallax image generation unit that determines, from one of the input images, a region that matches a region of a portion of the other input image, and thereby determines the parallax of the two input images and generates a parallax image. With regard to each region of the input images for which pixels of the reduced parallax image correspond, the second parallax image generation unit sets, on the basis of a parallax value of the pixels of the reduced parallax image, a search range for use when determining the matching region.

Description

Image processing device and image processing method

The present invention relates to an image processing device and an image processing method for recognizing obstacles outside the vehicle using a plurality of cameras.

In order to improve the driving safety of the vehicle, an on-vehicle forward monitoring sensor detects obstacles in front of the vehicle, and if there is a possibility of the vehicle colliding with the obstacle, the driver is warned and automatic braking is applied. There is a system to hang.

Forward monitoring sensors used in such systems include millimeter-wave radar, laser radar, and cameras. Camera types include a monocular camera and a stereo camera using multiple cameras. The stereo camera can measure the distance to the photographed object by using the parallax of the overlapping area photographed by two cameras spaced at a predetermined distance. Therefore, it is possible to accurately grasp the degree of risk of collision with an object in front.

The stereo camera obtains the parallax of the images shot by the two cameras and converts the parallax into distance. There is a feature that the parallax becomes smaller as the measurement distance becomes farther. As a parallax calculation method, a method of checking correspondence between left and right images by block matching is known.

However, the parallax calculation method of the block matching method has the problem that the effective parallax decreases in images with little texture and the accuracy of the parallax decreases. Methods for computing parallax have been developed.

For example, in the abstract of Patent Document 1, the problem is described as "to provide a technology capable of correcting the parallax value in the false matching area in the distance image", and the solution is "the distance mounted on the vehicle 10 The image generation device 110 uses the left and right captured images captured by the stereo camera 122 as a reference image and a comparison image to generate a distance image representing the distance to an object present in the reference image; A contrast image generation unit 112 that uses a neural network to generate a contrast image to be compared with the distance image, and a correction region in the distance image corresponding to an area in the reference image in which the image feature amount is equal to or less than a predetermined threshold. and a correction unit 114 that corrects the distance information of each pixel of the correction area according to the information of the corresponding part of the correction area in the control image." . That is, Patent Document 1 proposes a method of applying parallax calculation of a neural network method to a portion of an image (a portion having a small feature amount such as a road surface on which the vehicle is traveling).

JP 2020-143957 A

The neural network method of Patent Document 1 improves the problem of the block matching method and can calculate effective parallax with high accuracy even from images with little texture, but it has the problem of an enormous amount of calculation.

In an image processing apparatus using a plurality of cameras, the present invention calculates parallax by a neural network by reducing the amount of calculation, and based on the parallax, increases the effective parallax of the entire image by block matching, and creates a highly accurate parallax image. The purpose is to generate the entire input image.

In order to solve the above-mentioned problems, the present invention provides an image reduction unit that reduces two input images to generate two reduced images, and obtains parallax between the two reduced images by neural network processing to generate reduced parallax images. A first parallax image generating unit and a first parallax image generation unit that obtains a parallax between the two input images by obtaining a region matching a partial region of one of the input images from the other input image, thereby generating a parallax image. A two-parallax image generation unit is provided, and the second parallax image generation unit generates a parallax value of each pixel of the reduced parallax image for each region of the input image corresponding to each pixel of the reduced parallax image. and set the search range for finding the matching area.

According to the present invention, an effective parallax is calculated from a reduced image by a neural network, and a block matching method is used to calculate the parallax of an image before reduction at a position corresponding to the effective parallax so that it is equal to or approximates the effective parallax. Therefore, it is possible to increase the effective parallax of the image before reduction while reducing the computation amount of the neural network.

Functional block diagram of the image processing apparatus of the first embodiment 4 is a flow chart of image processing of the image processing apparatus according to the first embodiment; Diagram explaining the parallax calculation method of the block matching method FIG. 4 is a diagram for explaining an example of a method for determining a search range for block matching in the first embodiment; FIG. 4 is a diagram for explaining another example of a method for determining a search range for block matching in the first embodiment; An example of an image with large feature values Functional block diagram of the image processing apparatus of the second embodiment Flowchart of road surface height estimation processing of the image processing apparatus of the second embodiment Functional block diagram of the image processing apparatus of the third embodiment Flowchart of first parallax image generation processing according to the fourth embodiment Flowchart of Second Parallax Image Generation Processing of Embodiment 5

Hereinafter, embodiments of the present invention will be described using drawings and the like. The following description shows specific examples of the contents of the present invention, and the present invention is not limited to these descriptions. Various changes and modifications are possible. In addition, in all the drawings for explaining the present invention, parts having the same functions are denoted by the same reference numerals, and repeated explanations thereof may be omitted.

FIG. 1 is a functional block diagram of an image processing apparatus 100 according to Embodiment 1 of the present invention. As shown here, the image processing device 100 includes a right camera 1R, a left camera 1L, an image reduction unit 2, a first parallax image generation unit 3, a search range determination unit 4, a second parallax image generation unit 5, and a recognition processing unit. 6. The right camera 1R and the left camera 1L are not necessarily the left and right cameras of a stereo camera, and a pair of monocular cameras arranged on the left and right may be used as the right camera 1R and the left camera 1L, respectively. Further, the image reduction unit 2 to the recognition processing unit 6 are, for example, a computer having an arithmetic device such as a CPU, a storage device such as a semiconductor memory, and hardware such as a communication device. Although the functional units are implemented by executing programs, the details of each unit will be described below while omitting such well-known techniques in the computer field as appropriate.

As shown in FIG. 1, images captured by the right camera 1R and the left camera 1L are transferred to the image reduction unit 2 as a right image _PR and a left image _PL , respectively. The image reduction unit 2 reduces the input left and right images to generate a right reduced image _SR and a left reduced image _SL . The reason for reducing the image is to reduce the amount of calculation in the first parallax image generator 3, and the reduction ratio is determined according to the processing time of the first parallax image generator 3 allowed by the system. The first parallax image generator 3 receives the reduced right image _SR and the reduced left image _SL and generates a first parallax image D1 (reduced parallax image) and a feature map M through neural network processing.

The search range determination unit 4 uses the first parallax image D1 and the feature map M to determine the search range of the second parallax image generation unit 5 . The second parallax image generator 5 generates a second parallax image D2 from two images, the right image _PR and the left image _PL , by a block matching method. obey. The second parallax image D2 is a parallax image with a higher density than the first parallax image D1, since the parallax is generated from the image before being reduced. The recognition processing unit 6 uses the second parallax image D2 and the right image _PR to perform recognition processing of forward vehicles, pedestrians, obstacles, and the like. Then, the recognition result of the recognition processing unit 6 is input to an ECU (not shown) that controls the driving system, braking system, steering system, etc. of the vehicle, so that the driver can be warned or automatically controlled as necessary. You can apply the brakes.

FIG. 2 is a flowchart for explaining an outline of image processing of the image processing apparatus 100 of this embodiment.

First, in step S10, the image reduction unit 2 reduces the right image P _R and the left image P _L input from the left and right cameras to generate a right reduced image S _R and a left reduced image _SL .

Next, in step S11, the first parallax image generator 3 receives the reduced right image _SR and the reduced left image _SL and generates a first parallax image D1 and a feature map M through neural network processing.

In step S12, the search range determination unit 4 uses the first parallax image D1 or the first parallax image D1 and the feature map M to determine the search range of the second parallax image generation unit 5. . Details of the method of using the first parallax image D1 will be described with reference to FIG. 4, and details of the method of using the first parallax image D1 and the feature map M will be described with reference to FIG.

In step S13, the second parallax image generator 5 generates a second parallax image D2 from the two images, the right image _PR and the left image _PL , by a block matching method based on the search range determined by the search range determiner 4. do. Details of the block matching method will be described with reference to FIG.

In step S14, the recognition processing section 6 uses the second parallax image D2 and the right image _PR to perform recognition processing of forward vehicles, pedestrians, obstacles, and the like.

FIG. 3 illustrates parallax calculation processing by a general block matching method. In this example, the right image _PR captured by the right camera 1R is used as a reference image, and a reference block image _PB such as 16 pixels×16 pixels (the size is not limited to this example) is defined. Then, in the left image _PL photographed by the left camera 1L, a predetermined search width r (for example, 272 pixels) is set with reference to the same vertical position (Y coordinate) and horizontal position (X coordinate) as the reference block image _PB . , select the reference image P _ref of . After that, the difference between the base block image _PB and the reference image P _ref is calculated. This difference calculation is called SAD, and is calculated according to the following equation 1.

In Equation 1, I is an image block (eg, 16×16 pixels) in the reference image P _ref , T is an image block in the reference block image _PB , and i and j are coordinates within the image block. . In order to calculate one parallax, the reference position of _the reference image P _ref is shifted from the left end by one pixel and compared with all images of the search width r. If the width r is 272 pixels, the block image comparison operation is performed 256 times to search for the position where the SAD value is the smallest.

In the actual scenery seen from the front window of the vehicle in FIG. 3, there is a forward vehicle V running on the road surface. When there are a right image PR captured by the right camera _1R and a left image _PL captured by the left camera 1L, a part V1 of the forward vehicle _V is captured at the position of the reference block image _PB in the right image PR. , and the left image P _L is photographed at the position of the reference block image P _B '. As a result, the SAD values of the base block image _PB and the reference block image _PB ' are minimized at the position of parallax d. This parallax d has a large value when the preceding vehicle V is close to the image processing device 100, and has a small value when it is far away. The parallax obtained in this manner is obtained for the entire image. Using this parallax d, the distance to the image processing apparatus 100 can be measured according to the principle of triangulation. The distance Z is calculated from the parallax d by the following equation 2.

However, in Equation 2, f is the focal length of the left and right cameras, and B is the distance (baseline length) between the right camera 1R and the left camera 1L.

<Example of Detailed Operations of Search Range Determination Unit 4 and Second Parallax Image Generation Unit 5>
FIG. 4 shows an example of detailed operations of the search range determination unit 4 and the second parallax image generation unit 5. At step S12 in FIG. corresponds to the flow that determines

As shown in FIG. 4, the right image P _R and the left image P _L are given X coordinates from left to right in the horizontal direction and Y coordinates from top to bottom in the vertical direction. The right image _PR and the left image _PL are reduced by the image reducing unit 2 to become a right reduced image _SR and a left reduced image _SL . Here, as an example, the reduction ratio is set to 1/4. Next, the first parallax image generator 3 receives the reduced right image _SR and the reduced left image _SL and generates a first parallax image D1 through neural network processing. In this example, since the image reduction magnification is 1/4, the pixel with the parallax value A in the first parallax image D1 is located at X coordinate=0 and Y coordinate=0, and X coordinate=0 and Y coordinate in the right image _PR . =1, X-coordinate=1 and Y-coordinate=0, X-coordinate=1 and Y-coordinate=1.

The search range determination unit 4 determines the block matching range of the second parallax image generation unit 5 based on the first parallax image D1. For example, the second parallax image generation unit 5 sets X coordinate=0 and Y coordinate=0, X coordinate=0 and Y coordinate=1, X coordinate=1 and Y coordinate=0, and X coordinate=1 of the right image _PR . When generating the parallax of four pixels of Y coordinate=1, the search range is determined based on A, which is the parallax value of X coordinate=0 and Y coordinate=0 of the corresponding first parallax image D1. Specifically, the start position s of the search range is obtained by subtracting a fixed value from the parallax value A by the subtractor 41, and the end position e of the search range is obtained by adding the fixed value to the parallax value A by the adder . Let the fixed value be 10, for example. As a result, the four pixels (A1, A2, A3, A4) of the second parallax image D2 corresponding to the position of the parallax value A are the effective parallax values because the block matching search range is limited to the vicinity of the effective parallax value. The parallax value is close to or the same as the parallax value A, and there is a high possibility that it will be an effective parallax value. Also, the number of block matching operations for calculating the parallax of one pixel is only 20. In a general block matching method (FIG. 3), the search range is fixed and, for example, 256 SAD calculations are required.

<Other Examples of Detailed Operations of Search Range Determination Unit 4 and Second Parallax Image Generation Unit 5>
FIG. 5 shows another example of detailed operations of the search range determination unit 4 and the second parallax image generation unit 5. In step S12 of FIG. This corresponds to the flow for determining the search range for block matching. Differences from FIG. 4 will be described below.

The first parallax image generation unit 3 calculates parallax by neural network calculation. The neural network performs convolution operation processing, and a feature map M is generated as a result. A feature amount tends to be large in a portion of an image in which the change in pixel value is large. For example, as shown in FIG. 6, the image area R1 is an image of only the road surface, so the feature amount is small, and the distance in this area is almost the same, so the parallax value is also an approximate value. On the other hand, the image region R2 includes the road surface, a portion of the preceding vehicle V, and a traffic sign, and a plurality of objects overlap, resulting in a large feature amount. Since the forward vehicle V, the road surface, and the traffic sign in the image area R2 have different distances and different parallaxes, it is necessary to prevent the parallax in the image area R2 from being an approximate value.

In FIG. 4, the four pixels (pixel values A1, A2, A3, A4) of the second parallax image D2 corresponding to the pixel value A of the first parallax image D1 are all approximate values of the parallax value A. Therefore, if there is a boundary between a distant object and a near object among these four pixels, there is a possibility that the parallax value A is not an approximation.

On the other hand, in FIG. 5, a subtractor 43 and an adder 44 for the feature map M, which have the same functions as those of the subtractor 41 and the adder 42 for the first parallax image D1, are provided so that the feature map M shows In areas where the feature amount is small, the search range is widened so that the correct parallax can be calculated. Specifically, when the second parallax image generation unit 5 is processing the portion corresponding to the pixel of A in the first parallax image D1, the value of the i pixel of the feature map M is set to the starting position s of the search range. and add to the end position e. When processing the portion corresponding to the B pixel of the first parallax image D1, the pixel v of the feature map M is used. By doing so, it is possible to expand the search range for portions where the feature amount is small, and it becomes possible to calculate correct parallax.

According to the image processing apparatus of the present embodiment described above, the effective parallax is calculated from the reduced image by the neural network, and the parallax of the image before reduction at the position corresponding to the effective parallax is the same value as the effective parallax or an approximate value. Since calculation is performed by the block matching method so that , the effective parallax of the image before reduction can be increased while reducing the amount of calculation of the neural network.

Next, the image processing apparatus of Example 2 will be described with reference to FIGS. 7 and 8. FIG. It should be noted that redundant description of the points in common with the first embodiment will be omitted.

FIG. 7 is a functional block diagram of the image processing apparatus 100 according to the second embodiment, which differs from the first embodiment in that the first parallax image D1 is used when the recognition processing unit 6 recognizes the road surface. When the recognition processing unit 6 detects a forward vehicle V or an obstacle, it is necessary to determine whether they exist on the road surface. For example, when the first parallax image generation and the second parallax image generation are processed by hardware, and the recognition processing is processed by software, the recognition processing of the road surface (software processing) is the generation processing of the second parallax image D2 (hardware processing). processing) can be performed in parallel, and the processing speed of the entire system can be increased.

FIG. 8 shows a flowchart when the recognition processing unit 6 uses the first parallax image D1 when recognizing the height of the road surface. First, after generating the first parallax image D1 (step S20), the recognition processing unit 6 performs recognition processing of the height of the road surface using the first parallax image D1 (step S21), and concurrently generates the second parallax image. A second parallax image D2 is generated in the unit 5 (step S22). Thereafter, the recognition processing unit 6 uses the second parallax image D2 to detect the forward vehicle V and obstacles on the road (step S23). In this case, compared to the case where the recognition processing unit 6 performs all processing using the second parallax image D2, the processing speed of the entire system can be increased.

Next, the image processing apparatus of Example 3 will be described using FIG. It should be noted that redundant description of the points in common with the above embodiment will be omitted.

FIG. 9 is a functional block diagram of an image processing apparatus 100 according to the third embodiment, which is different from the first embodiment in that a parallax accuracy reduction area detection unit 7 is added. The parallax accuracy reduction area detection unit 7 discriminates an image area in which the first parallax image generation unit 3 may not generate a correct parallax. D1 and part of the feature map M are invalidated. For example, when the wiper is reflected in part of the right image _PR and the left image _PL during rainy weather, an accurate parallax cannot be calculated from the image of the wiper portion. The first parallax image generator 3 is notified to output invalid parallax for such a portion. As a detection method of the parallax accuracy reduction area detection unit 7, there is a method of determining a state in which the luminance value of a part of the image is extremely darker than that of the other part.

Next, the image processing apparatus of Example 4 will be described using FIG. It should be noted that redundant description of the points in common with the above embodiment will be omitted.

FIG. 10 shows a flowchart of a method for changing the reduction magnification of the image reduction unit 2 according to the processing position of the image. For example, in the case of an image of an actual landscape as shown in Fig. 3, the lower part of the road surface image is closer to the camera than the upper part of the road surface image, so the road width is wider and the object is larger. be filmed. Therefore, it is conceivable that the parallax can be calculated without any problem even if the magnification of the reduction is small on the lower side of the image.

In the flowchart of FIG. 10, the photographed image is vertically divided into two parts, and the image is reduced by changing the reduction ratio according to the Y-coordinate value of the two parts, and the processing of the first parallax image generation unit 3 is performed. Specifically, the Y coordinates of the left and right images are checked (step S40). Set (step S41). On the other hand, if the Y coordinate is smaller than the predetermined value (on the upper side of the image where there is a high possibility that the image is taken far away), the reduction ratio is set to 1/4 (step S42). Then, the image reduction unit 2 generates reduced images at the reduction ratios set in steps S41 and S42 (step S43), and the first parallax image generation unit 3 uses the left and right reduced images with different reduction ratios for the upper and lower sides. A one-parallax image D1 and a feature map M are generated. By doing so, the amount of calculation can be reduced by the amount of reduction in the reduction ratio of the image.

Next, the image processing apparatus of Example 5 will be described using FIG. It should be noted that redundant description of the points in common with the above embodiment will be omitted.

FIG. 11 is a flowchart for changing the fixed value that determines the search range according to the processing position of the image. As described above, the lower side of the road surface image is photographed in the near area, and the upper side of the road surface image is photographed in the distant area. As shown in Equation 2, since parallax is inversely proportional to distance, the difference of 1 in the parallax value is about several tens of centimeters of distance at the short distance, but the difference of 1 in the far distance is the number of distances. The difference is m. Therefore, in order to bring the accuracy of the second parallax image D2 close to that of the first parallax image D1, it is desirable to change the search range according to the distance. Therefore, when processing the image of the road surface, when the Y coordinate is smaller than the predetermined Y coordinate, the fixed value is changed to 5 to determine the search range, and the second parallax image generation unit 5 generates the second parallax image D2. Generate. Specifically, the Y coordinates of the left and right images are checked (step S50), and if the Y coordinates are equal to or greater than a predetermined value (at the lower side of the image where the vicinity is likely to be photographed), the fixed value is set to 10. (Step S51). On the other hand, if the Y coordinate is smaller than the predetermined value (on the upper side of the image where there is a high possibility that the image is taken far away), the fixed value is set to 5 (step S52). Then, the search range determination unit 4 determines the search range using the fixed values set in steps S51 and S52 (step S53), and the second parallax image generation unit 5 uses search regions having different widths in the upper and lower directions to obtain the second parallax image. Generate image D2. By doing so, the amount of calculation can be reduced by the amount of the reduced width of the search range.

It should be noted that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. In addition, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

100: image processing device, 1L: left camera, 1R: right camera, 2: image reduction unit, 3: first parallax image generation unit, 4: search range determination unit, 41, 43: subtractor, 42, 44: addition device, 5: second parallax image generator, 51: SAD calculator, 6: recognition processor, 7: parallax accuracy reduced area detector, _PL : left image, _PR : right image, _PB : reference reference block Image, P _B ': reference block image, S _R : right reduced image, S _L : left reduced image, D1: first parallax image, D2: second parallax image, M: feature map

Claims (9)

an image reduction unit that reduces two input images to generate two reduced images;
a first parallax image generation unit that obtains parallax between the two reduced images by neural network processing and generates a reduced parallax image;
a second parallax image generation unit that generates a parallax image by obtaining a parallax between the two input images by obtaining a region matching a partial region of one of the input images from the other input image; ,
The second parallax image generation unit obtains the matching region based on the parallax value of each pixel of the reduced parallax image for each region of the input image corresponding to each pixel of the reduced parallax image. An image processing apparatus characterized by setting a search range for each time. The image processing device according to claim 1,
The image processing device, wherein the second parallax image generation unit generates a parallax equal to or approximate to the reduced parallax image at the corresponding position. The image processing device according to claim 1,
The image processing apparatus, wherein the second parallax image generation unit performs matching processing within the search range. The image processing device according to claim 1,
The image processing apparatus according to claim 1, wherein the search range is widened for pixels having a large feature amount in the feature map output by the first parallax image generation unit. The image processing device according to claim 1,
An image processing apparatus, wherein recognition processing of the height of a road surface is performed using the reduced parallax image, and an object on the road surface is recognized based on the result of the second parallax image generation unit. The image processing device according to claim 1,
A parallax accuracy reduction area detection unit that determines from the reduced image that there is an area that cannot be calculated in a part of the parallax calculated by the first parallax image generation unit, and the image area detected by the parallax accuracy reduction area detection unit. An image processing apparatus characterized by indicating that parallax is invalidated. The image processing device according to claim 1,
1. An image processing apparatus, wherein when the input image is divided into upper and lower parts by the road surface portion and processed, the reduction ratio of the upper image is different from the reduction ratio of the lower image. The image processing device according to claim 1,
The image processing device, wherein the search range of the second parallax image generation unit is larger in the lower search range than in the upper search range when the input image is divided into upper and lower parts by the road surface portion for processing. . a first step of reducing two input images to generate two reduced images;
a second step of obtaining a parallax between the two reduced images by neural network processing and generating a reduced parallax image;
A third step of obtaining a parallax between the two input images by obtaining a region matching a partial region of one of the input images from the other input image, and generating a parallax image;
In the third step, for each region of the input image corresponding to each pixel of the reduced parallax image, a search is performed for obtaining the matching region based on the parallax value of each pixel of the reduced parallax image. An image processing method characterized by setting a range.

Images