WO2020054260A1 - Image recognition device - Google Patents

Abstract

There is a problem in that recognition performance degrades when the entirety of a solid object cannot be adequately sensed. In the present invention, a solid object region setting process 401 is carried out to set a solid object region 501 through enlargement or reduction of a sensing region 301 in a solid object obtained through a sensing process 208 using sensing characteristic information for the solid object. When the solid object region 501 set through the solid object region setting process 401 is used for a reference size in a recognition process 209, the reference size may not necessarily be an optimal recognition region for recognition. As such, in a recognition magnification setting process 402, the recognition region is corrected using recognition characteristic information for the solid object. In a scan region setting process 403, positioning characteristic information is used to set, for a recognition region, a scan region that is larger than the recognition region.

Description

Image recognition device

The present invention relates to an image recognition device.

In recent years, there has been an increasing demand for improved performance of image recognition devices required for driving support, automatic driving, and the like. For example, in the collision safety function for pedestrians, performance improvement is required, for example, a collision safety test for pedestrians at night is added in an automobile assessment. In order to realize this, high recognition performance for three-dimensional objects such as pedestrians is required.

Patent Literature 1 discloses that in a situation where a certain moving three-dimensional object and another three-dimensional object overlap, by tracking a feature point inside a predetermined area including the three-dimensional object, a pedestrian or the like existing inside the area is tracked. A recognition device that detects a moving three-dimensional object has been proposed.

JP-A-2017-142760

However, the conventional apparatus has a problem that the recognition performance is reduced when the entire three-dimensional object cannot be detected properly.

An image recognition device according to a first aspect of the present invention provides a three-dimensional object detection area based on the three-dimensional object detection characteristic information for a three-dimensional object detection area set on an image captured by an imaging unit. A three-dimensional object region setting unit that sets a three-dimensional object region by enlarging or reducing the three-dimensional object region, and a recognition process that performs a recognition process of specifying the type of the three-dimensional object with respect to the three-dimensional object region set by the three-dimensional object region setting unit Unit.
An image recognition device according to a second aspect of the present invention provides a three-dimensional object detection area based on first three-dimensional object characteristic information for a three-dimensional object detection area set on an image captured by an imaging unit. A three-dimensional object region setting unit that sets a three-dimensional object region by enlarging or reducing a detection region, and using the three-dimensional object region obtained by the three-dimensional object region setting unit as a reference size, based on second characteristic information of the three-dimensional object. A recognition magnification setting unit that defines a recognition area of a plurality of sizes; and a plurality of the recognition areas determined by the recognition magnification setting unit, based on third characteristic information of the three-dimensional object. A scanning area setting unit configured to set a plurality of wide scanning areas; and a recognition processing unit configured to perform a recognition process using the scanning area set by the scanning area setting unit.

According to the present invention, it is possible to provide an image recognition apparatus that accurately detects a three-dimensional object and improves recognition performance.

It is a block diagram showing the whole composition of an image recognition device. 6 is a flowchart illustrating an operation of the image recognition device. FIG. 9 is a diagram illustrating a three-dimensional object area set on an image by a detection process. It is a flowchart which shows the detail of a recognition process. It is a figure explaining the principle of solid object field setting processing. It is a figure explaining the principle of recognition magnification setting processing. FIG. 9 is a diagram illustrating normalization in a recognition magnification setting process. FIG. 4 is a diagram illustrating the principle of a scanning area setting process. It is a figure explaining the principle of magnification-specific scan recognition processing. FIG. 4 is a diagram for explaining the principle of an optimum magnification setting process. It is a figure explaining the principle of detailed recognition position deciding processing. FIG. 9 is a diagram for explaining the principle of the detailed recognition process. FIG. 11 is a block diagram illustrating an overall configuration of an image recognition device according to a modification.

FIG. 1 is a block diagram showing the overall configuration of an image recognition device 100 according to the present embodiment. The image recognition device 100 includes a left camera 101 and a right camera 102 mounted on a vehicle and arranged on the left and right in front of the vehicle. The cameras 101 and 102 constitute a stereo camera, and capture an image of a three-dimensional object such as a pedestrian, a vehicle, a signal, a sign, a white line, a tail lamp of a car, and a headlight. The image recognition device 100 recognizes the environment outside the vehicle based on image information of the front of the vehicle captured by the cameras 101 and 102. Then, the vehicle (own vehicle) controls braking, steering, and the like based on the recognition result by the image recognition device 100.

(4) The image recognition device 100 captures images captured by the cameras 101 and 102 from the image input interface 103. The image information captured from the image input interface 103 is sent to the image processing unit 104 via the internal bus 109. Then, the image data is processed by the arithmetic processing unit 105, and image information of the result in the middle of the processing and the final result are stored in the storage unit 106.

The image processing unit 104 compares the first image obtained from the image sensor of the left camera 101 with the second image obtained from the image sensor of the right camera 102, and compares each image with the image sensor. Correction of the device-specific deviation due to the correction and image correction such as noise interpolation are performed and stored in the storage unit 106. Further, between the first image and the second image, mutually corresponding portions are calculated to obtain disparity information, and this is stored in the storage unit 106 as distance information corresponding to each pixel on the image. I do. The image processing unit 104 is connected to an arithmetic processing unit 105, a CAN interface 107, and a control processing unit 108 via an internal bus 109.

The arithmetic processing unit 105 uses the image information and the distance information (disparity information) stored in the storage unit 106 to recognize a three-dimensional object in order to grasp the environment around the vehicle. A part of the recognition result of the three-dimensional object and a part of the intermediate processing result are stored in the storage unit 106. The arithmetic processing unit 105 performs a vehicle control calculation using the recognition result after performing recognition of a three-dimensional object on the captured image. A control policy of the vehicle obtained as a result of the vehicle control calculation and a part of the recognition result are transmitted to the in-vehicle network CAN 110 via the CAN interface 107, whereby the vehicle is controlled.

(4) The control processing unit 108 monitors whether each processing unit has performed an abnormal operation, whether an error has occurred during data transfer, and the like, and prevents the abnormal operation. The image processing unit 104, the arithmetic processing unit 105, and the control processing unit 108 may be configured by a single or a plurality of computer units.

FIG. 2 is a flowchart showing the operation of the image recognition device 100.
An image is captured by the left camera 101 and the right camera 102 provided in the image recognition apparatus 100, and the captured image information 203 and 204 are images such as corrections for absorbing a peculiar habit of the image sensor. Step 205 is performed. The processing result of the image processing 205 is stored in the image buffer 206. The image buffer 206 is provided in the storage unit 106 in FIG.

Next, parallax processing 207 is performed. Specifically, the images are collated using the two corrected images, and thereby the disparity information of the images obtained by the left camera 101 and the right camera 102 is obtained. From the parallax between the left and right images, a certain point of interest on the image of the three-dimensional object is obtained as the distance to the three-dimensional object according to the principle of triangulation. The image processing 205 and the parallax processing 207 are performed by the image processing unit 104 in FIG. 1, and the finally obtained image information and parallax information are stored in the storage unit 106.

Then, in the next detection processing 208, a three-dimensional object in a three-dimensional space is detected using the parallax information obtained by obtaining the parallax or distance of each pixel of the left and right images by the parallax processing 207. FIG. 3 is a diagram illustrating a detection area of a three-dimensional object set on an image by the detection processing 208. FIG. 3 shows a pedestrian detection area 301 and a vehicle detection area 302 detected by the cameras 101 and 102 on the image as a result of the detection processing 208. These detection areas 301 and 302 may be rectangular as shown in FIG. 3, or may be irregular areas obtained from parallax and distance. It is generally treated as a rectangle to facilitate handling on a computer in subsequent processing. Hereinafter, in the present embodiment, the region will be described as a rectangle, and a pedestrian will be mainly described as an example of a three-dimensional object.

Next, in the recognition process 209, a recognition process for specifying the type of the three-dimensional object is performed on the detection area set on the image by the detection process 208. The three-dimensional object to be recognized by the recognition process 209 is, for example, a pedestrian, a vehicle, a signal, a sign, a white line, a tail lamp or a headlight of the vehicle, and the type is identified as any of these. In order for the recognition process 209 to stably recognize a three-dimensional object, the detection region on the image and the target region to be recognized need to match. However, in the cameras 101 and 102, it may not be possible to completely match the regions on the image to be recognized due to the brightness of the external environment, the variation in the imaging performance between the cameras, and the like. This is the same even when a radar such as a millimeter wave is combined with an image sensor such as a camera. The details of the recognition processing 209 that solves this problem will be described later.

Next, in the vehicle control process 210, in consideration of the recognition result of the three-dimensional object and the state (speed, steering angle, etc.) of the own vehicle, for example, a warning is issued to an occupant, and braking and steering angle adjustment of the own vehicle are performed. And so on. Alternatively, avoidance control for the recognized three-dimensional object is determined, and the result is output via the CAN interface 107 as automatic control information. The recognition processing 209 and the vehicle control processing 210 are performed by the arithmetic processing unit 105 in FIG.

Note that the programs shown in the flowchart of FIG. 2 and the flowchart of FIG. 4 described below can be executed by a computer including a CPU, a memory, and the like. All or part of the processing may be realized by a hard logic circuit. Further, this program can be provided by being stored in a storage medium of the image recognition device 100 in advance. Alternatively, the program may be stored and provided in an independent storage medium, or the program may be recorded and stored in the storage medium of the image recognition device 100 via a network line. Various forms of computer-readable computer program products, such as data signals (carrier waves), may be provided.

FIG. 4 is a flowchart showing details of the recognition process 209. As shown in FIG. 4, the flowchart includes a three-dimensional object area setting process 401, a recognition magnification setting process 402, a scanning area setting process 403, a magnification-based scan recognition process 404, an optimum magnification setting process 406, a detailed recognition position determination process 407, A detailed recognition process 408 is performed. Hereinafter, each process will be described in order. Note that these processes will be described on the assumption that a stereo camera is used.

[Three-dimensional object area setting process]
In the three-dimensional object area setting processing 401, the three-dimensional object detection area 301 obtained by the detection processing 208 is enlarged or reduced based on the three-dimensional object detection characteristic information to set a three-dimensional object area 501.

FIG. 5 is a diagram for explaining the principle of the three-dimensional object area setting processing 401. FIG. 5 shows an example in which the three-dimensional object area 501 is set by enlarging the pedestrian detection area 301 based on the three-dimensional object detection characteristic information. The detection characteristic information includes, for example, (1) the identifiability of the three-dimensional object, (2) the distance from the three-dimensional object, (3) the size of the three-dimensional object, (4) the assumed size of the three-dimensional object, and (5) the brightness of the outside environment. (6) Headlight orientation, (7) Road surface height where a three-dimensional object exists, (8) Sensor resolution, and the like. Hereinafter, such detection characteristic information will be described.

(1) Regarding the distinguishability of a three-dimensional object, for example, in the cameras 101 and 102, a case where a three-dimensional object is difficult to obtain due to a combination with a background area may be considered. The pedestrian's clothes of the same color as the road surface and the top of the pedestrian at night correspond to this. It is also conceivable that the cameras 101 and 102 are partially blurred due to the influence of raindrops or the like and lack a three-dimensional object region. In such a case, the detection area is enlarged. In addition, a three-dimensional object may be formed by combining a person and a region other than the person in the three-dimensional space. This is because it is difficult to separate a three-dimensional object such as a telephone pole or fence on the roadside from a person before identification. In such a case, the detection area is reduced based on the color, luminance, and edge of the image. In addition, if the horizontal mounting height is different between different types of sensors such as a radar sensor and the cameras 101 and 102, hiding mainly occurs in the upper direction, and a three-dimensional object appears small. If the processing characteristic has such a configuration, the detection area is enlarged.

(2) If the distance from the three-dimensional object is large, the three-dimensional object region is enlarged if it is far, and small if the distance is short. This magnification may be determined by the resolution of the sensor including the cameras 101 and 102. This is because the farther the object goes, the larger the size of one pixel occupying the three-dimensional space becomes, and the more the error gets.

(3) As for the size of the three-dimensional object, the three-dimensional object region is enlarged if the three-dimensional object is small, and the three-dimensional object region is reduced if the three-dimensional object is large.

(4) The assumed size of the three-dimensional object is, for example, assuming that the three-dimensional object is a pedestrian, a three-dimensional object that is too small assuming a pedestrian enlarges the three-dimensional object region, and is too large assuming a pedestrian. Reduces the three-dimensional object area. The degree to which the object is targeted may be determined in consideration of (5) the brightness of the outside environment and (6) the direction of the headlight. For example, the three-dimensional object area is reduced in a bright environment in the daytime, and the three-dimensional object area is expanded in a dark environment at night. In addition, according to the direction of the headlight, for example, if the headlight is in the low beam direction, the light hits the feet, and the three-dimensional object area is enlarged in the height direction. If the direction of the headlight is a high beam, the entire body is illuminated, so that the three-dimensional object area is reduced. The three-dimensional object area may be enlarged or reduced depending on the distance to the three-dimensional object or (7) the height of the road surface at the position where the three-dimensional object exists. For example, when the height of the road surface is low, if the beam is a high beam, the light does not hit the feet, so that the three-dimensional object area is enlarged in the downward direction.

(8) If the sensors are the cameras 101 and 102, the size per pixel changes according to the distance. Therefore, the sensor resolution is enlarged or reduced by combining the size of the three-dimensional object and the distance of the object. I do. For example, when a three-dimensional object is in the vicinity, the resolution of the three-dimensional space per pixel is high, so that the three-dimensional object region is enlarged. When the three-dimensional object is distant, the resolution of the three-dimensional space per pixel is low, so that the three-dimensional object region is reduced. Further, the three-dimensional object area is reduced depending on the characteristics of the area acquired as the three-dimensional object area. The characteristic of the region acquired as the three-dimensional object region is, for example, a case where the region having the parallax and the sensor response region where the three-dimensional object region is obtained are set to be larger. In such a case, the three-dimensional object region is reduced.

An example in which the three-dimensional object area 501 is set based on the three-dimensional object detection characteristic information will be described. For example, in a pedestrian, a limb region having a large change on an image is likely to be missing and smaller. Also, in the case of nighttime, if the person has black hair, it is difficult to detect the head because it is mixed with the background. In such a case, the three-dimensional object area 501 on the image is changed based on the size of one pixel in the three-dimensional space. For example, in a bright daytime, the area considered to be the head is 0 cm, and the area of the feet is 10 cm. In the nighttime, the area of the head is 10 cm, and the area of the feet is 10 cm. Furthermore, if the low beam of the vehicle can be reached, the head area is set to 10 cm and the foot area is set to 0 cm. Similarly, the width is appropriately enlarged or reduced. Further, the correction may be performed based on the change amount of the width in the time series. Further, the recognition area may be reduced depending on the content of the recognition processing in the subsequent stage. For example, in the case of a pedestrian, recognition is performed only with the upper body. The size to be enlarged or reduced may be determined by a predetermined ratio or the size on the image. However, by setting the size on the basis of the size in the three-dimensional space, it is possible to exclude a size that cannot be recognized as a recognition target.
Also, the size of the three-dimensional object area 501 due to the enlargement / reduction may be the same as the detection area 301 due to the relationship between the distance in the three-dimensional space and the pixels.

Although the detection characteristic information has been described above, the three-dimensional object area setting process 401 sets the three-dimensional object area 501 with higher accuracy by combining a plurality of pieces of the detection characteristic information. For example, a three-dimensional object area 501 in which the influence of day and night is further reduced is set by combining distance, brightness of the outside environment, light direction, road surface height, sensor resolution, and the like. Further, the number of pixels and the size described here are merely examples, and the present invention is not limited to this range.

[Recognition magnification setting processing]
Next, the recognition magnification setting processing 402 shown in FIG. 4 will be described.
Assuming that the three-dimensional object area 501 set in the three-dimensional object area setting processing 401 is the reference size of the recognition processing 209, this reference size is not always the optimum recognition area for recognition. Therefore, in the recognition magnification setting processing 402, a plurality of sizes of recognition areas are determined using the recognition characteristic information of the three-dimensional object. At this time, since the optimal recognition area is unknown, the recognition area is enlarged or reduced based on the reference size to determine a plurality of size recognition areas. The recognition characteristic information includes, for example, (1) the distance from the three-dimensional object, (2) the size of the three-dimensional object, (3) the limit size of the three-dimensional object, and (4) the sensor resolution. Hereinafter, these pieces of recognition characteristic information will be described.

(1) The distance from the three-dimensional object is an index for determining the amount of enlargement or reduction when the recognition area is enlarged or reduced. For example, when the three-dimensional object is far away, the size of the three-dimensional object per pixel increases. In this case, since an image is input to a later-described classifier 901 that performs recognition processing, the number of pixels to be enlarged or reduced when a three-dimensional object is far away is smaller than when it is nearby. Therefore, based on the reference size, the recognition area is enlarged or reduced in accordance with the distance to the three-dimensional object, and recognition areas of a plurality of sizes are determined.

(2) The size of the three-dimensional object is an index for determining the amount of enlargement or reduction when the recognition area is enlarged or reduced. For example, when the three-dimensional object is large, the number of pixels when performing enlargement or reduction on the image is smaller than when the three-dimensional object is small. In the case where recognition areas of a plurality of sizes are determined based on the size in the real space, the number of pixels when performing enlargement or reduction on an image is smaller in a distant place than in a nearby place. If the recognition area is not set in sub-pixel units, the size may be the same as the reference size.

(3) The limit size of the three-dimensional object is a limit size assumed when the three-dimensional object is a recognition target. For example, when the three-dimensional object is a pedestrian, if the height of the three-dimensional object is larger than 2.5 meters, an area is set in the reduction direction. Conversely, if the height is smaller than 0.8 meters, the area is set in the enlargement direction. If they are in between, areas are set for both. The upper and lower limits of the region to be set may be determined based on a three-dimensional object to be recognized, restrictions on recognition processing, and the like.

(4) For the sensor resolution, for example, if the sensors are the cameras 101 and 102, the size per pixel changes according to the distance. Therefore, a range to be enlarged or reduced can be determined based on the sensor resolution. For example, in a distant place where the size in a three-dimensional space per pixel exceeds 20 cm, the range to be enlarged or reduced is defined as a small range of one pixel or two pixels. Conversely, in a short distance where the size of one pixel in a three-dimensional space is less than 1 cm, the image is enlarged or reduced in a large range such as 10 pixels or 20 pixels.

サ イ ズ Note that the size on the image may be calculated from the size of the three-dimensional object in the three-dimensional space. Further, with respect to the detection characteristic information not considered in the setting of the three-dimensional object area 501 in the three-dimensional object area setting processing 401, a variation of the recognition area may be set using the detection characteristic information in the recognition magnification setting processing 402. In this case, which condition of the detection characteristic information is used to enlarge or reduce the recognition area is the same as the content described in the three-dimensional object area setting processing 401. Further, the number of pixels and the size described here are merely examples, and the present invention is not limited to this range.

FIG. 6 illustrates the principle of the recognition magnification setting process 402. In the recognition magnification setting processing 402, the three-dimensional object area 501 is set as a recognition area of a reference size, and the recognition areas 601 and 602 obtained by enlarging or reducing this are determined. The recognition area 501 is a recognition area having a small reference magnification with a reduced reference size, and the recognition area 601 is a recognition area having a high recognition magnification with a reduced reference size. In the example of FIG. 6, two types of recognition areas that are enlarged or reduced with respect to the reference size are shown. However, this number may have many variations as long as the recognition processing time has a margin. Further, only the enlargement or the reduction may be set by the setting of the detection processing 208 or the three-dimensional object area setting processing 401. The amount of enlargement or reduction of the recognition area is set based on the recognition characteristic information. In this case, similarly to the three-dimensional object area setting processing 401, the recognition area may be the same as the recognition area of the reference size depending on the resolution of the image.

FIG. 7 is a diagram illustrating the normalization in the recognition magnification setting process 402.
As shown in FIG. 7, an area to be normalized is determined when the recognition area (501, 601 and 602) is subjected to the subsequent recognition processing. The recognition area indicates a range in which recognition processing is performed in recognition processing described later. The recognition area 501 is a recognition area having a small reference magnification with a reduced reference size, and the recognition area 601 is a recognition area having a high recognition magnification with a reduced reference size.

In recognition processing, it is necessary to match the number of dimensions of input information. There is no guarantee that the recognition area 501 of the reference size captures the target object neatly, and how it should be captured depends on the characteristics of the recognition processing implemented in the apparatus. Therefore, a region to be normalized is set in advance. In the example of FIG. 7, the recognition area 501 almost includes the head and the feet, whereas the recognition area 601 having a small recognition magnification has the top of the head and the limbs protruding, and the recognition area 602 having the large recognition magnification has a head. Margins are formed at the top and feet. When these recognition areas are normalized to the same size, as shown in FIG. 7, they become recognition areas 701, 702, and 703 after normalization, and similar processing can be performed in recognition processing described later. However, this normalization processing is not necessarily performed in the recognition magnification setting processing 402. It may be implemented as a part of the later-described scanning per-magnification processing 404 and the later-described detailed recognition processing 408.

[Scanning area setting processing]
Next, the scanning area setting process 403 in FIG. 4 will be described. The scanning area setting process 403 sets a scanning area larger than the recognition area for each recognition area based on the arrangement characteristic information of the three-dimensional object. The scanning area is set as an area on the image, and in the recognition processing, the set scanning area is scanned by the recognition area. That is, the recognition area indicates a range in which recognition processing is performed in recognition processing described later, and the scanning area is a range in which the recognition area is moved within the range of the scanning area. Thus, the recognition processing is performed while moving the recognition area within the range of the scanning area. The arrangement characteristic information for determining the size of the scanning area includes, for example, (1) the distance position of the three-dimensional object, (2) the road surface height where the three-dimensional object exists, and the like. Hereinafter, such arrangement characteristic information will be described.

(1) The perspective position of the three-dimensional object is an index when setting the scanning area. For example, when a three-dimensional object is nearby, the scanning area on the image is determined to be large. If the three-dimensional object is far away, the scanning area is set small. This is because the sensor resolution is high when the object is near, and the scanning amount in the three-dimensional space when one pixel is scanned is about several mm, whereas it exceeds 10 cm in a distant place. The scanning area is also determined by characteristics such as the amount of detection deviation generated by the three-dimensional object detection. For example, when using a recognition process that exhibits the best performance when the center of the horizontal position of a three-dimensional object is taken, the amount of variance and dispersion between the center of the horizontal position of the three-dimensional object and the center of the horizontal position of the actual recognition target can be used for the scanning area. You may set so that the center of the horizontal position of a recognition object may be settled.

(2) The road surface height at which a three-dimensional object exists is an index when setting a scanning area. For example, when the road surface is rising and a three-dimensional object (such as a pedestrian) is located at a position higher than the own vehicle, the height of the head side becomes smaller than the actual height due to increased hiding on the head side. If a three-dimensional object (such as a pedestrian) is at a low position, the foot may be cut off or hidden by a bumper depending on the angle of view. The scanning area is enlarged or reduced in accordance with such a state.

In addition, regarding the detection characteristic information and the recognition characteristic information which are not considered in setting the three-dimensional object area and the recognition area in the three-dimensional object area setting processing 401 and the recognition magnification setting processing 402, the scanning area setting processing 403 uses the information. May be determined. In this case, the conditions under which the scanning area is enlarged or reduced are the same as in the three-dimensional object area setting processing 401 and the recognition magnification setting processing 402. Further, the number of pixels and the size described here are merely examples, and the present invention is not limited to this range.

FIG. 8 is a diagram for explaining the principle of the scanning area setting process 403. The scanning area setting process 403 determines the scanning areas 801, 802, and 803 for the recognition areas 501, 601, and 602, respectively. The scanning areas 801, 802, 803 are the same as or larger than the recognition areas 501, 601, 602. However, since the scanning areas 801, 802, 803 are scanned in the recognition areas 501, 601, 602, the scanning amount is not always large. The scanning regions 801, 802, and 803 determine regions on the image from the arrangement characteristic information. At this time, depending on the resolution of the image, the recognition area and the scanning area may be the same on the image. The scanning area is individually determined for each recognition area. However, if there is enough processing time, one having the largest scanning area may be used. If there is not enough processing time, one small scanning area may be adapted to each recognition area.

[Scan recognition processing for each magnification]
Next, the magnification-specific scan recognition processing 404 shown in FIG. 4 will be described. In the magnification-based scan recognition processing 404, the images and the parallax areas (distance areas) corresponding to the scan areas 801, 802, and 803 are scanned in the recognition areas 501, 601 and 602, and the recognition processing is performed for each scan position of each size. Then, it is determined whether the target scanning position is a three-dimensional object.

Here, if the performance of the recognition process is sufficient, the vehicle control process 210 may be performed using the result of the magnification-based scan recognition process 404 as shown by a broken line 405 in FIG. The magnification-based scanning recognition processing 404 may have a plurality of results depending on the magnification, the scanning position, and the like. In this case, narrowing is performed by processing such as selecting one having the best recognition result.

FIG. 9 is a view for explaining the principle of the magnification-based scanning recognition processing 404. While scanning in each of the scanning areas 801, 802, and 803 with the recognition areas 501, 601, and 602, a response position 902 as a result of recognition by the discriminator 901 that performs recognition processing is obtained. The response position 902 is indicated by x in FIG. The greater the number of response positions 902, the better the recognition process. As an example of the result of recognizing the inside of the scanning areas 801, 802, and 803 by the discriminator 901, the scanning area 801 'is the largest as shown by the response position 902 of the scanning areas 801', 802 ', 803'.

The discriminator 901 may use machine learning or may use heuristic threshold value determination. If the result of this determination is sufficient, recognition may be terminated using this result, as shown by the broken line 405 in FIG. In that case, for example, the one with the best recognition processing is adopted.

When the performance of the recognition processing is insufficient due to reduction of the calculation cost of the recognition processing in the magnification-based scan recognition processing 404, detailed processing may be performed using the result of the magnification-based scan recognition processing 404. . In the present embodiment, a case will be described in which the optimum magnification setting processing 406, the detailed recognition position determination processing 407, and the detailed recognition processing 408 are provided as the detailed processing.

[Optimal magnification setting processing]
The optimum magnification setting process 406 shown in FIG. 4 selects an optimum recognition region for the detailed recognition process from the recognition regions of a plurality of sizes created in the recognition magnification setting process 402. The selection method uses, for example, the number of recognition targets determined in the recognition processing result obtained by scanning and its reliability, the number of non-recognition targets determined and its reliability, the distribution of recognition results, and the like. The amount and the reliability of a plurality of sizes of recognition areas are compared, and an optimum recognition area is used. The optimum magnification setting process 406 may be omitted if there is not enough time for the processing time.

FIG. 10 is a view for explaining the principle of the optimum magnification setting processing 406. From the recognition results of the plurality of magnifications, the optimum magnification with the best response is selected. As described above, the optimum magnification is selected using the number of responses in the scanning area of the recognition processing and the reliability thereof. In the example of FIG. 10, the scanning area 801 ′ having the largest number of responses is selected. This scanning area 801 ′ corresponds to the scanning area 801 of the reference size, and the scanning area 801 of the reference size recognizes the reference size. This corresponds to the area 501.

[Detailed recognition position determination processing]
The detailed recognition position determination processing 407 illustrated in FIG. 4 determines a representative position for performing detailed recognition on the optimal magnification obtained in the optimal magnification setting processing 406. For the detailed recognition, for example, a position where the reliability of the recognition processing obtained in the magnification-based scanning recognition processing 404 is the highest is selected. Alternatively, the position may be determined by using a clustering means such as an average displacement method (Mean Shift method). When the optimum magnification setting processing 406 is not performed, the detailed recognition position determination processing 407 may be performed for each magnification.

FIG. 11 is a diagram for explaining the principle of the detailed recognition position determination processing 407. From the one or more response positions obtained from the magnification-based scan recognition processing 404, a representative position 111 for performing the detailed recognition processing 408 is determined. When there are a plurality of reaction points, a clustering technique such as the Mean Shift method is used. An area centered on the determined representative position 111 is a detailed identification area.

[Detailed recognition processing]
The detailed recognition processing 408 illustrated in FIG. 4 performs detailed recognition on the representative position 111 determined in the detailed recognition position determination processing 407, and calculates the type and reliability of the target. Alternatively, detailed recognition is performed using a recognition area of the optimal size selected based on the response position by the scan recognition processing 404 for each magnification, and the type and reliability of the target are calculated. For the detailed recognition processing 408, the discriminator 120 having the classification performance equal to or higher than the recognition processing used in the magnification-based scanning recognition processing 404 is used.

FIG. 12 is a diagram for explaining the principle of the detailed recognition processing 408. A detailed recognition process is performed on the representative position 111 obtained by the detailed recognition position determination process 407 using the discriminator 120 to determine the type of the three-dimensional object. The type of the three-dimensional object is, for example, a pedestrian, a vehicle, a signal, a sign, a white line, a tail lamp of a car, a headlight, and the like.

認識 The recognition processing used in the magnification-based scanning recognition processing 404 and the detailed recognition processing 408 includes, for example, the following techniques. A technique using template matching for comparing a template having a recognition target likeness prepared in advance with a recognition area. A technology that uses a discriminator that combines a luminance image, a feature amount such as HOG or Haar-Like, and a machine learning method such as a support vector machine, Ada-Boost, or Deep Learning. Further, the edge shape or the like may be recognized by a threshold determination that is determined artificially. The magnification-based scanning recognition processing 404 and the detailed recognition processing 408 include image processing such as resizing, smoothing, edge extraction, normalization, isolated point removal, gradient extraction, color conversion, and histogram creation necessary for performing these. .

(Modification)
In the present embodiment, the image recognition apparatus 100 using a stereo camera has been described. However, it may be realized by using an image recognition device 100 'that does not use a stereo camera.
FIG. 13 is a diagram illustrating a processing operation in the image recognition device 100 ′. The same parts as those of the image recognition apparatus 100 shown in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted.

The image recognition device 100 ’includes an optical camera 1301 and a radar sensor 1302. Thereby, a three-dimensional object is detected. An image is captured by the optical camera 1301, and image processing 205 such as correction for absorbing the peculiar habit of the image sensor is performed on the captured image information. The processing result of the image processing 205 is stored in the image buffer 206. Further, the distance to the three-dimensional object can be obtained by the radar sensor 1302. The detection processing 1303 detects a three-dimensional object in a three-dimensional space based on the distance to the three-dimensional object. The recognition process 209 performs a recognition process of specifying the type of the three-dimensional object with respect to the detection area set by the detection process 1303.

The detection processing 1303 that inputs the distance to the three-dimensional object output from the radar sensor 1302 needs to perform the detection processing in consideration of the sensor characteristics of the radar sensor 1302 used for distance measurement, but after the detection area is determined. The processing can be performed in the same manner as the configuration using the stereo camera described in the image recognition device 100. Further, the image recognition device 100 ′ does not require a plurality of images in the image processing 205.

According to the embodiment described above, the following operation and effect can be obtained.
(1) The image recognition devices 100 and 100 ′ detect the three-dimensional object detection area based on the three-dimensional object detection characteristic information with respect to the three-dimensional object detection area 301 set on the images captured by the cameras 101 and 102. A three-dimensional object area setting process 401 for setting the three-dimensional object region 501 by enlarging or reducing 301 is performed, and a recognition process for specifying the type of the three-dimensional object is performed on the three-dimensional object region 501 set by the three-dimensional object region setting process 401. And recognition processing 209. The detection characteristic information includes, for example, the distinctiveness of the three-dimensional object, the distance to the three-dimensional object, the size of the three-dimensional object, the assumed size of the three-dimensional object, the brightness of the outside environment, the direction of the headlight, the height of the road surface on which the three-dimensional object exists. It is at least one of the sensor resolutions of the imaging unit. Thus, it is possible to provide an image recognition device that accurately detects a three-dimensional object and improves recognition performance.

(2) The image recognition devices 100 and 100 ′ detect the three-dimensional object based on the first characteristic information of the three-dimensional object with respect to the three-dimensional object detection area 301 set on the images captured by the cameras 101 and 102. A three-dimensional object region setting process 401 for enlarging or reducing the detection region 301 to set a three-dimensional object region 501, and second characteristic information of the three-dimensional object using the three-dimensional object region 501 obtained by the three-dimensional object region setting process 401 as a reference size Based on the recognition magnification setting processing 402 that determines the recognition areas 601 and 602 of a plurality of sizes, and the plurality of recognition areas 601 and 602 that are determined in the recognition magnification setting processing 402, the third characteristic information of the three-dimensional object is used. A scanning area setting process 403 for setting a plurality of scanning regions 802 and 803 wider than the recognition regions 601 and 602 based on the scanning regions Using regions 802 and 803 includes a recognition process 209 that the recognition process is performed, the. The first characteristic information to the third characteristic information include, for example, the identifiability of the three-dimensional object, the distance to the three-dimensional object, the size of the three-dimensional object, the assumed size of the three-dimensional object, the brightness of the outside environment, the direction of the headlight, It is at least one of the height of the road surface on which the three-dimensional object exists, the sensor resolution of the imaging unit, the limit size of the three-dimensional object, the near / far position of the three-dimensional object, and the height of the road surface on which the three-dimensional object exists. Thus, it is possible to provide an image recognition device that accurately detects a three-dimensional object and improves recognition performance.

The present invention is not limited to the above embodiments, and other forms considered within the scope of the technical idea of the present invention are also included in the scope of the present invention, as long as the features of the present invention are not impaired. . Further, a configuration in which the above-described embodiment and the modification are combined may be adopted.

100, 100 '{Image Recognition Device, 101, 102 Camera, 103 Image Input Interface, 104 Image Processing Unit, 105 Processing Unit, 106 Storage Unit, 107 CAN Interface, 108 Control Processing Unit, 109 Internal Bus, 110 Vehicle Network CAN

Claims (12)

A three-dimensional object for setting a three-dimensional object area by enlarging or reducing the three-dimensional object detection area based on the three-dimensional object detection characteristic information with respect to the three-dimensional object detection area set on the image captured by the imaging unit. Object area setting unit,
A recognition processing unit that performs recognition processing for specifying the type of the three-dimensional object with respect to the three-dimensional object region set by the three-dimensional object region setting unit;
An image recognition device comprising: The image recognition device according to claim 1,
The detection characteristic information includes the distinguishability of the three-dimensional object, the distance from the three-dimensional object, the size of the three-dimensional object, the assumed size of the three-dimensional object, the brightness of the outside environment, the direction of the headlight, and the presence of the three-dimensional object. An image recognition device that is at least one of a height of a road surface to be scanned and a sensor resolution of the imaging unit. In the image recognition device according to claim 1 or 2,
With the three-dimensional object region obtained by the three-dimensional object region setting unit as a reference size, based on the recognition characteristic information of the three-dimensional object, a recognition magnification setting unit that determines recognition regions of a plurality of sizes,
The image recognition device, wherein the recognition processing unit performs the recognition processing on each of the plurality of sizes of the recognition areas determined by the recognition magnification setting unit. The image recognition device according to claim 3,
The image recognition device, wherein the recognition characteristic information is at least one of a distance from the three-dimensional object, a size of the three-dimensional object, a limit size of the three-dimensional object, and a sensor resolution of the imaging unit. The image recognition device according to claim 3,
For a plurality of the recognition areas determined by the recognition magnification setting unit, a scanning area setting unit that sets a plurality of scanning areas wider than the recognition area based on the arrangement characteristic information of the three-dimensional object,
The image recognition device, wherein the recognition processing unit performs the recognition process by using the scanning area set by the scanning area setting unit. The image recognition device according to claim 5,
The image recognition device, wherein the arrangement characteristic information is at least one of a perspective position of the three-dimensional object and a road surface height at which the three-dimensional object exists. The image recognition device according to claim 5,
An image recognition apparatus comprising: a magnification-based scan recognition processing unit that scans the plurality of scan regions with the plurality of recognition regions to obtain a response position of a recognition result. The image recognition device according to claim 7,
An image recognition apparatus comprising: an optimum magnification setting unit that selects the recognition area having an optimum size based on the response position by the magnification-based scan recognition processing unit. The image recognition device according to claim 8,
An image recognition apparatus comprising: a detailed recognition position determination processing unit that determines a representative position at which the recognition processing is performed based on the response position of the scanning area corresponding to the recognition area selected by the optimal magnification setting unit. The image recognition device according to claim 9,
An image recognition device comprising a detailed recognition processing unit that performs the recognition process using the recognition area or the representative position having the optimum size and specifies a type of the three-dimensional object to be recognized. A three-dimensional object detection area is set by enlarging or reducing the three-dimensional object detection area based on the first characteristic information of the three-dimensional object with respect to the three-dimensional object detection area set on the image captured by the imaging unit. A three-dimensional object area setting unit to
A recognition magnification setting unit that defines a plurality of sizes of recognition regions based on the second characteristic information of the three-dimensional object, using the three-dimensional object region obtained by the three-dimensional object region setting unit as a reference size;
A scanning area setting unit configured to set a plurality of scanning areas wider than the recognition area based on third characteristic information of the three-dimensional object, for the plurality of recognition areas determined by the recognition magnification setting unit;
Using the scanning region set by the scanning region setting unit, a recognition processing unit that performs a recognition process of specifying the type of the three-dimensional object,
An image recognition device comprising: The image recognition device according to claim 11,
The first characteristic information, the second characteristic information, and the third characteristic information are respectively the identification of the three-dimensional object, the distance to the three-dimensional object, the size of the three-dimensional object, and the size of the three-dimensional object. The assumed size, the brightness of the outside environment, the direction of the headlights, the height of the road surface on which the three-dimensional object exists, the sensor resolution of the imaging unit, the limit size of the three-dimensional object, the near-far position of the three-dimensional object, and the three-dimensional object An image recognition device that is at least one of the existing road surface heights.

Images