WO2024171459A1 - Outside-world recognition device - Google Patents

Abstract

Provided is an outside-world recognition device that: can adjust the orientation of a device sensor with respect to a current environment by calculating camera orientation parameters and adjusting geometrical image conversion parameters; can increase the reliability of object detection by maintaining the accuracy of the distance to an object calculated in a monocular vision region of an acquired image, with no need for an additional external ranging sensor; can monitor and determine the reliability of the calculated camera orientation parameters and assist in scenarios in which there is a possibility that the position and/or velocity of the detected object has not been calculated correctly, particularly due to variations in sensor orientation produced by changes in roadway shape or host vehicle movement; and, accordingly, can avoid erroneous control results (warnings, braking, etc.). Erroneous control results are avoided by monitoring and determining the reliability of the calculated camera orientation parameters.

Description

External Recognition Device

The present invention relates to an in-vehicle external recognition device for image-based obstacle recognition in the vehicle's surrounding environment.

In recent years, image-based object recognition devices have been used to detect nearby moving objects and obstacles.

The image-based object recognition device described above can be used in surveillance systems to detect intrusions or abnormalities, or in vehicle-mounted systems to assist in safe driving.

In automotive applications, such devices are configured to detect moving objects (obstacles) around the vehicle, inform the driver of a potential risk of a collision between the vehicle and a moving or stationary obstacle, and, based on a decision system, automatically stop the vehicle to avoid a collision between the vehicle and the obstacle.

For example, the device disclosed in Patent Document 1 can adjust the relationship between device sensors to the current environment by calculating camera pose parameters and adjusting geometric image transformation parameters, thereby maintaining the accuracy of the calculated distance to detected objects without the need for additional external ranging sensors. In other words, by adjusting the camera pose parameters to estimate the distance to a target obstacle (object), it is possible to calculate the distance to an object detected in the monocular vision region of an image acquired by a stereo camera where distance information (parallax) is not available.

WO 2020/110435

However, in the device described in Patent Document 1, for example in a system used in a scenario where the vehicle speed and current road conditions affect the relationship between the on-board camera and the current road conditions, certain variations in the initially set camera attitude parameters may occur that may be difficult for the system to tolerate (out-of-tolerance situations), and therefore it is necessary to ensure the reliability of such camera attitude parameters to prevent a decrease in the reliability of the system.

The object of the present invention is to provide an external environment recognition device that can adjust the attitude of the device sensor with respect to the current environment by calculating camera attitude parameters and adjusting geometric image transformation parameters, thereby increasing the reliability of object detection by maintaining the accuracy of the calculated distance to an object detected in the monocular vision area of the acquired image without the need for an additional external ranging sensor, as well as monitoring and determining the reliability of the calculated camera attitude parameters, and in particular being able to assist in scenarios where the position and/or speed of a detected object may be calculated incorrectly due to fluctuations in the sensor attitude caused by changes in the road shape or the movement of the vehicle, thus avoiding erroneous control results (warning, braking, etc.).

In order to achieve the above object, the external environment recognition device of the present invention has an image acquisition unit that acquires an image of a camera sensor that captures the external environment, a sensor information acquisition unit that acquires sensor information collected from a sensor that detects information related to the current attitude of the camera sensor, a three-dimensional object detection unit that detects a three-dimensional object from the image, a camera attitude estimation unit that estimates the attitude of the camera sensor, a camera attitude reliability calculation unit that calculates the reliability of the attitude of the camera sensor estimated from the sensor information as a camera attitude reliability, an object selection unit that sets the reliability of the three-dimensional object detected by the three-dimensional object detection unit as a three-dimensional object reliability based on the camera attitude reliability, and a control judgment unit that makes at least one control judgment of an alarm or brake activation based on the three-dimensional object reliability.

According to the present invention, by adjusting the relationship of the device camera or between multiple device cameras to the current environment by calculating camera pose parameters, the accuracy of object detection and distance calculation can be increased simultaneously, thereby increasing the reliability of obstacle recognition, and by monitoring and judging the reliability of the calculated camera pose parameters, erroneous control results can be avoided, and driving safety can be increased even when the environmental conditions in which the device is moving change.

　Other issues, configurations, and advantages will become clear from the description of the embodiments below.

1 is a schematic configuration diagram of an external environment recognition device according to an embodiment of the present invention. FIG. 2 is a top view showing a schematic configuration of a sensing unit. FIG. 2 is a diagram illustrating the camera pose parameters pitch, yaw, roll, and camera height relative to a flat ground surface. FIG. 13 illustrates a scenario in which the external recognition device is mounted on a vehicle, and describes the results of determining the reliability of the camera pose parameters for three different cases. FIG. 13 is a diagram showing a scenario in which an external recognition device is mounted on a vehicle and uses the result of determining the reliability of camera pose parameters to set a reliability for a set of obstacle recognition results in two different cases. 11 is a flowchart showing a process in which the camera attitude parameter reliability determination unit 141 determines the reliability of a camera attitude parameter estimated at a certain time.

Below, a preferred embodiment of the external environment recognition device of the present invention will be described with reference to the drawings.

The configuration and performance of the external environment recognition device 100 will be described with reference to Figures 1 to 6. Although not shown in the figures, the external environment recognition device 100 has a configuration in which a CPU, RAM, ROM, etc. are connected via a bus, and the CPU controls the operation of the entire system by executing various control programs stored in the ROM. It should be noted that in the configuration described below, two camera sensors (hereinafter sometimes simply referred to as cameras or sensors) are paired as a single vehicle-mounted stereo camera and therefore correspond to the sensing unit 111. However, this does not limit the device to be used in other configurations in which a single monocular camera is used as the sensing unit 111.

FIG. 1 is a block diagram showing the configuration of an external environment recognition device according to this embodiment. The external environment recognition device 100 according to this embodiment is a device that is mounted on, for example, a vehicle (host vehicle), detects three-dimensional objects from images captured by a camera sensor (sensing unit 111) that captures the external world (surroundings of the host vehicle), and performs control decisions (control applications) such as issuing an alarm or activating the brakes based on the detection results of the three-dimensional objects.

In FIG. 1, the external environment recognition device 100 includes a sensing unit 111 including two camera sensors 111a and 111b arranged at the same height, an image acquisition unit 121, a camera attitude parameter calculation unit 131, a sensor information acquisition unit 132, a camera attitude parameter reliability determination unit 141, an obstacle detection unit 151, an object selection unit 161, and a control application processing unit 171.

(Sensing unit 111)
The two (pair) camera sensors 111a and 111b constituting the sensing unit 111 are in-vehicle stereo cameras, and are attached side by side in the left-right direction at the same height on the windshield or the rearview mirror in the vehicle interior, facing the front of the vehicle V, as shown in FIG. 2. The camera sensors 111a and 111b are also installed so that their respective fields of view (photographing areas) overlap. Therefore, in the image of the outside world (front of the vehicle V) photographed by the camera sensors 111a and 111b (in-vehicle stereo cameras), the center is a stereoscopic viewing area (an area where the fields of view overlap and where distance measurement information (parallax) can be used), and the left and right wide-angle areas are monocular viewing areas (areas where the fields of view do not overlap and where distance measurement information (parallax) cannot be used).

(Image Acquisition Unit 121)
The image acquisition unit 121 processes the images acquired by one or both of the two camera sensors 111a, 111b corresponding to the sensing unit 111 to adjust image characteristics for further processing. This processing may include, but is not limited to, image resolution adjustment, which can shrink or enlarge the input image to change the resulting image size, and image region of interest selection, which can cut (crop) a specific region of the input image from the original input image for further processing, image affine transformation such as rotation, scale, shear, and overhead image transformation, where a flat ground is considered as a reference. For affine transformation, the geometric formula or transformation table may be calculated or adjusted in advance. The parameters used for image resolution adjustment and image region of interest selection may be controlled based on the current driving environment and conditions (speed, turning speed, etc.). In addition, stereo matching is performed using the image signals received from both camera sensors 111a, 111b corresponding to the sensing unit 111 to create a three-dimensional range image of the scene in front of the vehicle equipped with the on-board obstacle detection and obstacle recognition device. In stereo matching, a unit area is determined in two images to be compared, where the difference between image signals is the smallest for each unit area. In other words, the area in which the same object is captured is detected. This results in a synthesis of a three-dimensional distance image (hereinafter referred to as a disparity image). It should be noted that the disparity data is used to calculate the distance value from the camera to the target in the real environment.

In the case of overhead image conversion, the image acquisition unit 121 also has the function of calculating a difference image showing the difference between at least two images acquired at different times by the sensing unit 111 and converted into an overhead image. Known methods can be applied for difference calculation, including but not limited to simple pixel-to-pixel difference calculation and filter-based image difference calculation.

(Camera Attitude Parameter Calculation Unit 131)
The camera attitude parameter calculation unit 131 has a function of calculating, using the information acquired by the sensor information acquisition unit 132, any or all of the camera external parameters (camera attitude parameters) relative to a flat ground (predetermined reference) defined by the camera pitch angle (rotation of the horizontal axis) (FIG. 3(a)), the camera roll angle (rotation of the vertical axis (front-back axis)) (FIG. 3(c)), the camera yaw angle (rotation of the vertical axis) (FIG. 3(b)), and the camera height (position of the Y axis relative to the flat ground) (FIG. 3(d)), as shown in FIG. 3. That is, the camera attitude parameter calculation unit 131 calculates, based on the information acquired by the sensor information acquisition unit 132, any or all of the camera attitude parameters that correct the deviation of the sensor optical axis OA (hereinafter simply referred to as the optical axis OA) of the sensing unit 111 (the camera sensors 111a and 111b) relative to the flat ground DR (predetermined reference) (i.e., the deviation of the current attitude of the sensing unit 111 (the camera sensors 111a and 111b)). The sensor information acquisition unit 132 also has a function of determining whether the camera attitude parameter calculation cannot be performed normally at a predetermined time due to a lack of information necessary for parameter calculation. This parameter calculation process differs depending on the information collected by the sensor information acquisition unit 132. For example, when the device configuration includes a process of estimating the inclination of the road in front of the vehicle based on the parallax information calculated by the stereo matching process, when information is acquired from the external environment recognition device, the estimated road is used to calculate the pitch angle and roll angle of the device with respect to the road in front of the vehicle, and its height, in the form of a flat ground. Also, if available, the movement of the vehicle during the processing period can be calculated taking into account the speed and turning speed of the vehicle acquired from the sensor information acquisition unit 132, and used accordingly to adjust the position of the sensor in three-dimensional space when the temporary (temporal) nature of the data acquired by the sensing unit 111 is taken into account. That is, the camera attitude parameter calculation unit 131 has a function as a camera attitude estimation unit that estimates the current attitude of the camera sensors 111a and 111b using the information acquired by the sensor information acquisition unit 132.

In another example, if the height, pitch, yaw, and roll angles can be obtained directly from external sensors, the camera pose parameters can be adjusted based on offsets from the mounting parameters. Other methods of calculating the above parameters can also be included.

(Sensor information acquisition unit 132)
The sensor information acquisition unit 132 has a function of collecting information used to calculate the camera attitude parameters in the camera attitude parameter calculation unit 131, and the above-mentioned information is collected from one or more sensors including, but not limited to, an external environment recognition device, a vehicle suspension system, a vehicle internal sensor (speed, turning speed, yaw rate, etc.), an attached inertial measurement sensor, etc. That is, the sensor information acquisition unit 132 has a function of acquiring sensor information collected from one or more sensors that detect information related to the current attitude of the camera sensors 111a and 111b (estimated by the camera attitude estimation unit).

(Camera Posture Parameter Reliability Determination Unit 141)
The camera attitude parameter reliability determination unit 141 has a function of determining the reliability of the camera attitude estimation (described by the parameters calculated by the camera attitude parameter calculation unit 131) by performing a time series data analysis of the parameter estimation result to determine whether the current state is reliable. That is, the camera attitude parameter reliability determination unit 141 has a function as a camera attitude reliability calculation unit that calculates the likelihood of the (current) attitude of the camera sensors 111a and 111b estimated by the camera attitude estimation unit (camera attitude parameter calculation unit 131) using sensor information as the camera attitude reliability. Furthermore, the reliability is determined for a set of three different regions based on the distance from the host vehicle (camera sensors 111a and 111b), hereinafter referred to as close-distance reliability (nearby reliability), medium-distance reliability (intermediate reliability), and far-distance reliability (far reliability). The distance and range covered by each region can be adjusted based on the configuration of the external environment recognition device and the control application executed on the host vehicle. The time series data analysis process takes into account (but is not limited to) the following conditions: the status/result of the estimation process (i.e., give up result, successful estimation result, etc.), the variation of individual parameters over a certain period of time (e.g., parameter difference with respect to the same parameter averaged over 5 seconds), and parameter difference with respect to the default value.

Examples of confidence settings for each range area ("trusted" or "untrusted") are shown below, although other variations may be included.

In order to set the confidence to "unreliable", any of the following conditions must be true: [nc1] the status/result of the estimation process fails for a predefined period (e.g., more than 3 seconds); [nc2] the variation of any of the estimated parameters over a certain period (e.g., moving average of the past 10 estimations) exceeds a predefined range for a defined time (e.g., more than 0.5 seconds); [nc3] the value of any of the estimated parameters shows a difference to the default value that is greater than a predefined range for a defined time (e.g., more than 0.5 seconds). If any of the above conditions are true for a predefined period [NCT1], first the confidence of the far-distance confidence region is set to "unreliable". If any of the above conditions remain true for a predefined period [NCT2], the confidence of the medium-distance confidence region is also set to "unreliable". Furthermore, if any of the above conditions remain true for a second predefined period [NCT3], the confidence of the near-distance confidence region is also set to "unreliable", provided that NCT1 < NCT2 < NCT3.

　Near, medium, and far confidence are set as "trusted" regardless of their previous status if all of the following conditions are true: [rc1] the status/result of the estimation process is successful for a given period of time (e.g., more than 5 seconds), [rc2] the variation of each estimation parameter over a certain period of time (e.g., a moving average of the past 10 estimations) is within a predefined range for a defined time (RCT1) (e.g., more than 1 second). This can also be considered as the recovery process.

(Obstacle detection unit 151)
The obstacle detection unit (three-dimensional object detection unit) 151 has a function of detecting a three-dimensional object using an image acquired by the image acquisition unit 121 and calculating its position. The obstacle detection unit 151 uses a method including, but not limited to, a method of creating a cluster of difference pixels that are close to each other and may represent a target obstacle on the road by using a known clustering method (e.g., K-means algorithm) that takes into account the distance between points (pixels) using a difference image. In this specification, "obstacle detection" refers to a process in which at least the following tasks are performed: target object detection (position in image space), target object ranging (position in three-dimensional space), target object identification (e.g., automobile/vehicle, motorcycle, bicycle, pedestrian, pole, etc.), target object tracking (e.g., searching for a previously detected object and tracking it if found), target object speed calculation (e.g., using the result of object tracking). The target object ranging process means obtaining a distance measurement value from the vehicle to the target object in three-dimensional space using a combination of a monocular-based method that relies on geometric calculations using camera attitude parameters when the object is located in the monocular vision area of the image (an area where disparity information cannot be calculated), and a method that uses disparity information calculated by the image acquisition unit 121 when the target object is located in an image area where disparity information is available and can be linked to the above-mentioned target object.

(Object Selection Unit 161)
The object selection unit 161 has the function of selecting and prioritizing obstacles detected by the obstacle detection unit 151 so that the appropriate control routine/application can be executed on the ego-vehicle. The location of the obstacles relative to the path of the ego-vehicle and their projected (movement) paths are used to select and prioritize targets based on their calculated proximity to the movement path (current and estimated) of the ego-vehicle. That is, if an obstacle is deemed to be on a movement path that will result in a collision (or near collision) with the ego-vehicle, it is selected as a target and prioritized if there are other selected targets. Target priorities are assigned based on the distance (proximity) and time to collision calculated for each target, resulting in a high priority being given to obstacles that have a high probability of collision or are near collision with the ego-vehicle both in time and distance. The above process takes into account the following parameters and conditions (but is not limited to): confidence in the camera pose parameters, identification of the obstacle, the period during which the obstacle has been tracked and identified, the distance between the detected obstacle and the ego-vehicle, the speed at which the obstacle is moving, the trajectory the obstacle is moving, the estimated path of the ego-vehicle, etc.

If the camera pose parameters are estimated to be unreliable for a particular area (monocular vision area + near/middle/far) where a candidate object is currently detected/tracked (and therefore the speed and direction at which such an obstacle is moving has been calculated), such an obstacle will be given a low priority or will not be selected as a target (this depends on the configuration of the vehicle's control application). However, this does not affect other processes that would continue in normal operation, such as obstacle tracking, if the camera reliability is improved. That is, the object selection unit 161 sets (prioritizes) the likelihood of an obstacle (three-dimensional object) detected by the obstacle detection unit 151 as a three-dimensional object reliability based on the camera pose reliability (reliability of the camera pose parameters).

In other words, in this embodiment, the image acquisition unit 121 acquires an overhead image based on the monocular vision areas of the left and right wide-angle parts of the on-board stereo camera (camera sensors 111a, 111b), and the obstacle detection unit 151 detects an obstacle from the overhead image. The obstacle detection unit 151 also calculates the distance and speed of the obstacle. The object selection unit 161 sets (prioritizes) three-dimensional object reliability for the obstacle (three-dimensional object) based on the camera pose reliability (reliability of the camera pose parameters) and the distance and speed of the obstacle.

(Control application processing unit 171)
The control application processing unit 171 has a function of determining an alarm routine (audible or visual alarm message to the driver) or a control application to be executed in the vehicle equipped with the external environment recognition device according to the result acquired from the object selection unit 161. That is, the control application processing unit 171 has a function as a control determination unit that performs control determination such as an alarm or brake operation to be executed in the vehicle based on the three-dimensional object reliability (priority) set by the object selection unit 161.

Here, a case where the external environment recognition device 100 is applied as a system for monitoring the surroundings of the vehicle V will be described with reference to Fig. 4 and Fig. 5. In the case of Fig. 4 described below, only a camera attitude parameter called the pitch angle (θ _Pitch ) is described, but other camera attitude parameters can be considered without changing the nature of the described processing.

Figure 4 shows a situation in which the vehicle V equipped with the external environment recognition device 100 is crossing an ideal flat road R1 with a gradient change (hereinafter referred to as R2), and the camera attitude parameters describing the relationship between the optical axis OA and the flat ground surface DR are estimated based on the slope of the road observed from the parallax information obtained within the range of the flat ground surface DR.

Figure 4(a) shows a situation in which the host vehicle V is crossing an ideal flat road R1, and the camera attitude parameter estimation performed by the camera attitude parameter calculation unit 131 was successful, with the calculated parameter variation being within the allowable range for the period during which the host vehicle was crossing the road. Therefore, all of the conditions necessary for the camera attitude parameter reliability determination unit 141 to set the "reliable" status are met (condition [rc1]: parameter estimation is successful, and condition [rc2]: parameter variation is small), and the short-distance reliability, medium-distance reliability, and long-distance reliability are set as "reliable".

FIG. 4(b) shows a situation where the vehicle V crosses an ideal flat road R1 before reaching a gradient change connecting to road R2, and the camera attitude parameters describing the relationship between the optical axis OA and the flat ground DR are estimated based on observation information obtained within the flat ground DR including the gradient change between roads R1 and R2. This causes fluctuations in the pitch angle parameters estimated by the camera attitude parameter calculation unit 131 with respect to the pitch angle value calculated before the information obtained within the flat ground DR includes the gradient change between the roads. Therefore, since the camera attitude parameter reliability determination unit 141 does not satisfy all the conditions necessary for setting the "reliable" status in the entire range (condition [rc2] unsatisfied due to large parameter variation), and since this status above has continued for a short period of time, the long-distance reliability is set as "unreliable", and the short-distance reliability and the medium-distance reliability are set as "reliable".

Figure 4(c) shows a situation where the vehicle V is crossing the junction between ideal flat road R1 and ideal flat road R2, and the camera pose parameters describing the relationship between the optical axis OA and the flat ground surface DR are estimated based on observational information obtained within the flat ground surface DR, which at this point only includes information from road R2.

In this case, the movement of the host vehicle when crossing the gradient change between roads R1 and R2 causes a variation in the pitch angle parameter estimated by the camera attitude parameter calculation unit 131 with respect to the pitch angle value calculated before the host vehicle reached the slope. Therefore, since the camera attitude parameter reliability determination unit 141 does not satisfy all the conditions necessary for setting the "reliable" status in the entire range (condition [rc2] unsatisfied due to large parameter variation) and since the pitch angle variation was constant for a long period of time (due to the scenario described in Figure 4(b) where the information obtained within the range of the flat ground DR includes gradient changes), both the mid-range reliability and the long-range reliability are set as "unreliable" and the short-range reliability is set as "reliable".

Figure 5 shows a situation in which the vehicle V equipped with the external environment recognition device 100 is crossing an ideal flat road R1, and the gradient change leading to road R2, showing different situations in which the camera attitude parameters describing the relationship between the optical axis OA and the flat ground surface DR are estimated based on the slope of the road observed from the parallax information obtained within the range of the flat ground surface DR.

Figure 5(a) shows a situation where the host vehicle V is crossing an ideal flat road R1, and the camera attitude parameter estimation performed by the camera attitude parameter calculation unit 131 is successful, with the close-range reliability, medium-range reliability, and long-range reliability being set as "reliable" (see Figure 4(a)). Therefore, when pedestrian P1 (within close range) and pedestrian P2 (within long range) are selected as objects by the object selection unit 161 based on their features (detection time, speed, etc.), they will continue to be candidates because the camera attitude parameters in the corresponding ranges are set as "reliable".

Figure 5(b) shows a situation in which the vehicle V is crossing an ideal flat road R1 before reaching a gradient change connecting to road R2, and the camera attitude parameters describing the relationship between the optical axis OA and the flat ground surface DR are estimated based on observation information obtained within the range of the flat ground surface DR including the gradient change between roads R1 and R2, and the camera attitude parameter estimation performed by the camera attitude parameter calculation unit 131 is successful, with the close-range reliability and mid-range reliability set as "reliable" but the long-range reliability set as "unreliable" (see Figure 4(b)). Thus, when pedestrian P1 (within close range) and pedestrian P2 (within long range) are selected as objects by the object selector 161 based on their features (detection time, speed, etc.), pedestrian P1 (within close range) will remain a candidate because the camera pose parameter for close range reliability is set to "reliable", but pedestrian P2 (within long range) will be lowered in priority or will not be selected as a target because the camera pose parameter for long range reliability is set to "unreliable" (this depends on the configuration of the control application of the host vehicle). However, this does not affect other processing that continues in normal operation, such as obstacle tracking, when the camera reliability is improved.

FIG. 6 is a flowchart showing the process executed by the camera attitude parameter reliability determination unit 141 to determine the reliability of the current camera attitude parameters and set a reliability flag for each appropriate range.

First, in step S1, a data storage process is performed to store the results (success and failure) obtained by the camera posture parameter calculation unit 131 for further analysis (time series data analysis) (save camera posture estimation history).

Next, in step S2, a check is performed to ensure that sufficient data has been stored since the start of processing (e.g. system startup at engine start). If not enough data has been stored, processing in step S3 is performed, otherwise step S4 is performed if sufficient data has been stored for further analysis. An example of sufficient data may correspond to 5 seconds or more of data stored.

If the result of the data check performed in step S2 is that not enough data has been stored, an initialization process is performed in step S3 to set the short-distance reliability, medium-distance reliability, and long-distance reliability as "unreliable." It should be noted that this initialization process needs to be performed every time until enough data has been stored for further analysis.

If the data check performed in step S2 shows that sufficient data has been stored, then in step S4, a time series data analysis is performed to determine the reliability of the current camera pose parameters, and the result of this determination is stored for further evaluation. Examples of this determination (but not limited to these) are as follows:

To determine the "unreliable" status, any of the following conditions must be true: [nc1] The status/result of the estimation process fails for a given period of time (e.g., more than 3 seconds); [nc2] The variation of any of the estimated parameters over a certain period of time (e.g., a moving average of the last 10 estimations) exceeds a predefined range for a defined time (e.g., more than 0.5 seconds); [nc3] The value of any of the estimated parameters shows a difference against the default value that is greater than a predefined range for a defined time (e.g., more than 0.5 seconds).

To determine a "trusted" status, all of the following conditions must be true: [rc1] The status/result of the estimation process is successful for a given period of time (e.g., more than 5 seconds); [rc2] The variation of each estimation parameter over a certain period of time (e.g., a moving average of the past 10 estimations) is within a predefined range for a defined time (e.g., more than 1 second).

Next, in step S5, the current and past results of step S4 (e.g., up to 10 seconds before the current time) are obtained to determine the confidence flag to be set for each range. The close, mid, and far confidences are set as "trusted" regardless of their previous status if the confidence of the current camera pose parameters (from step S4) is "trusted" and the same result (trusted) is constant for a defined time [RCT1] (e.g., more than 1 second). If the confidence of the current camera pose parameters (from step S4) is "untrusted" and the same result (untrusted) exists (not necessarily constant) for a defined time [NCT1], first the confidence of the far confidence area is set to "untrusted". If the above condition (untrusted and non-consecutive "trusted" results) continues for a predefined period [NCT2], the confidence of the far and mid confidence areas is set to "untrusted". In addition, if the above condition (a "trusted" result not followed by an "untrusted" result) continues for a specified period [NCT3], the long-distance trust level, the medium-distance trust region, and the short-distance trust region are set to "untrusted."

By employing the process described above with reference to FIG. 6, the reliability of the camera pose parameters can be determined for each range covered by the external environment recognition device 100. The result of this process is passed to the object selection unit 161, which performs a process of selecting an object from the obstacles detected by the obstacle detection unit 151.

As described above, the external environment recognition device 100 shown in FIG. 1 according to this embodiment has the following features:
a sensing unit 111 consisting of two sensing units (camera sensors 111a, 111b) capable of capturing images of the scene in front of the device to which the apparatus is attached;
an image acquisition unit 121 that processes the image acquired by the sensing unit 111 to adjust its characteristics (including, but not limited to, image size, image resolution, and image area of interest), and that can perform three-dimensional data generation by matching images acquired from both sensing units and calculating parallax for each pixel;
a camera attitude parameter calculation unit 131 that uses information acquired by a sensor information acquisition unit 132 to calculate any one or all of the camera attitude parameters defined by the camera pitch angle (horizontal axis rotation), the camera roll angle (vertical axis rotation), the camera yaw angle (vertical axis rotation), and the camera height (position on the Y axis relative to the plane on which the vehicle is standing) (and updates the geometric image transformation parameters based on the camera attitude parameters calculated by the camera attitude parameter calculation unit 131);
a sensor information acquisition unit 132 for collecting information used by the camera pose parameter calculation unit 131 to calculate the camera pose parameters, where said information is collected from one or more sensors;
a camera attitude parameter reliability determination unit for performing a time series data analysis process on the calculated camera attitude parameters and determining the reliability of the camera attitude estimation at a predetermined time;
an obstacle detection unit 151 that performs object detection and position identification using the images acquired by the image acquisition unit 121;
an object selection unit 161 which selects an obstacle to be input to a control application processing unit 171 from among the obstacles detected by the obstacle detection unit 151, taking into consideration the results of the camera attitude parameter reliability determination unit 141 and the obstacle detection unit 151;
and a control application processing unit 171 that determines a control application to be executed by a device in which the in-vehicle environment recognition device is mounted.

In other words, the external environment recognition device 100 according to this embodiment is
an image acquisition unit 121 that acquires an image of a camera sensor that captures an external environment;
a sensor information acquisition unit 132 that acquires sensor information collected from a sensor that detects information related to a current attitude of the camera sensor;
A three-dimensional object detection unit (obstacle detection unit 151) that detects a three-dimensional object (obstacle) from within the image;
a camera attitude estimation unit (camera attitude parameter calculation unit 131) that estimates the (current) attitude of the camera sensor (based on the sensor information);
a camera posture reliability calculation unit (camera posture parameter reliability determination unit 141) that calculates the likelihood of the (current) posture of the camera sensor estimated from the sensor information (by the camera posture estimation unit) as a camera posture reliability;
an object selection unit 161 that sets the likelihood of the three-dimensional object (obstacle) detected by the three-dimensional object detection unit as a three-dimensional object reliability based on the camera pose reliability;
and a control determination unit (control application processing unit 171) that performs at least one control determination of an alarm or a brake operation (to be executed in the vehicle) based on the three-dimensional object reliability.

By adopting such a configuration, the camera attitude parameter calculation unit 131 updates the camera attitude parameters to match the current environment, thereby improving the accuracy of object detection and position identification performed by the obstacle detection unit 151, and the camera attitude parameter reliability determination unit 141 can be used to monitor and determine the reliability of the calculated camera attitude parameters, thereby avoiding erroneous control caused by fluctuations in the camera attitude that would result in erroneous calculation of the position and/or speed of a detected obstacle. Therefore, even if the environmental conditions in which the device is moving change, reliable and accurate object detection and distance calculation can be performed simultaneously.

The above describes the configuration and operation of the external environment recognition device 100 according to this embodiment. According to the external environment recognition device 100 according to this embodiment, the relationship between the device camera or between multiple device cameras with respect to the current environment is adjusted by calculating camera attitude parameters, thereby simultaneously increasing the accuracy of object detection and distance calculation, thereby making it possible to increase the reliability of obstacle recognition, and by monitoring and judging the reliability of the calculated camera attitude parameters, erroneous control results can be avoided, making it possible to increase driving safety even when the environmental conditions in which the device moves change.

The above describes the currently contemplated preferred embodiment of the present invention, but various modifications can be made to the present embodiment, and all modifications within the true spirit and scope of the present invention are intended to be within the scope of the appended claims.

Furthermore, the present invention is not limited to the above-described embodiment, but includes various modified forms. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to having all of the configurations described.

Furthermore, each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware, in part or in whole, for example by designing them as integrated circuits. Furthermore, each of the above configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, files, etc. that realize each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

Furthermore, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

100 External environment recognition device 111 Sensing unit 121 Image acquisition unit 131 Camera attitude parameter calculation unit (camera attitude estimation unit)
132 Sensor information acquisition unit 141 Camera attitude parameter reliability determination unit (camera attitude reliability calculation unit)
151 Obstacle detection unit (three-dimensional object detection unit)
161 Object selection unit 171 Control application processing unit (control determination unit)
OA: Sensor optical axis DR: Reference axis of flat ground V: Vehicle (own vehicle)

Claims (8)

an image acquisition unit that acquires an image of a camera sensor that captures an external environment;
a sensor information acquisition unit that acquires sensor information collected from a sensor that detects information related to a current attitude of the camera sensor;
a three-dimensional object detection unit that detects a three-dimensional object from within the image;
a camera attitude estimation unit that estimates an attitude of the camera sensor;
a camera pose reliability calculation unit that calculates the likelihood of the pose of the camera sensor estimated based on the sensor information as a camera pose reliability;
an object selection unit that sets a likelihood of the three-dimensional object detected by the three-dimensional object detection unit as a three-dimensional object reliability based on the camera pose reliability;
and a control determination unit that performs at least one control determination of an alarm or a brake activation based on the three-dimensional object reliability. The external environment recognition device according to claim 1,
The external environment recognition device includes a camera attitude parameter calculation unit that calculates at least one of camera attitude parameters defined by a camera pitch angle, a camera roll angle, and a camera yaw angle to correct the current attitude of the camera sensor with respect to a predetermined reference based on the sensor information acquired by the sensor information acquisition unit. The external environment recognition device according to claim 2,
An external environment recognition device that performs geometric image transformation on the image acquired by the image acquisition unit based on the camera attitude parameters calculated by the camera attitude parameter calculation unit. The external environment recognition device according to claim 1,
the camera sensor is a stereo camera;
The image acquisition unit acquires an overhead image based on the monocular vision areas of the left and right wide-angle sections of the stereo camera, and the three-dimensional object detection unit detects the three-dimensional object from the overhead image. The external environment recognition device according to claim 4,
An external environment recognition device that calculates the distance and speed of the three-dimensional object. The external environment recognition device according to claim 5,
The object selection unit is an external environment recognition device that sets the three-dimensional object reliability based on the camera pose reliability and the distance and speed of the three-dimensional object. The external environment recognition device according to claim 1,
The camera pose reliability calculation unit is an external environment recognition device that sets the camera pose reliability based on a distance from the camera sensor. The external environment recognition device according to claim 1,
The object selection unit is an external environment recognition device that sets the three-dimensional object reliability based on the time and distance until collision calculated for each three-dimensional object.

Images