WO2025022609A1 - Object detection method and object detection device - Google Patents

Abstract

Provided is an object detection method, wherein: the optical flow of a feature point is detected from a captured image obtained by photographing the surroundings of a host vehicle with a camera (S2); a relative distance from the host vehicle to the feature point is measured (S3); a vehicle behavior of the host vehicle is measured (S4); a first feature point speed, which is the speed of the feature point on a spatial coordinate system, is estimated on the basis of the relative distance (S5); a second feature point speed, which is a speed component of the feature point in a lateral direction orthogonal to an optical axis direction of the camera, is calculated on the basis of the optical flow, the relative distance, and the vehicle behavior (S6); the feature point is clustered into a feature point group on the basis of the second feature point speed (S7); and the object is detected from the feature point group.

Description

Object detection method and object detection device

The present invention relates to an object detection method and an object detection device.

The following Patent Document 1 describes a moving speed detection device that detects an object in a monitored area by detecting an optical flow from a captured image of the monitored area, calculating a moving speed based on the optical flow, and grouping pixel blocks based on the moving speed.

JP 2005-214914 A

When detecting objects by clustering feature points on an image based on the movement speed detected from the captured image, there is a risk that objects moving at different speeds in the direction of the camera's optical axis may be mistakenly recognized as the same object. The present invention aims to improve the detection accuracy when detecting objects from captured images.

In one embodiment of the object detection method of the present invention, the optical flow of feature points is detected from an image captured by a camera around the host vehicle, the relative distance from the host vehicle to the feature points is measured, the vehicle behavior of the host vehicle is measured, a first feature point velocity, which is the speed of the feature point in a spatial coordinate system, is estimated based on the relative distance, a second feature point velocity, which is the lateral velocity component of the feature point perpendicular to the optical axis direction of the camera, is calculated based on the optical flow, the relative distance, and the vehicle behavior, the feature points are clustered into feature point groups based on the second feature point velocity, and objects are detected from the feature point groups.

According to the present invention, it is possible to improve the detection accuracy when detecting an object from a captured image.
The objects and advantages of the invention will be realized and attained by means of the elements and combinations recited in the appended claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention as claimed.

1 is a schematic configuration diagram of an example of a vehicle control device according to an embodiment; 1A is a schematic diagram of an image coordinate system and a spatial coordinate system, and FIG. 1B is a schematic diagram of the effect of the measurement accuracy of the vertical velocity on the horizontal velocity. 4 is a flowchart illustrating an example of an object detection method according to the first embodiment. FIG. 11 is a block diagram illustrating an example of a functional configuration of a controller according to a second embodiment. 10 is a flowchart illustrating an example of an object detection method according to a second embodiment. 13 is a flowchart illustrating an example of a modified object detection method. FIG. 13 is a block diagram illustrating an example of a functional configuration of a controller according to a third embodiment. 5A is a schematic diagram of an example of an intersection region, and FIG. 5B is a schematic diagram for explaining a method of calculating an intersection speed. 13 is a flowchart of an example of a method for calculating an intersection time.

First Embodiment
(composition)
1 is a schematic diagram of an example of a vehicle control device according to an embodiment. The vehicle 1 includes a vehicle control device 10 that controls the traveling of the vehicle 1. For example, the vehicle control device 10 may execute an autonomous driving control that automatically drives the vehicle 1 without the driver's involvement based on the traveling environment around the vehicle 1, or a driving assistance control that assists the driver in driving the vehicle 1 by controlling at least one of the steering mechanism, driving force, or braking force of the vehicle 1. The driving assistance control may be, for example, an automatic brake, a preceding vehicle following control, a constant speed traveling control, or a merging assistance control.

The vehicle control device 10 includes an external sensor 11 , a vehicle sensor 12 , a positioning device 13 , a map database (map DB) 14 , an actuator 17 , and a controller 18 .
The external sensor 11 includes a camera 11a mounted on the vehicle 1 and capturing an image of the surroundings of the vehicle 1. The camera 11a may be, for example, a stereo camera. The external sensor 11 may also include a plurality of different types of object detection sensors that detect objects around the vehicle 1, such as a laser radar, a millimeter wave radar, or a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

The vehicle sensor 12 is mounted on the host vehicle 1 and detects various information (vehicle signals) obtained from the host vehicle 1. The vehicle sensor 12 includes, for example, a vehicle speed sensor that detects the vehicle speed of the host vehicle 1, a wheel speed sensor that detects the rotational speed of the tires of the host vehicle 1, an acceleration sensor that detects the acceleration and deceleration of the host vehicle 1, a steering angle sensor that detects the steering angle of the steering wheel, a turning angle sensor that detects the turning angle of the steered wheels, a gyro sensor that detects the angular velocity of the host vehicle 1, a yaw rate sensor that detects the yaw rate, an accelerator sensor that detects the accelerator opening of the host vehicle, and a brake sensor that detects the amount of brake operation.

The positioning device 13 includes a Global Navigation System (GNSS) receiver and receives radio waves from a plurality of navigation satellites to measure the current position of the vehicle 1. The GNSS receiver may be, for example, a Global Positioning System (GPS) receiver. The positioning device 13 may be, for example, an inertial navigation system.
The map database 14 stores road map data. For example, the map database 14 may store high-precision map data (hereinafter simply referred to as a "high-precision map") suitable as map information for automated driving.

Actuator 17 operates the steering wheel, accelerator opening, and brake device of the host vehicle in response to control signals from controller 18 to generate vehicle behavior of the host vehicle. Actuator 17 includes a steering actuator, an accelerator opening actuator, and a brake control actuator. The steering actuator controls the steering direction and steering amount of the host vehicle. The accelerator opening actuator controls the accelerator opening of the host vehicle. The brake control actuator controls the braking operation of the brake device of the host vehicle.

The controller 18 is an electronic control unit that controls the running of the vehicle 1. The controller 18 includes a processor 18a and peripheral components such as a storage device 18b. The processor 18a may be, for example, a CPU or an MPU. The storage device 18b may include a semiconductor storage device, a magnetic storage device, an optical storage device, or the like. The storage device 18b may include memories such as a register, a cache memory, and a ROM and a RAM used as a main storage device.
The functions of the controller 18 described below are realized, for example, by the processor 18a executing a computer program stored in the storage device 18b. Note that the controller 18 may be formed of dedicated hardware for executing each information processing described below.

When controlling the driving of the host vehicle 1, the controller 18 detects the optical flow on the image captured by the camera 11a, and detects moving objects around the host vehicle 1 based on the detection result of the optical flow.
Specifically, the controller 18 detects feature points of an object from the image captured by the camera 11a, calculates the optical flow of the feature points, and calculates the lateral velocity _Vx of the feature points on the image coordinate system of the image captured by the camera 11a based on the optical flow.
Fig. 2A is a schematic diagram of an image coordinate system SI and a spatial coordinate system SS. The image coordinate system SI is a coordinate system that has a reference point (e.g., the center of the image) on the captured image IM of the camera 11a as its origin and represents the two-dimensional coordinates of the pixels of the captured image. In Fig. 2A, the horizontal direction and the vertical direction on the image coordinate system SI are represented by the symbols "x" and "y", respectively.

The spatial coordinate system SS is a coordinate system that represents three-dimensional coordinates in a three-dimensional space. In this specification, a camera coordinate system having the viewpoint of the camera 11a (the center point of the image sensor) as the origin O is exemplified as the spatial coordinate system SS, but a stationary coordinate system having a fixed point as the origin, such as a map coordinate system, may be used as the spatial coordinate system SS.
In Fig. 2(a), the longitudinal direction of the spatial coordinate system SS is represented by the symbol "Z". The longitudinal direction is the optical axis direction AO of the camera 11a. The vertical direction is represented by the symbol "Y", and the lateral direction perpendicular to the longitudinal direction and the vertical direction is represented by the symbol "X". For example, the longitudinal direction and lateral direction of the spatial coordinate system SS may respectively coincide with the longitudinal direction and the vehicle width direction of the vehicle 1. The optical axis direction AO may be inclined in the pitching direction with respect to the longitudinal direction of the vehicle 1.

The controller 18 measures the relative distance and azimuth angle from the vehicle 1 to the feature point on the spatial coordinate system SS. Hereinafter, the relative distance from the vehicle 1 to the feature point will be simply referred to as "relative distance". For example, the controller 18 may measure the relative distance based on the parallax of the feature point on a pair of captured images taken by the camera 11a, which is a stereo camera, and measure the azimuth angle from the coordinate position of the feature point. In addition, for example, the controller 18 may measure the relative distance and azimuth angle based on the laser radar, millimeter wave radar, LIDAR, or the like of the external sensor 11.
The controller 18 estimates a first feature point velocity, which is the velocity of the feature point of the object on the spatial coordinate system SS, based on the change over time of the relative distance and the azimuth angle.

The measurement accuracy of a stereo camera deteriorates inversely proportional to the square of the distance to the measurement target, so it is difficult to accurately calculate the velocity in the spatial coordinate system SS for feature points of a distant object.
For this reason, the controller 18 calculates the lateral velocity _VX of the feature point on the spatial coordinate system SS using the movement velocity of the feature point on the image coordinate system SI.
However, as shown by the arrow AR in Fig. 2B, even if the feature point _Pf moves in the depth direction _Dd of the view of the feature point _Pf from the origin O (camera viewpoint), the horizontal position x of the feature point _Pf on the image coordinate system SI does not change. Therefore, the accuracy of the vertical movement speed on the spatial coordinate system SS affects the calculation accuracy of the horizontal speed.

Specifically, if the lateral position and lateral velocity of feature point _Pf on the image coordinate system SI are denoted as "x" and " _Vx ", respectively, the vertical position and vertical velocity on the spatial coordinate system SS are denoted as "Z" and " _Vz ", respectively, and the focal length of camera 11a is denoted as "f", then the lateral velocity _VX on the spatial coordinate system SS can be expressed by the following equation (1).
V _X = (ZV _x + xV _Z )/f...(1)
From the above formula (1), it can be seen that the accuracy of the lateral velocity _VX is affected by the accuracy of the vertical velocity _VZ . Note that in this specification, an example is shown in which the "front-back direction Z" is used as the "vertical direction" in the spatial coordinate system SS, but instead, the "depth direction _Dd " shown in Fig. 2(b) may be used.

For this reason, it is difficult to accurately determine the moving speed in the spatial coordinate system SS for feature points of distant objects, where the measurement accuracy of the stereo camera decreases.
When the host vehicle 1 is stopped, the longitudinal speed of the object is approximated to be "0", and the lateral speed _VX of the object in the spatial coordinate system SS can be calculated by the following equation (2), thereby obtaining the lateral speed _VX of the object.
_Vx = _ZVx /f...(2)
However, when the host vehicle 1 is moving, the background and objects approach due to the vehicle behavior of the host vehicle, resulting in an inappropriate approximation to the actual observed values.

The controller 18 measures the vehicle behavior of the host vehicle 1 based on the vehicle signals detected by the vehicle sensors 12, and approximates the longitudinal speed based on the vehicle behavior.
For example, the controller 18 measures, as the vehicle behavior, the amount of movement and the amount of change in attitude (e.g., the amount of change in yaw angle) of the host vehicle 1 during a processing cycle of the controller 18. The controller 18 estimates the longitudinal speed of the feature point by calculating a homogeneous transformation matrix T relating to the time change in the position and attitude of the camera 11a based on the measured vehicle behavior.

For example, if the coordinate vector _xt = ( _Xt , _Yt , _Zt , 1) of a feature point at a certain time t is given, the estimated value x't _-1 = (X't-1, _Y't-1 , Z't _{-1 ,} 1) of the coordinate vector of the feature point at the time (t-1) one time before time t is found using x't _-1 = _Txt .
If the time width (t-(t-1)) of the processing cycle of the controller 18 is represented as Δt, the estimated value V _eZ of the vertical velocity of the characteristic point on the spatial coordinate system SS can be obtained by the following equation (3).
V _eZ = (Z _t - Z' _t-1 )/Δt...(3)
In addition, the estimated value of the longitudinal velocity V eZ may be calculated by calculating an estimated value of the coordinate vector of the characteristic point at time t based on the observed value x _{t-1 =} (X _t-1 , Y _t-1 , Z _t-1 , 1) of the coordinate vector of the characteristic point at time (t-1).

The longitudinal velocity V _eZ estimated in this way means the relative velocity resulting from the vehicle behavior of the host vehicle 1 when the feature point of the object is stationary in the longitudinal direction.
The controller 18 calculates the lateral velocity _V2X of the feature point on the spatial coordinate system SS, which is the velocity component of the feature point in the lateral direction perpendicular to the optical axis direction of the camera 11a, based on the estimated value _VeZ of the vertical velocity, using the following equation (4).
_V2X = ( _ZVx + _xVeZ )/f...(4)
This makes it possible to calculate the lateral velocity _V2X that does not depend on the measurement accuracy of the vertical velocity of the stereo camera. The lateral velocity _V2X is an example of a "second feature point velocity" described in the claims.
The controller 18 clusters the feature points extracted from the image captured by the camera 11a into feature point groups based on the second feature point velocities, and detects individual objects from the feature point groups.

3 is a flowchart of an example of the object detection method of the first embodiment. In step S1, the camera 11a captures an image of the surroundings of the host vehicle 1. In step S2, the controller 18 detects the optical flow of feature points from the captured image of the surroundings of the host vehicle 1. In step S3, the controller 18 measures the relative distance from the host vehicle to the feature points. In step S4, the controller 18 measures the vehicle behavior of the host vehicle 1 based on the vehicle signal detected by the vehicle sensor 12.
In step S5, the controller 18 estimates a first feature point velocity based on a time change in the relative distance. In step S6, the controller 18 calculates a second feature point velocity (e.g., a lateral velocity _V2X ) based on the optical flow, the relative distance, and the vehicle behavior. In step S7, the controller 18 clusters the feature points into feature point groups based on the second feature point velocity, and detects an object from the feature point groups.

Second Embodiment
4 is a block diagram showing an example of a functional configuration of the controller 18 according to the second embodiment. The controller 18 includes a spatial position calculation unit 20, an optical flow calculation unit 21, a first feature point velocity estimation unit 22, a second feature point velocity calculation unit 23, a moving object detection unit 24, and a vehicle control unit 25.
The spatial position calculation unit 20 measures the relative distance and azimuth angle from the vehicle 1 to the feature point. For example, the spatial position calculation unit 20 may measure the relative distance and azimuth angle based on the parallax between a pair of captured images taken by the camera 11a, which is a stereo camera, or may measure the relative distance and azimuth angle based on a laser radar, a millimeter wave radar, a LIDAR, or the like. The spatial position calculation unit 20 calculates the position (X, Y, Z) of the feature point on the spatial coordinate system SS based on the relative distance and the azimuth angle.

The optical flow calculation unit 21 detects the optical flow of characteristic points from a captured image of the surroundings of the vehicle 1 .
The first feature point velocity estimation unit 22 estimates the horizontal velocity _VX and the vertical velocity _VZ of the feature point on the spatial coordinate system SS as the first feature point velocity based on the time change of the position calculated by the spatial position calculation unit 20. For example, the first feature point velocity estimation unit 22 may estimate the vertical velocity _VZ using a Kalman filter. In particular, when the relative distance to the feature point is greater than a predetermined threshold Th described later, the vertical velocity _VZ may be estimated using a Kalman filter. That is, the vertical velocity _VZ may be estimated based on the relative distance and the azimuth angle measured from multiple frames of the captured image of the stereo camera used as the camera 11a. In addition, the estimation can be stabilized by using the estimated value V _eZ of the vertical velocity given by the above formula (3) as the initial state of the vertical velocity _VZ in the Kalman filter.

The second feature point velocity calculation unit 23 calculates the lateral velocity _Vx of the feature point on the image coordinate system SI based on the optical flow calculated by the optical flow calculation unit 21. The second feature point velocity calculation unit 23 calculates the lateral velocity _V2X given by the above equations (3) and (4) as the second feature point velocity based on the lateral velocity _Vx , the longitudinal position Z of the feature point, and the vehicle behavior of the host vehicle 1.

The moving object detection unit 24 classifies the feature points extracted from the captured image into groups based on the lateral velocity _VX and vertical velocity _VZ estimated by the first feature point velocity estimation unit 22 and the lateral velocity _V2X calculated by the second feature point velocity calculation unit 23, and detects the classified groups as individual moving objects. The moving object detection unit 24 includes a clustering unit 24a and a separation unit 24b.
The clustering unit 24a judges whether the relative distance from the vehicle 1 to the feature point is equal to or smaller than a predetermined threshold Th. The predetermined threshold Th may be set to a threshold distance at which the detection speed by the stereo camera used as the camera 11a has a predetermined allowable error. For example, the distance at which the detection speed has an allowable error can be calculated by determining the speed at which the parallax changes by one tone from the parallax characteristics and observation period of the stereo camera.

When the relative distance is equal to or smaller than the threshold value Th, the clustering unit 24 a performs labeling and clustering of the feature points extracted from the captured image based on the lateral velocity _VX estimated by the first feature point velocity estimating unit 22 .
First, the clustering unit 24a determines whether a feature point is moving to the right or left, or is stationary in the horizontal direction, based on the lateral velocity _VX , and performs labeling to classify the feature points into feature points moving to the right, feature points moving to the left, and stationary feature points. For example, the sign of the lateral velocity _VX when the x coordinate of the feature point is moving to the right in the image coordinate system SI may be defined as positive, and the sign of the lateral velocity _VX when the x coordinate of the feature point is moving to the left in the image coordinate system SI may be defined as negative.
The clustering unit 24a then forms a feature point group by clustering the feature points labeled as moving to the right based on the lateral velocity V _X. Similarly, the clustering unit 24a clusters the feature points labeled as moving to the left and the feature points labeled as stationary.

On the other hand, when the relative distance is not equal to or less than the threshold value Th, the clustering unit 24a performs labeling and clustering of the feature points extracted from the captured image based on the lateral velocity _V2X calculated by the second feature point velocity calculation unit 23. The contents of the labeling and clustering processes are similar to the above processes except that the lateral velocity _V2X is used instead of the lateral velocity _VX .
The separation unit 24b separates the feature point groups generated by the clustering unit 24a based on the vertical speed _VZ estimated by the first feature point speed estimation unit 22, and detects the separated feature point groups as individual moving objects. Note that the separation unit 24b may execute classification processing limited to feature point groups with high estimation accuracy of the vertical speed _VZ . For example, when the first feature point speed estimation unit 22 estimates the vertical speed _VZ using a Kalman filter, the estimation accuracy of the vertical speed _VZ may be calculated based on an error covariance matrix.

The vehicle control unit 25 executes vehicle control to control at least one of the steering mechanism, drive device, or braking device of the host vehicle 1 based on the moving object detected by the moving object detection unit 24. For example, the vehicle control unit 25 may control at least one of the steering mechanism, drive device, or braking device of the host vehicle 1 so as to avoid the detected moving object.

FIG. 5 is a flowchart of an example of an object detection method of the second embodiment. The processing of steps S10 to S14 is similar to steps S1 to S5 of FIG. 3. In step S15, the moving object detection unit 24 determines whether the relative distance from the vehicle 1 to the feature point is equal to or less than a predetermined threshold Th. If the relative distance is equal to or less than the predetermined threshold Th (step S15: Y), the processing proceeds to step S19. If the relative distance is not equal to or less than the predetermined threshold Th (step S15: N), the processing proceeds to step S16. The processing of step S16 is similar to step S6 of FIG. 3.

In step S17, the clustering unit 24a labels the feature points according to whether they are moving to the right or left, or are stationary in the lateral direction, based on the lateral speed _V2X .
In step S18, the clustering unit 24a forms a feature point group by clustering feature points labeled as moving to the right based on the lateral velocity _V2X . Feature points labeled as moving to the left and feature points labeled as stationary are similarly clustered. The process then proceeds to step S21.
The processes in steps S19 and S20 are similar to those in steps S17 and S18, except that the lateral velocity _V2X is replaced with the lateral velocity _VX .
The process of step S21 is similar to step S8 in FIG.

(Modification)
The clustering unit 24a in the modified example of the second embodiment clusters feature points based on a weighted velocity _VwX, which is a weighted sum of the lateral velocity _VX estimated as the first feature point velocity and the lateral velocity _V2X calculated as the second feature point velocity, weighted by a weighting coefficient a according to the relative distance to the feature point.
For example, the lateral velocity _VX is expressed by the above formula (1), and the lateral velocity _V2X is expressed by the above formula (4). Therefore, for example, the weighted velocity _VwX can be obtained by weighting the above formulas (1) and (4) with a weighting coefficient a, as in the following formula (5).
_{Vw ​ ​ ​}
The weighting coefficient a is set to be larger when the relative distance is large than when the relative distance is small. For example, the weighting coefficient a may be set to be larger as the relative distance increases.
The clustering unit 24a calculates a weighted velocity _VwX by the following equation (5) based on the longitudinal velocity _VZ estimated by the first feature point velocity estimating unit 22, _VeZ calculated by the above equation (3), and the weighting coefficient a.

Fig. 6 is a flowchart of an example of the object detection method according to the modified example. The processes in steps S30 to S35 are similar to those in steps S1 to S6 in Fig. 3. In step S36, the clustering unit 24a calculates a weighted velocity _VwX .
In step S37, the clustering unit 24a labels the feature points according to whether they are moving to the right or left, or are stationary in the horizontal direction, based on the weighted speed Vw _X. In step S38, the clustering unit 24a forms a feature point group by clustering feature points labeled as moving to the right, based on the weighted speed Vw _X. Similarly, the clustering unit 24a clusters feature points labeled as moving to the left, and feature points labeled as stationary. The process of step S39 is the same as step S8 in FIG. 3.

Third Embodiment
When a moving object detected by the moving object detection unit 24 crosses the path of the host vehicle 1, the controller 18 of the third embodiment estimates a crossing time T _cross from the current time until the moving object crosses the path of the host vehicle 1.
As described above, when a stereo camera is used to calculate the speed of a distant moving object in the spatial coordinate system SS, the measurement accuracy becomes unstable. On the other hand, when a Kalman filter is used to observe coordinates in the spatial coordinate system SS and calculate the speed, the delay of the Kalman filter becomes a problem when detecting a road user (vehicle or pedestrian) jumping out.
Therefore, the controller 18 of the third embodiment obtains the speed at which the moving object moves in a direction that intersects with the path of the vehicle 1 (hereinafter referred to as "intersection speed V _cross ") from the lateral speed V _2X calculated as described above, and estimates the intersection time T _cross .

Fig. 7 is a block diagram showing an example of the functional configuration of the controller 18 according to the third embodiment. The controller 18 according to the third embodiment includes an intersection region extraction unit 26, an intersection determination unit 27, and an intersection time estimation unit 28 in addition to the configuration shown in Fig. 4.
The intersection area extraction unit 26 extracts an intersection area R _cross , which is a candidate area where an object expected to intersect with the path of the vehicle 1 may exist, from the area around the vehicle 1. FIG. 8(a) is a schematic diagram of an example of the intersection area R _cross . For example, when the path T _r of the vehicle 1 intersects with the crosswalk CW, the area occupied by the crosswalk CW may be extracted as the intersection area R _cross . For example, when the path T _r of the vehicle 1 intersects with an oncoming lane, the area occupied by the oncoming lane may be extracted as the intersection area R _cross . For example, the area occupied by the intersecting lane that intersects with the lane in which the vehicle 1 is traveling may be extracted as the intersection area R _cross . The vehicle 1 may extract the intersection area R _cross from the map database 14, for example, or may extract the intersection area R _cross from the image captured by the camera 11a.

The intersection area extraction unit 26 obtains the expected direction in which road users present in the intersection area R _cross will move as the intersection direction D _cross . For example, the intersection area extraction unit 26 may obtain information on the intersection direction D _cross from high-precision map data suitable as map information for automated driving.
The intersection determination unit 27 detects, among the moving objects detected by the moving object detection unit 24, an intersection object that may intersect with the path T _r of the host vehicle 1. For example, the intersection determination unit 27 determines that the moving object MO detected by the moving object detection unit 24 is an intersection object when the moving object MO is located within the intersection area R _cross , or the distance between the moving object MO and the intersection area R _cross is equal to or smaller than a threshold, and the difference between the moving direction Dm of the moving object MO and the intersection direction D _cross is equal to or smaller than a threshold.
The intersection determination unit 27 calculates an intersection point P _cross between a line segment extending from the current position of the moving object MO in the intersection direction D _cross and the course T _r of the host vehicle 1 .

The crossing time estimation unit 28 converts the lateral direction speed _V2X calculated as described above into a crossing speed _Vcross , which is the speed in the crossing direction _Dcross . Fig. 8B is a schematic diagram for explaining a method of calculating the crossing speed _Vcross .
If the angle between the vertical direction (Z direction) on the spatial coordinate system SS and the crossing direction D _cross is θ, the horizontal (X direction) component _VX and the vertical (Z direction) component _VZ of the crossing speed V _cross are expressed by the following equations (6) and (7).
_VX =sinθ× _Vcross …(6)
_VZ =cosθ× _Vcross …(7)

On the other hand, the lateral velocity _Vx on the image coordinate system SI is expressed by the following equation (8).
V _x = (f/Z) x V _x - (fX/Z ² ) x V _Z ...(8)
Furthermore, if the vehicle speed of the host vehicle 1 is approximated to be 0, the lateral speed _V2X is given by the following equation (9).
V _2X = ZV _x /f (9)
The following equation (10) is derived from the above equations (8) and (9).
V _2X = V _X - (fX/Z) x V _Z ...(10)

By substituting the above equations (6) and (7) into the above equation (10), the conversion equation from the lateral velocity V _2X to the crossing velocity V _cross is derived by the following equation (11).
V _cross =V _2X /(sinθ−(X/Z)cosθ)…(11)
The intersection time estimation unit 28 calculates the intersection velocity V cross based on the horizontal velocity V _2X calculated by the second feature point velocity calculation unit 23, the vertical position Z and horizontal position X of the object calculated by the spatial position calculation _unit 20, and the above equation (11).
The intersection time estimation unit 28 estimates the intersection time T _cross by dividing the distance between the current position of the moving object MO and the intersection point P _cross by the intersection speed V _cross .
The vehicle control unit 25 may control at least one of the steering angle, driving force, or braking force of the host vehicle 1 based on the crossing time _Tcross . For example, when the host vehicle 1 is stopped and the difference between the time when the host vehicle 1 reaches the crossing point _Pcross after starting and the crossing time _Tcross is equal to or greater than a threshold, the host vehicle 1 may be permitted to start, and when the difference is less than the threshold, the host vehicle 1 may be prohibited from starting. Also, for example, when the host vehicle 1 is traveling and the difference between the time when the host vehicle 1 reaches the crossing point _Pcross and the crossing time _Tcross is less than a threshold, the host vehicle 1 may generate a braking force or perform automatic steering control to avoid the moving object MO.

9 is a flowchart of an example of a method for calculating the intersection time _Tcross . In step S40, the intersection area extraction unit 26 extracts the intersection area _Rcross . In step S41, the intersection determination unit 27 detects an intersection object. In step S42, the intersection determination unit 27 calculates the intersection point _Pcross . In step S43, the intersection time estimation unit 28 calculates the intersection speed _Vcross , and then calculates the intersection time _Tcross . In step S44, the vehicle control unit 25 controls at least one of the steering angle, driving force, or braking force of the host vehicle 1 based on the intersection time _Tcross .

(Effects of the embodiment)
(1) The controller 18 detects the optical flow of feature points from captured images obtained by photographing the surroundings of the host vehicle 1 with the camera 11a, measures the relative distance from the host vehicle 1 to the feature points, measures the vehicle behavior of the host vehicle 1, estimates a first feature point velocity, which is the speed of the feature point in the spatial coordinate system, based on the relative distance, calculates a second feature point velocity, which is the lateral velocity component of the feature point that is perpendicular to the optical axis direction of the camera 11a, based on the optical flow, the relative distance, and the vehicle behavior, clusters the feature points into feature point groups based on the second feature point velocity, and detects objects from the feature point groups.
This allows the second feature point velocity, which is not dependent on the measurement accuracy of the vertical velocity of the stereo camera, to be calculated from the captured images, and the feature points can be clustered based on the second feature point velocity, thereby improving the object detection accuracy. By separating the feature point groups generated by clustering based on the first feature point velocity calculated from the relative distance measurement results, it is possible to separate objects that have different velocities in the optical axis direction of the camera. As a result, the object detection accuracy can be further improved.

(2) If the relative distance is greater than a predetermined threshold, the controller 18 may cluster the feature points based on the second feature point velocity, and if the relative distance is equal to or less than the predetermined threshold, the controller 18 may cluster the feature points based on the first feature point velocity. The controller 18 may cluster the feature points based on a weighted sum of the first feature point velocity and the second feature point velocity, weighted according to the relative distance. In this way, feature points at close range where the distance measurement accuracy by the stereo camera is high can be clustered using the first feature point velocity estimated from the measurement results of the stereo camera as a priority, and feature points at long range where the distance measurement accuracy by the stereo camera is low can be clustered using the second feature point velocity that does not depend on the measurement accuracy of the vertical velocity by the stereo camera as a priority.

(3) The controller 18 may measure the relative distance from the captured images captured by the stereo camera 11a, and when the relative distance is greater than a predetermined threshold, estimate the speed of the feature point in the forward/backward direction based on the relative distance measured from the captured images of multiple frames. This improves the accuracy of estimating the speed of a feature point at a long distance where the distance measurement accuracy of the stereo camera is low.
(4) The controller 18 may label the feature points according to whether the feature points are moving to the right or left or stationary in the horizontal direction based on the second feature point velocity, and cluster feature points with the same label. This makes it possible to prevent feature points moving to the right from being clustered in the same group as feature points moving to the left or stationary feature points. Similarly, it is possible to prevent feature points moving to the left from being clustered in the same group as feature points moving to the right or stationary feature points, and to prevent stationary feature points from being clustered in the same group as moving feature points.

(5) The controller 18 may detect an object from feature points located within a candidate area in which an object is expected to intersect with the path of the host vehicle 1. This makes it possible to efficiently detect an object that is likely to approach the host vehicle 1.
(6) The controller 18 may estimate an intersection direction, which is a moving direction in which the object detected by separating the feature point group intersects with the path of the host vehicle, convert the second feature point speed of the object into a speed in the intersection direction, and estimate an intersection time until the object intersects with the path of the host vehicle based on the speed of the object in the intersection direction. For example, the controller 18 may estimate the intersection time for a moving object that is located in a candidate area in which an object that intersects with the path of the host vehicle exists and has a moving direction close to the intersection direction. This makes it possible to quickly estimate the intersection time until a moving object that may intersect with the path of the host vehicle 1 intersects with the path of the host vehicle 1.

All examples and conditional terms described herein are intended for educational purposes to assist the reader in understanding the present invention and the concepts provided by the inventor for the advancement of technology, and should be construed without limitation to the above specifically described examples and conditions, and the configuration of the examples in this specification with respect to showing the advantages and disadvantages of the present invention. Although the embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

1... host vehicle, 10... vehicle control device, 11... external sensor, 11a... camera, 12... vehicle sensor, 13... positioning device, 14... map database, 17... actuator, 18... controller, 18a... processor, 18b... storage device

Claims (9)

Detecting the optical flow of characteristic points from captured images of the surroundings of the vehicle using a camera;
measuring a relative distance from the vehicle to the feature point;
Measure a vehicle behavior of the host vehicle;
estimating a first feature point velocity, which is a velocity of the feature point in a spatial coordinate system, based on the relative distance;
calculating a second feature point velocity, which is a lateral velocity component of the feature point perpendicular to an optical axis direction of the camera, based on the optical flow, the relative distance, and the vehicle behavior;
clustering the feature points into feature point groups based on the second feature point velocities;
detecting an object from the group of feature points;
13. An object detection method comprising: clustering the feature points into the feature point group based on the second feature point velocity when the relative distance is greater than a predetermined threshold, and clustering the feature points into the feature point group based on the first feature point velocity when the relative distance is equal to or less than a predetermined threshold;
The method according to claim 1 , further comprising: separating the feature point groups based on the first feature point velocities; and detecting the separated feature point groups as objects. The object detection method according to claim 1, characterized in that the feature points are clustered based on a weighted sum of the first feature point velocity and the second feature point velocity weighted according to the relative distance. measuring the relative distance from an image captured by a stereo camera;
The object detection method according to any one of claims 1 to 3, characterized in that, when the relative distance is greater than a predetermined threshold, a speed of the feature point in a forward/backward direction, which is a direction of an optical axis of the camera, is estimated based on the relative distance measured from multiple frames of captured images. The object detection method according to any one of claims 1 to 4, characterized in that the feature points are labeled according to whether they are moving to the right or left or are stationary in the lateral direction based on the second feature point velocity, and feature points with the same label are clustered together. The object detection method according to any one of claims 1 to 5, characterized in that an object is detected from the feature points located within a candidate area in which an object that is expected to intersect with the path of the vehicle is present. estimating an intersection direction, which is a moving direction in which the object detected by separating the feature point group intersects with a path of the host vehicle;
converting the second feature point velocity of the object into a velocity in the crossing direction;
estimating an intersection time until the object intersects with the path of the host vehicle based on a speed of the object in the intersection direction;
The object detection method according to any one of claims 1 to 6. The object detection method according to claim 7, characterized in that the intersection time is estimated for a moving object that is located within or near a candidate area in which an object intersects with the vehicle's path and has a moving direction close to the intersection direction. A camera that captures images of the surroundings of the vehicle;
A sensor for measuring a vehicle behavior of the host vehicle;
a controller that detects an optical flow of a feature point from an image captured by the camera, measures a relative distance from the host vehicle to the feature point, estimates a first feature point velocity which is the velocity of the feature point on a spatial coordinate system based on the relative distance, calculates a second feature point velocity which is a lateral velocity component of the feature point perpendicular to an optical axis direction of the camera based on the optical flow, the relative distance, and the vehicle behavior, clusters the feature points into feature point groups based on the second feature point velocity, and detects an object from the feature point groups;
An object detection device comprising:

Images