CN111366917B - Method, device and equipment for detecting travelable area and computer readable storage medium - Google Patents

Abstract

The application discloses a method, a device and equipment for detecting a drivable area and a computer readable storage medium, and relates to the technical field of autonomous parking. The specific implementation scheme is as follows: acquiring a matching image and a target image acquired by shooting a surrounding area of a vehicle with a vehicle-mounted monocular camera, wherein the matching image is a previous frame image continuous with the target image; then, performing three-dimensional reconstruction on the target image according to the matched image to obtain target three-dimensional point cloud data corresponding to the target image, so that the reconstruction of the three-dimensional point cloud data of the continuous frame image acquired by the monocular camera is realized; and then, according to the target three-dimensional point cloud data, determining the nearest barrier point in the shooting area of the target image, and according to each nearest barrier point, determining the travelable area in the shooting area of the target image, without acquiring a three-dimensional image or a large number of image samples for machine learning, thereby reducing the dependence on an image acquisition mode, reducing the processing difficulty and improving the accuracy and reliability of detection.

Description

Driving area detection method, apparatus, device, and computer-readable storage medium

technical field

The embodiments of the present application relate to the technical field of image processing, and in particular, to an autonomous parking technology.

Background technique

In order to avoid the collision between the body and the obstacle or beyond the road boundary, the vehicle needs to detect the drivable area during the process of autonomous driving and autonomous parking.

At present, it is mainly based on supervised deep learning image detection or using 3D cameras to collect 3D point clouds for drivable area detection. However, image detection based on supervised deep learning requires a lot of manual annotation costs, and models trained on limited datasets are difficult to solve the generalization problem. The 3D camera has a complex structure, is difficult to manufacture, and has a high detection cost.

It can be seen that the reliability of the existing drivable area detection method is not high enough.

SUMMARY OF THE INVENTION

The purpose of the present application is to provide a drivable area detection method, apparatus, device and computer-readable storage medium, which improve the reliability of drivable area detection.

According to a first aspect of the present application, a method for detecting a drivable area is provided, comprising:

Obtaining a matching image and a target image captured by the vehicle-mounted monocular camera on the surrounding area of the vehicle, wherein the matching image is a previous frame image that is continuous with the target image;

Perform 3D reconstruction on the target image according to the matching image to obtain target 3D point cloud data corresponding to the target image;

According to the target three-dimensional point cloud data, determine the nearest obstacle point in the shooting area of the target image;

According to each of the nearest obstacle points, a drivable area in the shooting area of the target image is determined.

In the embodiment of the present application, the 3D point cloud data is reconstructed from the continuous frame images collected by the monocular camera, and the nearest obstacle point is extracted to obtain the drivable area, which reduces the dependence of the drivable area detection on the image acquisition method, and reduces the processing difficulty , Improve the accuracy and reliability of detection.

In some embodiments, performing 3D reconstruction on the target image according to the matching image to obtain target 3D point cloud data corresponding to the target image includes:

Projecting the matching image onto a plurality of preset projection surfaces according to the viewing angle of the target image to obtain a plurality of projection images, wherein each of the projection surfaces corresponds to a depth relative to the origin of the camera;

Determine the estimated depth of the pixels in the target image according to the matching cost of the pixels in the target image and the corresponding pixels in the plurality of projection images;

Acquire target three-dimensional point cloud data corresponding to the target image according to the estimated depth of the pixels in the target image.

The embodiment of the present application utilizes the projected images of the matching images on projection surfaces of different depths to achieve depth recovery of the target image at cost matching, and transforms to obtain the target 3D point cloud data, which improves the accuracy and reliability of the 3D reconstruction of the monocular image. , thereby improving the accuracy and reliability of drivable area detection.

In some embodiments, the projection planes include: N1 vertical projection planes;

The N1 vertical projection planes are parallel to the front face of the camera, and the distances from the camera origin to the N1 vertical projection planes are distributed in inverse proportion, wherein N1 is an integer greater than 1.

The embodiment of the present application realizes the depth recovery of the area in front of the camera through the vertical projection plane, improves the accuracy of depth recovery in complex environments such as curves, and further improves the accuracy and reliability of the three-dimensional reconstruction of the target image.

In some embodiments, the projection surface further includes: N2 horizontal projection planes and/or N3 projection spherical surfaces;

Wherein, the N2 horizontal projection planes are parallel to the ground directly below the camera, and the N2 horizontal projection planes are evenly arranged within the ground distribution range with the ground as the center of symmetry, where N2 is an integer greater than 1;

The N3 projection spheres are concentric spheres with the origin of the camera as the sphere center, and the radii of the N3 projection spheres are inversely proportional and equally distributed, wherein N3 is an integer greater than 1.

In the embodiment of the present application, the depth of the ground area in the target image is recovered by the horizontal projection plane, and more normal samples are introduced by the projection sphere, so as to improve the accuracy and reliability of the depth recovery. For the points in the target image that are neither on the horizontal nor vertical planes, the depth recovery accuracy of these points can be improved by increasing the normal sampling by combining with the projection sphere. In addition, the introduction of a parallel projection sphere can also provide a favorable projection surface for the fisheye target image with a viewing angle greater than 180 degrees, and improve the accuracy and reliability of depth recovery.

In some embodiments, the determining the estimated depth of the pixels in the target image according to the matching cost of the pixels in the target image and the corresponding pixels in the plurality of projection images includes:

Obtain the target pixel window feature of the pixel in the target image;

acquiring projected pixel window features of corresponding pixels in the plurality of projected images;

According to the target pixel window feature and the projection pixel window feature, obtain the matching cost of the pixel in the target image and the corresponding pixel in each of the projection images;

The depth corresponding to the corresponding pixel with the smallest matching cost is taken as the estimated depth of the pixel in the target image, wherein the depth corresponding to the corresponding pixel is the depth corresponding to the projection surface where the corresponding pixel is located.

The embodiment of the present application improves the accuracy of restoring the depth of pixels in the target image by using the depth corresponding to the corresponding pixel with the smallest matching cost as the estimated depth of the pixel in the target image.

In some embodiments, before the determining the estimated depth of the pixels in the target image according to the matching cost of the pixels in the target image and the corresponding pixels in the plurality of projection images, the method further includes:

According to the camera relative poses of the target image and the matching image, determine the corresponding pixels in the matching image that correspond to the pixels in the target image one-to-one;

According to the corresponding pixels in the matching image, the corresponding pixels in each of the projection images corresponding to the pixels in the target image one-to-one are determined.

The embodiment of the present application improves the accuracy and reliability of the depth recovery of the target image by matching the corresponding pixels in the image and the target image through the relative pose positioning of the camera.

In some embodiments, before determining the corresponding pixels in the matching image that correspond one-to-one with the pixels in the target image according to the relative poses of the cameras of the target image and the matching image, the method further includes:

Collect wheel speedometer data of the rear wheel of the vehicle and on-board inertial measurement unit IMU data, wherein the wheel speedometer data of the rear wheel of the vehicle indicates the horizontal movement distance of the vehicle-mounted monocular camera, and the vehicle-mounted IMU data indicates all Describe the horizontal movement direction of the vehicle-mounted monocular camera;

Determine the camera pose data of the vehicle-mounted monocular camera according to the wheel speedometer data of the rear wheel of the vehicle and the vehicle-mounted IMU data;

According to the camera pose data of the vehicle-mounted monocular camera, the relative camera poses of the target image and the matching image are determined.

In the embodiment of the present application, since there is no relative rotation between the rear wheel and the body in the horizontal direction, the wheel speed of the rear wheel can directly represent the moving speed of the body, and the camera pose is obtained by combining the wheel speedometer data of the rear wheel of the vehicle and the data of the on-board IMU, which improves the The reliability of the relative pose of the camera is improved, thereby improving the accuracy and reliability of the drivable area detection.

In some embodiments, determining the closest obstacle point in the shooting area of the target image according to the target three-dimensional point cloud data includes:

Determine the number of obstacle points included in each grid in the polar coordinate grid network according to the target three-dimensional point cloud data and the polar coordinate grid network that horizontally divides the shooting area of the target image, wherein the The obstacle point is a target three-dimensional point whose height above the ground is greater than the preset obstacle height threshold;

According to the nearest obstacle grid in each sector partition of the polar coordinate grid network, determine the nearest obstacle point in each direction in the shooting area of the target image, wherein the nearest obstacle grid is the sector The grid in the partition with the closest radial distance to the origin of the camera and containing the number of obstacle points greater than the preset number threshold.

In this embodiment of the present application, the polar coordinate grid network established for the camera origin is used to spatially divide the shooting area of the target image, and the grid that may be an obstacle and the smallest radial distance from the camera origin in the fan-shaped partitions in each direction is extracted, and the It is used to determine the nearest obstacle points in all directions of the origin of the camera, which improves the accuracy of the nearest obstacle points, thereby improving the accuracy and reliability of the drivable area detection.

In some embodiments, determining the nearest obstacle points in each direction in the shooting area of the target image according to the nearest obstacle grid in each sector of the polar coordinate grid network includes:

Obtain the average position point of the obstacle points contained in the nearest obstacle grid;

The average position point corresponding to each of the nearest obstacle grids is taken as the nearest obstacle point in the shooting area of the target image.

The embodiment of the present application further optimizes the position of the nearest obstacle point, and improves the accuracy of the obstacle point.

In some embodiments, the determining a drivable area in the shooting area of the target image according to each of the nearest obstacle points includes:

In the uniform segmentation network for horizontally segmenting the shooting area of the target image, the weighted value of each grid unit is determined according to the position of each grid unit in the uniform segmentation network relative to the nearest obstacle point;

According to the initial weight of each grid unit and the weight value, a new weight value of each grid unit is determined, wherein the new weight value is greater than or equal to the minimum weight threshold value and less than or equal to the maximum weight threshold value , the initial weight of the grid unit is 0 or a new weight determined when the drivable area is determined for the previous frame of image;

Determine a drivable area in the shooting area of the target image according to the first-type grid unit or the second-type grid unit indicated by the new weight, wherein the first-type grid indicated by the new weight There is the closest obstacle point between the unit and the camera origin, and there is no closest obstacle point between the second type of grid unit indicated by the new weight and the camera origin.

In the embodiment of the present application, the weighted value of each grid unit in the uniform segmentation network of the target image shooting area is calculated, and the weighted value of the grid unit is updated. , which can not only smooth the noise, but also make up for the problem of missing detection of the nearest obstacle point in a single frame, and improve the reliability of the detection of the drivable area.

In some embodiments, determining the weighted value of each grid unit according to the position of each grid unit in the uniformly divided network relative to the nearest obstacle point includes:

If the radial distance between the grid unit in the uniform segmentation network and the camera origin, minus the difference between the radial distance between the nearest obstacle point in the direction of the grid unit and the camera origin, is greater than or equal to the upper limit of the distance threshold, then the weighted value of the grid unit is the first value;

If the radial distance between the closest obstacle point and the camera origin, minus the difference between the radial distance between the grid unit and the camera origin in the direction of the closest obstacle point, is less than or equal to the lower threshold of the distance, then the grid The weighted value of the unit is the second value;

If the radial distance between the closest obstacle point and the camera origin, minus the difference between the radial distance between the grid unit and the camera origin in the direction of the closest obstacle point, is less than the distance upper threshold and greater than the distance lower limit threshold, the weighted value of the grid unit is a third value, wherein the third value is a value determined according to the difference and a preset smooth continuous function;

Wherein, the distance upper threshold is the inverse of the distance lower threshold, the first value is the inverse of the second value, and the absolute value of the third value is less than the absolute value of the distance upper threshold or The absolute value of the distance lower threshold.

The embodiment of the present application realizes the determination of the weighted value of each grid unit, and uses a smooth continuous function to smoothly transition the grid units whose difference is between the upper threshold of the distance and the lower threshold of the distance, further reducing the weighted accumulation process of multi-frame fusion. , which improves the reliability of the determination of the new weights of the grid cells, and further improves the reliability of the detection of the drivable area.

In some embodiments, the matching image and the target image are both fisheye images.

By adopting the fisheye image, the embodiment of the present application achieves an increase in the horizontal viewing angle (which can exceed 180 degrees), enlarges the field of view of the image shooting area, and improves the reliability of the detection of the drivable area.

According to a second aspect of the present application, there is provided a drivable area detection device, comprising:

an image acquisition module, configured to acquire a matching image and a target image captured by the vehicle-mounted monocular camera on the surrounding area of the vehicle, wherein the matching image is a previous frame of the target image;

a first processing module, configured to perform three-dimensional reconstruction on the target image according to the matching image, and obtain target three-dimensional point cloud data corresponding to the target image;

a second processing module, configured to determine the nearest obstacle point in the shooting area of the target image according to the target three-dimensional point cloud data;

The third processing module is configured to determine a drivable area in the shooting area of the target image according to each of the nearest obstacle points.

According to a third aspect of the present application, an electronic device is provided, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the first and first aspects of the present application The drivable area detection method described in the embodiment.

According to a fourth aspect of the present application, there is provided a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the first aspect of the present application and any embodiment of the first aspect. drivable area detection method.

An embodiment in the above application has the following advantages or beneficial effects: by acquiring a matching image and a target image captured by shooting the surrounding area of the vehicle with a vehicle-mounted monocular camera, wherein the matching image is a preceding image that is continuous with the target image. One frame of image; then carry out three-dimensional reconstruction of the target image according to the matching image to obtain the target three-dimensional point cloud data corresponding to the target image, so as to realize the reconstruction of three-dimensional point cloud data for the continuous frame images collected by the monocular camera; then Determine the nearest obstacle point in the shooting area of the target image according to the target three-dimensional point cloud data, and determine the drivable area in the shooting area of the target image according to each of the nearest obstacle points, without collecting three-dimensional Machine learning is performed on images or a large number of image samples, which reduces the dependence on the image acquisition method, reduces the processing difficulty, and improves the accuracy and reliability of detection.

Other effects of the above-mentioned optional manners will be described below with reference to specific embodiments.

Description of drawings

The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present application. in:

1 is a schematic diagram of an application scenario of autonomous parking provided by an embodiment of the present application;

2 is a schematic flowchart of a method for detecting a drivable area provided by an embodiment of the present application;

FIG. 3 is a schematic flowchart of the method of step S102 in FIG. 2 provided by an embodiment of the present application;

4 is a schematic diagram of projecting matching depths on multiple projection surfaces provided by an embodiment of the present application;

5 is a top view of a vertical projection plane distribution provided by an embodiment of the present application;

6 is a side view of a multi-type projection plane distribution provided by an embodiment of the present application;

7 is a top view of a polar coordinate grid network provided by an embodiment of the present application;

8 is a schematic diagram of the nearest obstacle point in a shooting area of a target image provided by an embodiment of the present application;

9 is a top view of a uniformly divided network provided by an embodiment of the present application;

10 is a schematic diagram of a drivable area provided by an embodiment of the present application;

11 is a schematic structural diagram of a driveable area detection device provided by an embodiment of the present application;

FIG. 12 is a block diagram of an electronic device of a drivable area detection method according to an embodiment of the present application.

Detailed ways

Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

In the scenario of autonomous driving or autonomous parking, the vehicle needs to detect the drivable area around the vehicle in order to plan a safe and feasible driving path or parking path, and realize the automatic avoidance of obstacles or automatic parking in autonomous driving. Inventory. For example, in the process of autonomous parking in a garage, it is necessary to detect the drivable area in the forward or backward direction, and then control the vehicle to enter the parking space according to the drivable area. Referring to FIG. 1 , it is a schematic diagram of an application scenario of autonomous parking provided by an embodiment of the present application. In the parking scene shown in Figure 1, when the vehicle reverses into the parking space, it needs to avoid other vehicles on the left side of the vacant parking space and the stone pillars on the right side. First, it is necessary to collect the three-dimensional information of the object behind the vehicle when it is reversing, and identify the positions of other vehicles and stone pillars, so as to accurately obtain the drivable area when the vehicle is reversing.

In the existing drivable area detection methods, image detection through supervised deep learning requires a large number of image samples for learning, and there may be a risk of inaccurate recognition when encountering new obstacles or complex environments. Using a 3D camera to collect a 3D point cloud behind the vehicle requires configuring a 3D camera with a complex structure for the vehicle, which may cause unreliable detection while the vehicle is running.

The present application provides a drivable area detection method, device, equipment and computer-readable storage medium, which utilizes a vehicle-mounted monocular camera to detect the drivable area, reduces the difficulty of detection, and improves the accuracy and reliability of detection.

Referring to FIG. 2 , it is a schematic flowchart of a drivable area detection method provided by an embodiment of the present application. The execution body of the method shown in FIG. 2 may be a software and/or hardware drivable area detection device, for example, various types of drivable area detection devices may be used. The combination of one or more of the terminal, vehicle detection system, cloud, etc. The method shown in FIG. 2 includes steps S101 to S104, and the details are as follows:

S101: Acquire a matching image and a target image captured by a vehicle-mounted monocular camera on a surrounding area of a vehicle, where the matching image is a previous frame image that is continuous with the target image.

Vehicle monocular cameras can be installed in the front or rear of the vehicle. The vehicle-mounted monocular camera shoots the surrounding area of the vehicle, for example, the area in front of the vehicle may be photographed, or the area behind the vehicle may be photographed. Alternatively, the image of the area in front of the vehicle or the image of the area behind the vehicle can be selected to be collected according to the forward or reverse motion of the vehicle.

The target image is determined from the consecutive frame images collected by the vehicle-mounted monocular camera, and the previous frame of the target image is used as the matching image. The target image here may be, for example, a current frame image collected in real time by a vehicle-mounted monocular camera, but may also be a historical frame image, which is not limited in this embodiment.

The vehicle-mounted monocular camera may be a camera using a non-wide-angle lens, a wide-angle lens, or an ultra-wide-angle lens. Fisheye lens is an optical imaging system with large field of view and large aperture. Generally, two or three negative meniscus lenses are used as the front light group to compress the large field of view on the object side to the field of view required by conventional lenses. A camera with a fisheye lens captures images with a viewing angle of, for example, 220° or 230°.

In some embodiments, the vehicle-mounted monocular camera may be a camera using a fisheye lens, and the matched image and the target image collected by the camera may both be fisheye images. In this embodiment, by using a fisheye image, the horizontal viewing angle (for example, exceeding 180°) can be increased, the field of view of the image shooting area can be expanded, and the reliability of the detection of the drivable area can be improved.

S102: Perform three-dimensional reconstruction on the target image according to the matching image, to obtain target three-dimensional point cloud data corresponding to the target image.

The target image is a two-dimensional image, but the pose change of the camera can be obtained between it and the previous frame image, and then a depth image can be generated by projecting into the target image, thereby realizing three-dimensional reconstruction. There are various implementations of step S102, which are described below with reference to FIG. 3 and specific embodiments. Referring to FIG. 3 , it is a schematic flowchart of the method of step S102 in FIG. 2 provided by an embodiment of the present application. The method shown in FIG. 3 specifically includes steps S201 to S203, which are as follows:

S201 , project the matching image onto a plurality of preset projection surfaces according to the viewing angle of the target image to obtain a plurality of projection images, wherein each of the projection surfaces corresponds to a depth relative to the origin of the camera.

Referring to FIG. 4 , it is a schematic diagram of projecting matching depths on multiple projection surfaces according to an embodiment of the present application. As shown in the figure, according to the viewing angle of the target image, the matching image is projected onto multiple projection surfaces to obtain projected images with various depths. It can be understood that the camera is preset with projection surfaces with multiple depths. Depth can be understood as the distance from the vertical plane where the camera origin is located. Multiple projection surfaces at different depths can be understood as multiple projection surfaces in space, and each projection surface has a different radial distance from the camera origin (also the distance from the vertical plane where the camera origin is located), so each The projection surface corresponds to a depth. After the matching image is projected onto the projection surface, the projection image on the projection surface has a depth corresponding to the projection surface, that is, the depth of all pixels in a single projection image is the depth of the projection surface.

In some embodiments, the above-mentioned projection planes include, for example, N1 vertical projection planes, where N1 is an integer greater than 1. Specifically, the N1 vertical projection planes are parallel to the front face of the camera, and the distances from the camera origin to the N1 vertical projection planes are distributed in inverse proportion and equally. In the scene shown in Figure 1, the on-board monocular camera added to the vehicle is facing the wall of the front garage, then the N1 vertical projection planes can be understood as N1 space planes parallel to the front garage wall. In the image captured by the camera, the resolution of the near object image is usually larger than that of the distant object image, which can be understood as the actual size represented by the far-field pixels is larger than the actual size represented by the near-field pixels. Therefore, in order to improve the reliability of depth recovery, the distribution density of the vertical projection plane close to the camera origin is greater than the distribution density of the vertical projection plane far from the camera origin. In addition, the closer the obstacle is to the vehicle, the greater the possible impact or danger. In the area closer to the vehicle, the accuracy of depth recovery can be improved to generate more accurate depth data for the area close to the vehicle. For example, the distribution densities of the N1 vertical projection planes may satisfy: the distribution density near the camera origin is greater than the distribution density far from the camera origin. Referring to FIG. 5 , it is a top view of a vertical projection plane distribution provided by an embodiment of the present application. 64 vertical projection planes (not all shown in the figure) are arranged in the front shooting area of the camera origin shown in Figure 5, and the distances from the camera origin to each vertical projection plane are distributed in inverse proportion, for example: 20/ 64 meters, 20/63 meters, 20/62 meters, ...., 20/3 meters, 20/2 meters, 20 meters. Among them, the closer to the camera origin, the denser the distribution of the vertical projection plane, and the greater the accuracy of the recovered depth. This embodiment realizes the depth recovery of the area in front of the camera through the vertical projection plane, improves the accuracy of depth recovery in complex environments such as curves, and further improves the accuracy and reliability of the three-dimensional reconstruction of the target image.

In other embodiments, on the basis of the above vertical projection plane, N2 horizontal projection planes and/or N3 projection spheres may also be introduced as projection surfaces. Among them, the horizontal projection plane can be used for the depth recovery of the ground, and the projection sphere can be used for the depth recovery of the distorted image in the fisheye image.

Referring to FIG. 6 , it is a side view of a multi-type projection plane distribution provided by an embodiment of the present application. FIG. 6 illustrates a plurality of mutually parallel vertical projection planes, mutually parallel horizontal projection planes and concentric projection spheres. The N2 horizontal projection planes are parallel to the ground directly below the camera, and the N2 horizontal projection planes are evenly arranged within a ground distribution range with the ground as the center of symmetry, where N2 is an integer greater than 1. Taking Figure 6 as an example, after the camera is calibrated, the plane position of the ground directly below the camera origin can be determined, and four horizontal projection planes are evenly distributed in the ground distribution range of -5cm to 5cm near the ground directly below the camera. The depth of the point close to the ground is recovered. The number of horizontal projection planes may also be 8. In this embodiment, through the horizontal projection plane, the depth of the pixels in the ground area in the target image can be recovered, and the accuracy and reliability of the depth recovery of the ground image area can be improved.

In the above embodiment, the N3 projection spheres are concentric spheres with the camera origin as the sphere center, and the radii of the N3 projection spheres are inversely proportional and equally distributed, wherein N3 is an integer greater than 1. Continuing to refer to Figure 6, taking the camera origin as the center of the sphere, the 64 radii from 0.5m to 32m are equally distributed in inverse proportion to form a projection sphere. Setting the projection sphere can introduce more normal sampling on the basis of the projection plane, especially for pixels in the target image that are neither on the horizontal nor vertical plane, by adding normal sampling in combination with the projection sphere, the depth recovery of these pixels can be improved. accuracy. In addition, in the above-mentioned embodiment in which the target image and the matching image are both fisheye images, the introduction of N3 parallel projection spheres can also provide favorable projection surfaces for fisheye target images with a viewing angle greater than 180°, improving depth recovery efficiency. Accuracy and reliability.

S202: Determine the estimated depth of the pixels in the target image according to the matching cost of the pixels in the target image and the corresponding pixels in the multiple projection images.

Continuing to refer to Figure 4, the target image has corresponding pixels in each projection image. The smaller the matching cost, the more relevant the pixel features are. Therefore, the depth of the corresponding pixel with the smallest matching cost is used as the estimation of the pixel in the target image. Depth, to achieve depth recovery of the target image, the depth map corresponding to the target image can be obtained.

In some embodiments, before performing the matching cost calculation, the corresponding pixels in the matching image that correspond one-to-one with the pixels in the target image may be determined according to the relative camera poses of the target image and the matching image. . Specifically, various existing tracking algorithms can be used to implement the tracking of the corresponding pixels, which is not limited here. After the corresponding pixels in the matching image are determined, the corresponding pixels in each of the projection images that correspond one-to-one with the pixels in the target image may be determined according to the corresponding pixels in the matching image. See Figure 4. Each projected image has a corresponding pixel to a pixel in the target image. This embodiment improves the accuracy and reliability of the depth recovery of the target image by matching the corresponding pixels in the image and the target image through the relative pose positioning of the camera.

The positional relationship between the on-board monocular camera, the body of the vehicle, and the on-board inertial measurement unit (IMU) is pre-calibrated. According to the moving direction and distance of the vehicle body, information such as the camera origin position, camera shooting area, and angle of view of the vehicle-mounted monocular camera can be determined.

Before determining the corresponding pixels in the matching image that correspond one-to-one with the pixels in the target image according to the relative poses of the cameras of the target image and the matching image, the relationship between the matching image and the target image may be determined first. The relative pose of the camera. For example, first collect wheel speedometer data and onboard IMU data of the rear wheel of the vehicle, wherein the wheel speedometer data of the rear wheel of the vehicle indicates the horizontal movement distance of the onboard monocular camera, and the onboard IMU data indicates all The horizontal movement direction of the vehicle-mounted monocular camera; the camera pose data of the vehicle-mounted monocular camera can be determined according to the wheel speedometer data of the rear wheels of the vehicle and the vehicle-mounted IMU data. Each frame of image captured by the vehicle-mounted monocular camera has a corresponding camera pose. Thereby, the relative pose of the camera between the matched image and the target image can be determined according to the camera pose data of the vehicle-mounted monocular camera. In this implementation, since there is no relative rotation between the rear wheels of the vehicle and the vehicle body in the horizontal direction, the wheel speed of the rear wheels can directly represent the moving speed of the vehicle body. The reliability of the camera pose is improved, thereby improving the accuracy and reliability of the drivable area detection.

In step S202, a specific implementation manner may be, for example, to first acquire target pixel window characteristics of pixels in the target image and projection pixel window characteristics of corresponding pixels in the multiple projection images. The window feature here is, for example, sliding a sampling window of a preset size on the target image and the projection image, and taking the mean value of the pixel features in the window as the window feature of the center pixel of the window. The size of the sampling window may be 7*7, 5*5, or 1*1, which is not limited here. The target pixel window feature is, for example, the average gray value of the pixels in the sampling window centered on the pixel in the target image. The matching pixel window feature is, for example, the gray mean value of the pixels in the sampling window centered on the corresponding pixel. Then, according to the feature of the target pixel window and the feature of the projected pixel window, the matching cost of the pixel in the target image and the corresponding pixel in each of the projected images is obtained. For example, the gray mean error obtained by sampling the pixels in the target image and the corresponding pixels in the projection image with a 7*7 window is used as the matching cost between the corresponding pixels and the pixels in the target image. After obtaining the matching cost of the corresponding pixel in each projection image to the pixel in the target image, the depth corresponding to the corresponding pixel with the smallest matching cost can be used as the estimated depth of the pixel in the target image, wherein the corresponding pixel The corresponding depth is the depth corresponding to the projection surface where the corresponding pixel is located. This embodiment improves the accuracy of restoring the depth of pixels in the target image by using the depth corresponding to the corresponding pixel with the smallest matching cost as the estimated depth of the pixel in the target image.

S203: Acquire target three-dimensional point cloud data corresponding to the target image according to the estimated depth of the pixels in the target image.

The process of determining the estimated depth for each pixel in the target image can be performed in parallel. After all the pixels in the target image have determined the estimated depth, a depth image corresponding to the target image is obtained. Then, combining the depth image with the pixel position of each pixel in the target image, the three-dimensional information of each pixel can be obtained, and the target three-dimensional point cloud data corresponding to the target image can be obtained.

In the embodiment of 3D reconstruction shown in FIG. 2 , by using the projection images of the matching images on the projection surfaces of different depths, the depth recovery of the target image is achieved by matching the cost, and the target 3D point cloud data is obtained by transformation, which improves the 3D reconstruction of the monocular image. The accuracy and reliability of the reconstruction, in turn, improve the accuracy and reliability of the drivable area detection.

S103, according to the target three-dimensional point cloud data, determine the nearest obstacle point in the shooting area of the target image.

The target 3D point cloud data reflects the 3D stereo information in the shooting area of the target image, so that the 3D points of obstacles can be filtered out, and the 3D points used to determine the boundary of the drivable area can be filtered again according to the radial distance relative to the origin of the camera. The nearest obstacle point.

In some embodiments, it may be determined that each grid in the polar coordinate grid network contains obstacles according to the target three-dimensional point cloud data and the polar coordinate grid network that horizontally divides the shooting area of the target image The number of points is determined, and then the nearest obstacle points in each direction in the shooting area of the target image are determined according to the nearest obstacle grids in each sector of the polar coordinate grid network.

Specifically, firstly, a polar coordinate grid network can be used to spatially divide the area captured by the camera, so as to extract the nearest obstacle points therein. For example, the target three-dimensional point in each grid of the preset polar coordinate grid network may be determined according to the target three-dimensional point cloud data and the camera origin position corresponding to the target image. The camera origin position is, for example, pre-calibrated information used to indicate the position of the camera origin relative to the ground height and relative to the vehicle body. Referring to FIG. 7 , it is a top view of a polar coordinate grid network provided by an embodiment of the present application. As shown in Figure 7, the polar coordinate grid network is a segmentation network formed by superimposing and segmenting the first type of cutting planes arranged in a fan shape and the second type of cutting planes facing the camera and arranged in parallel for the shooting area of the target image. , the intersection line of the first type of cutting plane is collinear with the vertical line to the ground of the origin of the camera. Both the first type of cutting surface and the second type of cutting surface are planes perpendicular to the ground. Referring to FIG. 7 , the division of the shooting area of the target image by a plurality of first-type cutting planes can be formed, for example, in a plan view, with the camera origin as the center, dividing the horizontal direction 175° into 128 sector divisions. On this basis, multiple second-type cutting planes are distributed in parallel with the front face of the camera for superposition and segmentation. In some embodiments, the closer to the vehicle the higher the pixel resolution, the denser the target 3D points, and the closer the vehicle to the obstacle may have a greater impact, the density of the segmentation close to the camera origin should be greater than the segmentation far from the camera origin density. The distribution density of the second type of cut surfaces close to the camera origin is greater than that of the second type of cut surfaces far from the camera origin. For example, the second type of cutting plane is divided into 63 segments to form a grid according to the radial distance from the camera origin from 0.5m to 32m in inverse proportion.

Continuing to refer to FIG. 7 , after determining the target three-dimensional point in each grid of the preset polar coordinate grid network, the number of obstacle points contained in each grid can be determined, wherein the obstacle point is the ground The target 3D point whose height is greater than the preset obstacle height threshold. The preset obstacle height threshold may be, for example, a limit height that the vehicle can directly cross, such as 4 cm, 5 cm, 6 cm, etc., for example, the chassis height of the vehicle. Then, in the sector partition segmented by the first type of cutting plane, a nearest obstacle grid is determined, wherein the nearest obstacle grid is the radial distance closest to the camera origin in the sector partition, and A raster that contains more than a preset number of obstacle points. For example, search for the first grid with the number of obstacle points greater than the preset number threshold in each direction of the polar coordinate grid network shown in FIG. 7 from the origin as the center, as the nearest obstacle grid in this direction. Finally, the nearest obstacle point in the shooting area of the target image may be determined according to the nearest obstacle grid. Referring to FIG. 8 , it is a schematic diagram of a nearest obstacle point in a shooting area of a target image provided by an embodiment of the present application. In the scene shown in Figure 8, other vehicles and stone pillars on both sides of the parking space are outside the nearest obstacle point. In the area on the camera side delineated by the nearest obstacle point, the drivable area can be determined. In this embodiment, the polar coordinate grid network established for the camera origin is used to spatially divide the shooting area of the target image, and the grids that may be obstacles and the smallest radial distance from the camera origin in the fan-shaped partitions in each direction are extracted for use in The nearest obstacle points in all directions of the origin of the camera are determined, which improves the accuracy of the nearest obstacle points, thereby improving the accuracy and reliability of the drivable area detection.

The closest obstacle point can be the center point of the closest obstacle grid or a point determined from the target 3D point in the closest obstacle grid. In some embodiments, the average position point of the obstacle points included in the nearest obstacle grid may be obtained; the average position point corresponding to each of the nearest obstacle grids may be used as the shooting area of the target image The nearest obstacle point in the middle. This embodiment further optimizes the position of the nearest obstacle point and improves the accuracy of the obstacle point.

S104: Determine a drivable area in the shooting area of the target image according to each of the nearest obstacle points.

After the nearest obstacle point is determined, the nearest obstacle point can be used as the boundary of the drivable area. In order to improve the reliability of the drivable area, multi-frame fusion processing can also be performed in combination with the evenly divided network scene, and the boundary of the drivable area can be continuously optimized.

In some embodiments, a uniform segmentation network including the closest obstacle point may be obtained according to the closest obstacle point and the camera origin position corresponding to the target image. Wherein, the uniform segmentation network is a segmentation network formed by dividing the shooting area of the target image with a uniform square grid in the horizontal direction. Referring to FIG. 9 , it is a plan view of a uniformly divided network provided by an embodiment of the present application. As shown in Figure 9, the horizontal space is evenly divided into a uniformly divided network according to a grid of 0.1m*0.1m. The square grid is actually a three-dimensional grid that uniformly divides the horizontal direction. Above is a prismatic region with a square cross-section.

In this embodiment, the space is evenly divided by evenly dividing the top view of the network. In the evenly dividing network that horizontally divides the shooting area of the target image, each grid unit in the evenly dividing network can be divided according to the relative position of each grid unit to the nearest obstacle. The position of the object point determines the weighted value of each grid cell. Specifically, each grid unit may be determined according to the radial distance from each grid unit in the uniform segmentation network to the camera origin, and the radial distance from the nearest obstacle point in the direction of the grid unit to the camera origin. The weighted value of the unit. Referring to FIG. 9 , each arrow represents a radial direction, and a respective weighting value is determined for the grid cells in each radial direction. The calculation of the weighted value is determined according to the radial distance of the grid unit to the camera origin, and the radial distance of the obstacle point to the camera origin in the direction of the grid unit. Specifically, the calculation of the weighted value will be illustrated in the following embodiments. When each grid cell obtains a weighted value, a truncated directional distance field with the nearest obstacle point as the truncated surface is formed. According to the directional distance relative to the truncated surface, each grid cell in the truncated effective distance field can be calculated. Weight calculation. After the weighted value of each grid unit is obtained, a new weight of each grid unit is determined according to the initial weight of each grid unit and the weighted value, wherein the new weight is greater than or equal to the minimum value The weight threshold is less than or equal to the maximum weight threshold, and the initial weight of the grid unit is 0 or a new weight determined when the drivable area is determined for the previous frame image. It should be understood that the maximum weight threshold and the minimum weight threshold are used to limit the new weight to avoid an infinite increase of the new weight. For example, assuming that in multiple consecutive frames, the weight value of the grid unit corresponding to the garage wall remains +1, then its new weight value will not increase after it increases to 10 (the maximum weight threshold).

Then, a drivable area in the shooting area of the target image may be determined according to the first type of grid unit or the second type of grid unit indicated by the new weight, wherein the first type of grid unit indicated by the new weight There is the closest obstacle point between the grid-like unit and the camera origin, and there is no closest obstacle point between the second grid unit indicated by the new weight and the camera origin. It can be understood that the grid cells can be classified according to the value of the new weight, and it is determined whether each grid cell is a first-type grid cell or a second-type grid cell. For example, according to the new weight of the grid unit, it may be determined that the grid unit is a first-type grid unit or a second-type grid unit, wherein the first-type grid unit and the camera origin There is the closest obstacle point in between, and there is no closest obstacle point between the second type of grid cell and the camera origin. The second type of grid cell can be understood as the grid cell between the nearest obstacle point and the camera origin. Finally, a drivable area in the shooting area of the target image is determined according to the second type of grid unit, wherein the second type of grid unit adjacent to the first type of grid unit is the the boundary of the drivable area. It should be understood that the second type of grid cells can form a drivable area in the shooting area of the target image after being merged.

In the above-mentioned embodiment, before determining the new weight value of each grid unit according to the initial weight value and the weight value of each grid unit, each grid unit corresponding to the target image may also be determined first. the initial weight of . The initial weight of each grid unit of the target image is based on the camera relative pose of the matching image and the target image, and the truncated directional distance field corresponding to the matching image is rotated and translated into the truncated directional distance field corresponding to the target image to determine the matching. Correspondence between the grid unit corresponding to the image and the grid unit corresponding to the target image. The new weight determined in the grid unit corresponding to the matching image is the initial weight of the grid unit corresponding to the target image.

If the target image is the first frame image of consecutive frame images, the initial weight is 0.

In the above-mentioned embodiment, by calculating the weighted value of each grid unit in the uniform segmentation network of the target image shooting area, the weighted value of the grid unit is updated, and the new weighted value of the grid unit is continuously updated with the continuous updating of continuous image frames, It can not only smooth the noise, but also make up for the problem of missed detection of the nearest obstacle point in a single frame, and improve the reliability of the detection of the drivable area.

In the above embodiment, the implementation manner of determining the weighted value of each grid unit may be, for example:

If the radial distance between the grid unit in the uniform segmentation network and the camera origin, minus the difference between the radial distance between the nearest obstacle point in the direction of the grid unit and the camera origin, is greater than or equal to the upper limit of the distance threshold, Then the weighted value of the grid unit is the first value.

If the radial distance between the closest obstacle point and the camera origin, minus the difference between the radial distance between the grid unit and the camera origin in the direction of the closest obstacle point, is less than or equal to the lower threshold of the distance, then the grid The weighted value of the cell is the second value.

If the radial distance between the closest obstacle point and the camera origin, minus the difference between the radial distance between the grid unit and the camera origin in the direction of the closest obstacle point, is less than the distance upper threshold and greater than the distance lower limit the threshold value, the weighted value of the grid unit is a third value, where the third value is a value determined according to the difference value and a preset smooth continuous function.

Wherein, the distance upper threshold is the inverse of the distance lower threshold, the first value is the inverse of the second value, and the absolute value of the third value is less than the absolute value of the distance upper threshold or The absolute value of the distance lower threshold.

As an example, the weighted value f(c) of the grid cells in the specified direction can be determined according to the following formula 1:

where c is the radial distance from the grid unit to the camera origin, and d is the radial distance from the closest obstacle point in the grid direction to the camera origin. In formula 1, the upper threshold of the distance is 0.3, the lower threshold of the distance is -0.3, the first value is 1, the second value is -1, and the smooth continuous function is a preset trigonometric function.

Referring to FIG. 10 , it is a schematic diagram of a drivable area provided by an embodiment of the present application. The shaded area in Figure 10 is the drivable area. The drivable area boundary shown in FIG. 10 is the result obtained by the cumulative weighted update of the weighted values of the multi-frame images, which is a further optimization of the position marked by the nearest obstacle point shown in FIG. 8 . The drivable area boundary shown in Figure 10 combines the grid cell weights determined by the nearest obstacles in the target image and the initial weights accumulated in the previous consecutive frames, which has high reliability.

The above embodiment realizes the determination of the weighted value of each grid unit, and the smooth transition of the grid unit with the difference between the distance upper threshold and the distance lower threshold through a smooth continuous function, further reduces the multi-frame fusion weighted accumulation process. noise, which improves the reliability of the determination of the new weights of the grid cells, and further improves the reliability of the detection of the drivable area.

The embodiment shown in FIG. 1 acquires a matching image and a target image obtained by shooting the surrounding area of the vehicle with a vehicle-mounted monocular camera, wherein the matching image is the previous frame image that is continuous with the target image; and then according to the Performing three-dimensional reconstruction on the target image by matching the image to obtain the target three-dimensional point cloud data corresponding to the target image, so as to reconstruct the three-dimensional point cloud data of the continuous frame images collected by the monocular camera; then according to the target three-dimensional point cloud data , determine the nearest obstacle point in the shooting area of the target image, and determine the drivable area in the shooting area of the target image according to each of the nearest obstacle points, without collecting three-dimensional images or a large number of image samples for machine learning , reducing the dependence on the image acquisition method, reducing the processing difficulty, and improving the accuracy and reliability of detection.

Referring to FIG. 11 , it is a schematic structural diagram of a drivable area detection device provided by an embodiment of the present application. The drivable area detection device 30 shown in FIG. 11 includes:

The image acquisition module 31 is configured to acquire a matching image and a target image captured by the vehicle-mounted monocular camera on the surrounding area of the vehicle, wherein the matching image is a previous frame image of the target image.

The first processing module 32 is configured to perform three-dimensional reconstruction on the target image according to the matching image, and obtain target three-dimensional point cloud data corresponding to the target image.

The second processing module 33 is configured to determine the nearest obstacle point in the shooting area of the target image according to the target three-dimensional point cloud data.

The third processing module 34 is configured to determine a drivable area in the shooting area of the target image according to each of the nearest obstacle points.

The drivable area detection device provided by this embodiment acquires a matching image and a target image collected by shooting the surrounding area of the vehicle with a vehicle-mounted monocular camera, wherein the matching image is a previous frame image that is continuous with the target image Then carry out three-dimensional reconstruction of the target image according to the matching image, and obtain the target three-dimensional point cloud data corresponding to the target image, thereby realizing the reconstruction of three-dimensional point cloud data for the continuous frame images collected by the monocular camera; then according to the Target three-dimensional point cloud data, determine the nearest obstacle point in the shooting area of the target image, and determine the drivable area in the shooting area of the target image according to each of the nearest obstacle points, without collecting three-dimensional images or a large number of Machine learning is performed on image samples, which reduces the dependence on the image acquisition method, reduces the processing difficulty, and improves the accuracy and reliability of detection.

In some embodiments, the first processing module 32 is specifically configured to project the matching image onto a plurality of preset projection surfaces according to the viewing angle of the target image to obtain a plurality of projection images, wherein each of the The projection surface corresponds to a depth relative to the origin of the camera; according to the matching cost of the pixels in the target image and the corresponding pixels in the plurality of projection images, the estimated depth of the pixels in the target image is determined; The estimated depth of the pixel is obtained, and the target 3D point cloud data corresponding to the target image is obtained.

In this embodiment, the projected images of the matched images on the projection surfaces of different depths are used to achieve depth recovery of the target image at cost matching, and the target three-dimensional point cloud data is obtained by transformation, which improves the accuracy and reliability of the three-dimensional reconstruction of the monocular image. Thus, the accuracy and reliability of the drivable area detection are improved.

In some embodiments, the projection planes include: N1 vertical projection planes; the N1 vertical projection planes are parallel to the front of the camera, and the distances from the camera origin to the N1 vertical projection planes are equal to Inversely proportional equal difference distribution, where N1 is an integer greater than 1. This embodiment realizes the depth recovery of the area in front of the camera through the vertical projection plane, improves the accuracy of depth recovery in complex environments such as curves, and further improves the accuracy and reliability of the three-dimensional reconstruction of the target image.

In some embodiments, the projection surface further includes: N2 horizontal projection planes and/or N3 projection spherical surfaces; wherein the N2 horizontal projection planes are parallel to the ground directly below the camera, and the N2 horizontal projection planes Evenly arranged in the ground distribution range with the ground as the center of symmetry, where N2 is an integer greater than 1; the N3 projection spheres are concentric spheres with the camera origin as the center of the sphere, and the N3 projection spheres are The radius of the sphere is inversely proportional and equally distributed, where N3 is an integer greater than 1. In this embodiment, the depth of the ground area in the target image is recovered through the horizontal projection plane, and more normal samples are introduced through the projection sphere to improve the accuracy and reliability of depth recovery. For the points in the target image that are neither on the horizontal nor vertical planes, the depth recovery accuracy of these points can be improved by increasing the normal sampling by combining with the projection sphere. In addition, the introduction of a parallel projection sphere can also provide a favorable projection surface for the fisheye target image with a viewing angle greater than 180 degrees, and improve the accuracy and reliability of depth recovery.

In some embodiments, the first processing module 32 is specifically configured to acquire target pixel window characteristics of pixels in the target image; acquire projection pixel window characteristics of corresponding pixels in the multiple projection images; feature and the feature of the projected pixel window, obtain the matching cost between the pixel in the target image and the corresponding pixel in each of the projected images; take the depth corresponding to the corresponding pixel with the smallest matching cost as the depth of the corresponding pixel in the target image. The estimated depth of the pixel, wherein the depth corresponding to the corresponding pixel is the depth corresponding to the projection surface where the corresponding pixel is located.

This embodiment improves the accuracy of restoring the depth of pixels in the target image by using the depth corresponding to the corresponding pixel with the smallest matching cost as the estimated depth of the pixel in the target image.

In some embodiments, the first processing module 32, before determining the estimated depth of the pixels in the target image according to the matching cost of the pixels in the target image and the corresponding pixels in the plurality of projection images, further use according to the camera relative poses of the target image and the matching image, to determine the corresponding pixels in the matching image that correspond to the pixels in the target image one-to-one; according to the corresponding pixels in the matching image, determine The corresponding pixels in each of the projection images correspond to the pixels in the target image one-to-one.

This embodiment improves the accuracy and reliability of the depth recovery of the target image by matching the corresponding pixels in the image and the target image through the relative pose positioning of the camera.

In some embodiments, the first processing module 32 determines, according to the camera relative poses of the target image and the matching image, corresponding pixels in the matching image that correspond to pixels in the target image one-to-one Before, it is also used to collect the wheel speedometer data of the rear wheel of the vehicle and the IMU data of the vehicle-mounted inertial measurement unit, wherein the wheel speedometer data of the rear wheel of the vehicle indicates the horizontal movement distance of the vehicle-mounted monocular camera, and the vehicle-mounted The IMU data indicates the horizontal movement direction of the vehicle-mounted monocular camera; the camera pose data of the vehicle-mounted monocular camera is determined according to the wheel speedometer data of the rear wheels of the vehicle and the vehicle-mounted IMU data; according to the vehicle-mounted monocular camera The camera pose data of the monocular camera is used to determine the relative camera pose of the target image and the matching image.

In this embodiment, since there is no relative rotation between the rear wheel and the body in the horizontal direction, the wheel speed of the rear wheel can directly represent the moving speed of the body, and the camera pose is obtained by combining the wheel speedometer data of the rear wheel of the vehicle and the data of the vehicle IMU, which improves the performance of the camera. The reliability of the camera pose, thereby improving the accuracy and reliability of the drivable area detection.

In some embodiments, the second processing module 33 is configured to, according to the target three-dimensional point cloud data and the polar coordinate grid network horizontally segmenting the shooting area of the target image, determine the The number of obstacle points contained in each grid, wherein the obstacle point is the target three-dimensional point whose height above the ground is greater than the preset obstacle height threshold; according to the nearest obstacle in each sector partition of the polar coordinate grid network The object grid is used to determine the nearest obstacle points in each direction in the shooting area of the target image, wherein the nearest obstacle grid is the radial distance closest to the camera origin in the fan-shaped partition and contains obstacles A raster whose number of object points is greater than the preset number threshold. Specifically, for example, according to the target 3D point cloud data and the camera origin position corresponding to the target image, the target 3D point in each grid of the preset polar coordinate grid network is determined, wherein the polar coordinate grid The grid network is a segmentation network formed by superimposing and segmenting the first type of cutting planes arranged in a fan shape and the second type of cutting planes facing the camera and arranged in parallel for the shooting area of the target image. The intersection of the first type of cutting planes Collinear with the vertical line to the ground of the camera origin; determine the number of obstacle points contained in each of the grids, wherein the obstacle points are the target three-dimensional points whose height to the ground is greater than the preset obstacle height threshold; In the sector partition divided by the first type of cutting plane, determine the nearest obstacle grid, wherein the nearest obstacle grid is the radial distance closest to the camera origin in the sector partition, and the obstacles included A grid with a number of points greater than a preset number threshold; and according to the nearest obstacle grid, determine the nearest obstacle point in the shooting area of the target image.

In this embodiment, the polar coordinate grid network established for the camera origin is used to spatially divide the shooting area of the target image, and the grids that may be obstacles and the smallest radial distance from the camera origin in the fan-shaped partitions in each direction are extracted for use in The nearest obstacle points in all directions of the origin of the camera are determined, which improves the accuracy of the nearest obstacle points, thereby improving the accuracy and reliability of the drivable area detection.

In some embodiments, the second processing module 33 is configured to obtain the average position point of the obstacle points included in the nearest obstacle grid; the average position point corresponding to each of the nearest obstacle grids is used as the The nearest obstacle point in the shooting area of the target image.

This embodiment further optimizes the position of the nearest obstacle point and improves the accuracy of the obstacle point.

In some embodiments, the third processing module 34 is configured to, in the uniform segmentation network for horizontally segmenting the shooting area of the target image, according to the relative position of each grid unit in the uniform segmentation network to the nearest obstacle The weighted value of each grid unit is determined according to the position of the point, and the new weight of each grid unit is determined according to the initial weight of each grid unit and the weighted value, wherein the new weight is greater than or equal to the minimum weight threshold, and less than or equal to the maximum weight threshold, the initial weight of the grid unit is 0 or a new weight determined when the drivable area is determined for the previous frame image; according to the new weight The indicated first-type grid unit or the second-type grid unit determines a drivable area in the shooting area of the target image, wherein the first-type grid unit indicated by the new weight is related to the camera origin There is the closest obstacle point in between, and there is no closest obstacle point between the second type of grid unit indicated by the new weight and the camera origin. For example, specifically, according to the closest obstacle point and the camera origin position corresponding to the target image, obtain a uniform segmentation network including the closest obstacle point, where the uniform segmentation network is the shooting area of the target image A segmentation network formed by dividing a uniform square grid in the horizontal direction; according to the radial distance of each grid unit in the uniform segmentation network to the camera origin, and the distance between the nearest obstacle point in the grid unit direction to the camera origin The radial distance is used to determine the weighted value of each of the grid units; according to the initial weight of each of the grid units and the weighted value, the new weight of each of the grid units is determined, wherein the new weight is The value is greater than or equal to the minimum weight threshold and less than or equal to the maximum weight threshold, the initial weight of the grid unit is 0 or a new weight determined when the drivable area is determined for the previous frame image; according to the grid The new weight of the grid unit, which determines whether the grid unit is a first-type grid unit or a second-type grid unit, wherein there is the closest obstacle between the first-type grid unit and the camera origin object point, there is no nearest obstacle point between the second type grid unit and the camera origin; according to the second type grid unit, determine the drivable area in the shooting area of the target image, Wherein, the grid unit of the second type adjacent to the grid unit of the first type is the boundary of the drivable area.

In this embodiment, by calculating the weighted value of each grid unit in the uniform segmentation network of the target image shooting area, the weight value of the grid unit is updated. It can not only smooth the noise, but also make up for the problem of missed detection of the nearest obstacle point in a single frame, and improve the reliability of the detection of the drivable area.

In some embodiments, the third processing module 34 is specifically configured to subtract the closest obstacle point in the direction of the grid unit to the camera origin if the radial distance between the grid unit in the uniform segmentation network and the camera origin is If the difference between the radial distances is greater than or equal to the distance upper limit threshold, the weighted value of the grid unit is the first value; if the radial distance between the nearest obstacle point and the camera origin is subtracted from the nearest obstacle The difference between the radial distance between the grid unit and the camera origin in the direction of the object point is less than or equal to the lower threshold of the distance, then the weight of the grid unit is the second value; if the distance between the closest obstacle point and the camera origin is To the distance, minus the difference between the radial distance between the grid unit and the camera origin in the direction of the nearest obstacle point, if it is less than the upper threshold of the distance and greater than the lower threshold of the distance, then the weighted value of the grid unit is a third value, wherein the third value is a value determined according to the difference value and a preset smooth continuous function; wherein the distance upper threshold is the inverse of the distance lower threshold, and the first value is the inverse of the second value, and the absolute value of the third value is smaller than the absolute value of the distance upper threshold value or the absolute value of the distance lower threshold value.

This embodiment realizes the determination of the weighted value of each grid unit, and uses a smooth continuous function to smoothly transition the grid units whose difference is between the upper and lower distance thresholds, thereby further reducing the multi-frame fusion weighted accumulation process. noise, which improves the reliability of the determination of the new weights of the grid cells, and further improves the reliability of the detection of the drivable area.

In some embodiments, the matching image and the target image are both fisheye images. By adopting the fisheye image, the embodiment of the present application realizes the increase of the horizontal viewing angle, the expansion of the field of view of the image shooting area, and the improvement of the reliability of the detection of the drivable area.

According to the embodiments of the present application, the present application further provides an electronic device and a readable storage medium.

Referring to FIG. 12 , it is a block diagram of an electronic device of a drivable area detection method according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the application described and/or claimed herein.

As shown in FIG. 12, the electronic device includes: one or more processors 1201, a memory 1202, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or otherwise as desired. The processor may process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used with multiple memories and multiple memories, if desired. Likewise, multiple electronic devices may be connected, each providing some of the necessary operations (eg, as a server array, a group of blade servers, or a multiprocessor system). In FIG. 12, a processor 1201 is used as an example.

The memory 1202 is the non-transitory computer-readable storage medium provided by the present application. Wherein, the memory stores instructions executable by at least one processor, so that the at least one processor executes the drivable area detection method provided by the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions for causing the computer to execute the drivable area detection method provided by the present application.

As a non-transitory computer-readable storage medium, the memory 1202 can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as program instructions/modules corresponding to the drivable area detection method in the embodiments of the present application (for example, , the image acquisition module 31, the first processing module 32, the second processing module 33 and the third processing module 34 shown in FIG. 11). The processor 1201 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 1202, that is, implementing the method for detecting a drivable area in the above method embodiments.

The memory 1202 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the electronic device detected in the drivable area Wait. Additionally, memory 1202 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1202 may optionally include memory located remotely relative to the processor 1201, and these remote memories may be connected to the electronic device for drivable area detection via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The electronic device of the method for detecting a drivable area may further include: an input device 1203 and an output device 1204 . The processor 1201 , the memory 1202 , the input device 1203 and the output device 1204 may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 12 .

The input device 1203 can receive input numerical or character information, and generate key signal input related to user settings and function control of the electronic equipment for driving area detection, such as touch screen, keypad, mouse, trackpad, touchpad, pointing stick , one or more mouse buttons, trackballs, joysticks and other input devices. Output devices 1204 may include display devices, auxiliary lighting devices (eg, LEDs), haptic feedback devices (eg, vibration motors), and the like. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.

Various implementations of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuit systems, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

These computational programs (also referred to as programs, software, software applications, or codes) include machine instructions for programmable processors, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages calculation program. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or apparatus for providing machine instructions and/or data to a programmable processor ( For example, magnetic disks, optical disks, memories, programmable logic devices (PLDs), including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present application can be performed in parallel, sequentially or in different orders, and as long as the desired results of the technical solutions disclosed in the present application can be achieved, no limitation is imposed herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims (18)

1. A travelable region detection method, comprising:

acquiring a matching image and a target image acquired by shooting a surrounding area of a vehicle by using a vehicle-mounted monocular camera, wherein the matching image is a previous frame image continuous with the target image;

performing three-dimensional reconstruction on the target image according to the matching image to obtain target three-dimensional point cloud data corresponding to the target image;

determining the nearest barrier point in the shooting area of the target image according to the target three-dimensional point cloud data;

in a uniform segmentation network for horizontally segmenting the shooting area of the target image, determining the weighted value of each grid unit according to the radial distance of each grid unit in the uniform segmentation network to the origin of the camera and the radial distance of the nearest barrier point in the direction of the grid unit to the origin of the camera;

determining a new weight of each grid cell according to the initial weight of each grid cell and the weighted value, wherein the new weight is greater than or equal to a minimum weight threshold and less than or equal to a maximum weight threshold, and the initial weight of each grid cell is 0 or is a new weight determined when a travelable area is determined for a previous frame of image;

and determining a travelable area in the shooting area of the target image according to the first type of grid cells or the second type of grid cells indicated by the new weight, wherein the nearest barrier point is arranged between the first type of grid cells indicated by the new weight and the camera origin, and the nearest barrier point is not arranged between the second type of grid cells indicated by the new weight and the camera origin.

2. The method of claim 1, wherein the three-dimensional reconstruction of the target image according to the matching image to obtain target three-dimensional point cloud data corresponding to the target image comprises:

projecting the matched image onto a plurality of preset projection surfaces according to the visual angle of the target image to obtain a plurality of projection images, wherein each projection surface corresponds to a depth relative to the origin of the camera;

determining the estimated depth of the pixels in the target image according to the matching cost of the pixels in the target image and the corresponding pixels in the plurality of projection images;

and acquiring target three-dimensional point cloud data corresponding to the target image according to the estimated depth of the pixels in the target image.

3. The method of claim 2, wherein the projection surface comprises: n1 vertical projection planes;

the N1 vertical projection planes are parallel to the right opposite of the camera, and the distances from the camera origin to the N1 vertical projection planes are in an inversely proportional equal difference distribution, wherein N1 is an integer greater than 1.

4. The method of claim 3, wherein the projection surface further comprises: n2 horizontal projection planes and/or N3 projection spheres;

the N2 horizontal projection planes are parallel to the ground right below the camera, and the N2 horizontal projection planes are uniformly arranged in the ground distribution range taking the ground as a symmetry center, wherein N2 is an integer larger than 1;

the N3 projection spherical surfaces are concentric spherical surfaces taking the camera origin as the sphere center, and the radiuses of the N3 projection spherical surfaces are in inverse proportion equal difference distribution, wherein N3 is an integer larger than 1.

5. The method of any of claims 2 to 4, wherein determining the estimated depth of the pixel in the target image based on the cost of matching the pixel in the target image to the corresponding pixel in the plurality of projection images comprises:

acquiring target pixel window characteristics of pixels in the target image;

acquiring projection pixel window characteristics of corresponding pixels in the plurality of projection images;

according to the target pixel window characteristic and the projection pixel window characteristic, obtaining the matching cost of the pixel in the target image and the corresponding pixel in each projection image;

and taking the depth corresponding to the corresponding pixel with the minimum matching cost as the estimated depth of the pixel in the target image, wherein the depth corresponding to the corresponding pixel is the depth corresponding to the projection plane where the corresponding pixel is located.

6. The method according to claim 1, wherein the determining a nearest obstacle point in a shooting area of the target image according to the target three-dimensional point cloud data comprises:

determining the number of barrier points contained in each grid in the polar coordinate grid network according to the target three-dimensional point cloud data and the polar coordinate grid network which horizontally divides the shooting area of the target image, wherein the barrier points are target three-dimensional points with the ground height larger than a preset barrier height threshold;

and determining the nearest barrier points in each direction in the shooting area of the target image according to the nearest barrier grid in each fan-shaped partition of the polar coordinate grid network, wherein the nearest barrier grid is the grid which has the closest radial distance with the origin of the camera in the fan-shaped partition and contains barrier points with the number larger than a preset number threshold.

7. The method of claim 6, wherein determining nearest obstacle points in each direction in the capture area of the target image from the nearest obstacle grids in each sector of the polar grid mesh network comprises:

obtaining the average position point of the obstacle points contained in the nearest obstacle grid;

and taking the average position point corresponding to each nearest obstacle grid as the nearest obstacle point in the shooting area of the target image.

8. The method of claim 1, 2, 3, 4, 6 or 7, wherein the matching image and the target image are both fisheye images.

9. A travelable region detection apparatus, characterized by comprising:

the system comprises an image acquisition module, a matching image acquisition module and a target image acquisition module, wherein the image acquisition module is used for acquiring the matching image and the target image acquired by shooting the surrounding area of a vehicle by using a vehicle-mounted monocular camera, and the matching image is the previous frame image of the target image;

the first processing module is used for carrying out three-dimensional reconstruction on the target image according to the matching image to obtain target three-dimensional point cloud data corresponding to the target image;

the second processing module is used for determining a nearest barrier point in a shooting area of the target image according to the target three-dimensional point cloud data;

the third processing module is used for determining the weighted value of each grid unit in an evenly-divided network for horizontally dividing the shooting area of the target image according to the radial distance of each grid unit in the evenly-divided network to the camera origin and the radial distance of the nearest barrier point in the direction of the grid unit to the camera origin; determining a new weight of each grid cell according to the initial weight of each grid cell and the weighted value, wherein the new weight is greater than or equal to a minimum weight threshold and less than or equal to a maximum weight threshold, and the initial weight of each grid cell is 0 or is a new weight determined when a travelable area is determined for a previous frame of image; and determining a travelable area in the shooting area of the target image according to the first type of grid cells or the second type of grid cells indicated by the new weight, wherein the nearest barrier point is arranged between the first type of grid cells indicated by the new weight and the origin of the camera, and the nearest barrier point is not arranged between the second type of grid cells indicated by the new weight and the origin of the camera.

10. The apparatus according to claim 9, wherein the first processing module is specifically configured to project the matching image onto a plurality of preset projection surfaces according to the view angle of the target image, so as to obtain a plurality of projection images, where each projection surface corresponds to a depth relative to an origin of a camera; determining the estimated depth of the pixels in the target image according to the matching cost of the pixels in the target image and the corresponding pixels in the plurality of projection images; and acquiring target three-dimensional point cloud data corresponding to the target image according to the estimated depth of the pixels in the target image.

11. The apparatus of claim 10, wherein the projection surface comprises: n1 vertical projection planes; the N1 vertical projection planes are parallel to the right opposite of the camera, and the distances from the camera origin to the N1 vertical projection planes are in an inversely proportional equal difference distribution, wherein N1 is an integer greater than 1.

12. The apparatus of claim 11, wherein the plane of projection further comprises: n2 horizontal projection planes and/or N3 projection spheres; the N2 horizontal projection planes are parallel to the ground right below the camera, and the N2 horizontal projection planes are uniformly arranged in the ground distribution range taking the ground as a symmetry center, wherein N2 is an integer larger than 1; the N3 projection spherical surfaces are concentric spherical surfaces taking the camera origin as the sphere center, and the radiuses of the N3 projection spherical surfaces are in inverse proportion equal difference distribution, wherein N3 is an integer larger than 1.

13. The apparatus according to any one of claims 10 to 12, wherein the first processing module is specifically configured to obtain a target pixel window characteristic of a pixel in the target image; acquiring projection pixel window characteristics of corresponding pixels in the plurality of projection images; according to the target pixel window characteristic and the projection pixel window characteristic, obtaining the matching cost of the pixel in the target image and the corresponding pixel in each projection image; and taking the depth corresponding to the corresponding pixel with the minimum matching cost as the estimated depth of the pixel in the target image, wherein the depth corresponding to the corresponding pixel is the depth corresponding to the projection plane where the corresponding pixel is located.

14. The apparatus according to claim 9, wherein the second processing module is configured to determine, according to the target three-dimensional point cloud data and a polar grid network horizontally dividing a shooting area of the target image, a number of obstacle points included in each grid in the polar grid network, where the obstacle points are target three-dimensional points whose height to ground is greater than a preset obstacle height threshold; and determining the nearest barrier points in each direction in the shooting area of the target image according to the nearest barrier grid in each fan-shaped partition of the polar coordinate grid network, wherein the nearest barrier grid is the grid which has the closest radial distance with the origin of the camera in the fan-shaped partition and contains barrier points with the number larger than a preset number threshold.

15. The apparatus according to claim 14, wherein the second processing module 33 is configured to obtain an average position point of obstacle points included in the nearest obstacle grid; and taking the average position point corresponding to each nearest obstacle grid as the nearest obstacle point in the shooting area of the target image.

16. The apparatus of claim 9, 10, 11, 12, 14 or 15, wherein the matching image and the target image are both fisheye images.

17. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the travelable region detection method of any of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the travelable region detection method of any of claims 1 to 8.

Images