CN114994699A - Vehicle-road cooperation based category recall perception identification method, device, equipment and medium - Google Patents

Abstract

The invention discloses a perception identification method of category recall based on vehicle-road cooperation, which relates to the technical field of vehicle-road cooperation and comprises the steps of scanning through a road-end three-dimensional laser radar to obtain point cloud data, and shooting through a road-end camera to obtain a two-dimensional picture; clustering point cloud data; carrying out three-dimensional detection on the point cloud data, and recording unidentified point cloud clusters; calibrating a road end three-dimensional laser radar and a road end camera to obtain a conversion matrix; obtaining the positions of all unidentified point clouds in the two-dimensional picture through the transformation matrix, and intercepting the point clouds by using a minimum rectangular surrounding frame to obtain a picture block; and classifying the picture blocks through a classification network model to obtain the class label of each picture block and the orientation angle information of the head of each picture block. According to the invention, the three-dimensional missed-detection object of the laser radar is recalled by using the two-dimensional picture of the camera, so that the recall rate and accuracy of sensing and identification can be improved, and the cost of a vehicle body sensor can be effectively saved.

Description

A perceptual recognition method, device, device and medium for category recall based on vehicle-road collaboration

technical field

The present invention relates to the technical field of vehicle-road collaboration, in particular to a perceptual identification method, device, device and medium based on vehicle-road collaboration based on category re-recall.

Background technique

Under the recognition of sensors at the road or vehicle end, whether the existing automatic driving technology uses 2D detection alone or 3D detection alone, it is easy to generate occluded objects, distant and small objects, objects with unclear laser point cloud texture, etc. False detections and missed detections.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art, and to provide a perceptual identification method, device, device and medium based on vehicle-road coordination-based recall of categories.

In order to solve the above technical problems, the technical scheme of the present invention is as follows:

A perceptual recognition method based on vehicle-road collaboration based on category re-recall, including,

The point cloud data is obtained by scanning the roadside 3D lidar, and the 2D picture is captured by the roadside camera;

Clustering the point cloud data to obtain several point cloud clusters that are separated from each other;

using a three-dimensional detection algorithm to perform three-dimensional detection on the point cloud data, and record the point cloud clusters not identified by the three-dimensional detection algorithm;

Calibrate the road-side 3D lidar and road-side camera to obtain the transformation matrix from point cloud to image;

Obtain the positions of all the point cloud clusters in the two-dimensional picture that are not identified by the three-dimensional detection algorithm through the transformation matrix, and intercept them from the two-dimensional picture with the smallest rectangular bounding box to obtain several picture blocks;

The picture blocks are classified by a classification network model, and the category label of each picture block and the orientation angle information of its head are obtained.

As a preferred solution of the perceptual recognition method based on vehicle-road collaboration based on category re-recall of the present invention, wherein: the three-dimensional detection algorithm includes a Center Point algorithm.

As a preferred solution of the perceptual recognition method based on vehicle-road collaboration based on category re-recall of the present invention, wherein: the three-dimensional detection algorithm is used to perform three-dimensional detection on the point cloud data, and the record is not recognized by the three-dimensional detection algorithm. The said point cloud cluster includes,

Use the Center Point algorithm to perform three-dimensional detection on the point cloud data, obtain several types of three-dimensional bounding boxes, and assign corresponding category labels to each three-dimensional bounding box;

The point cloud that did not get a class label was recorded.

As a preferred solution of the perceptual recognition method based on vehicle-road collaboration based on category re-recall of the present invention, wherein: obtaining all the point clouds in the two-dimensional picture that are not identified by the three-dimensional detection algorithm through the transformation matrix The locations include,

Calculate the position of the point cloud in the two-dimensional picture by the formula [u,v,1]=M _t *[x,y,z,1], where u and v indicate that the point cloud is in the two-dimensional picture The pixel position of , x, y, z represent the three-dimensional coordinates of the point cloud, and M _t represents the transformation matrix from the point cloud to the image.

As a preferred solution of the perceptual recognition method based on vehicle-road collaboration based on category re-recall of the present invention, wherein: the classification network module includes an Efficient Net network model.

The invention also discloses a perceptual recognition device based on vehicle-road collaboration for category re-recall, comprising:

Sensor modules, including road-side 3D lidar and road-side cameras, are used to scan point cloud data and capture 2D pictures respectively;

The point cloud clustering module is used for clustering the point cloud data scanned by the three-dimensional laser radar at the roadside to obtain several point cloud clusters separated from each other;

a three-dimensional detection module, configured to perform three-dimensional detection on the point cloud data scanned by the road-end three-dimensional laser radar, and record the point cloud clusters that are not identified by the three-dimensional detection algorithm;

The calibration module is used to calibrate the road-side 3D lidar and road-side camera, and obtain the transformation matrix from point cloud to image;

The calculation module is used to calculate the positions of all the point cloud clusters in the two-dimensional picture that are not identified by the three-dimensional detection algorithm through the transformation matrix, and intercept them from the two-dimensional picture with the smallest rectangular bounding box to obtain several picture blocks;

The classification module is used for classifying the picture blocks through a classification network model to obtain the category label of each of the picture blocks and the orientation angle information of its head.

The present invention also discloses a computer device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor. Synergistic Class Recall for Perceptual Recognition Methods.

The present invention also discloses a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in the above-mentioned perceptual identification method based on vehicle-road coordination based on category recall is implemented.

The beneficial effects of the present invention are:

(1) The present invention uses the two-dimensional image of the camera to recall the three-dimensional missed object of the lidar, effectively combines the complementary characteristics of the 2D and 3D data, uses the 3D clustering result, and then projects it into the 2D to locate the object position, Finally, mandatory classification recall is carried out, which improves the robustness and accuracy of the algorithm and enhances the safety factor of driving.

(2) The present invention uses road-end sensors. Compared with vehicle body sensors, the present invention reduces unstable factors such as jitter and ghosting of sensor data under high-speed driving, improves the accuracy and stability of detection and recognition, and ensures driving safety.

(3) The present invention is mainly based on the sensors at the road end, which takes advantage of the unique advantages of vehicle-road coordination and reduces the equipment cost of the body sensors.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

1 is a schematic flowchart of a perceptual recognition method based on vehicle-road collaboration based on category re-recall provided by the present invention;

FIG. 2 is a schematic diagram of a perceptual recognition device based on vehicle-road collaboration based on category re-recall provided by the present invention;

FIG. 3 is a schematic diagram of a computer device provided by the present invention.

Detailed ways

In order to make the content of the present invention easier to understand clearly, the present invention will be further described in detail below according to specific embodiments and in conjunction with the accompanying drawings.

FIG. 1 is a schematic flowchart of a perceptual recognition method based on vehicle-road collaboration based on category re-recall provided by an embodiment of the present application. The method includes steps S101 to S106, and the specific steps are described as follows:

Step S101 : obtaining point cloud data by scanning the road-side 3D lidar, and simultaneously obtaining a two-dimensional picture by photographing by the road-side camera.

Specifically, a frame of original point cloud data is obtained by scanning the road-side 3D lidar, which is called PCD. At the same time, the roadside camera captures a two-dimensional picture. That is, the 2D image captured by the roadside camera and the point cloud data scanned by the roadside 3D lidar are in the same scene and at the same time.

Step S102: Clustering the point cloud data to obtain several point cloud clusters separated from each other.

Specifically, the original point cloud data PCD obtained by scanning the roadside 3D lidar is clustered, and the purpose is to obtain several areas separated from each other. It is approximately considered that the point cloud clusters in the same area belong to the same physical object. Therefore, several separate point cloud clusters can be obtained by clustering the original point cloud data PCD.

Step S103: Use a three-dimensional detection algorithm to perform three-dimensional detection on the point cloud data, and record the point cloud clusters that are not identified by the three-dimensional detection algorithm.

In this embodiment, three-dimensional detection is performed on the above-mentioned original point cloud data PCD using the Center Point algorithm, which is a three-dimensional point cloud detection algorithm, and three types of three-dimensional bounding boxes (3D Bounding Box) can be obtained, namely Vehicle (vehicle), Pedestrian (pedestrian), Cyclist (cyclist), and the corresponding three-dimensional bounding box can be assigned a category label. That is, for the clustering result in step S102, a three-dimensional bounding box can be obtained for each point cloud, and a corresponding category label is assigned to each three-dimensional bounding box. If a point cloud is detected as a pedestrian, a 3D bounding box can be generated for the point cloud and a Pedestrian label is assigned. For occluded objects, distant and small objects, objects with unclear laser point cloud texture and other point cloud clusters, the Center Point algorithm cannot identify them, then these point cloud clusters do not get the corresponding category labels, and it is necessary to identify these unrecognized points. Clouds are recorded individually.

Step S104 : calibrating the road-side 3D lidar and the road-side camera to obtain a transformation matrix from the point cloud to the picture.

Specifically, by calibrating the road-side 3D lidar and road-side camera, a transformation matrix (lidar to image) from point cloud to image can be obtained. The dimension of the matrix is 3*4, which is denoted as M _t .

Step S105: Obtain the positions of all the point clouds in the two-dimensional picture that are not identified by the three-dimensional detection algorithm through the transformation matrix, and intercept them from the two-dimensional picture with a minimum rectangular bounding box to obtain several pictures piece.

Specifically, all point clouds without labels are traversed, and the position of each point cloud group in the two-dimensional picture is obtained through the transformation matrix obtained by calibration in step S104. The specific conversion method is:

Calculate the position of the point cloud cluster in the two-dimensional image by the formula [u,v,1]=M _t *[x,y,z,1], where u and v represent the pixels of the point cloud in the two-dimensional image Point position, x, y, z represent the three-dimensional coordinates of the point cloud, and M _t represents the transformation matrix from the point cloud to the image.

After obtaining the position of each point cloud in the two-dimensional picture, the minimum rectangular bounding box is used to cut it out from the two-dimensional picture, and several picture blocks are obtained.

Step S106: Classify the picture blocks by using a classification network model to obtain the category label of each of the picture blocks and the orientation angle information of the head thereof.

In this embodiment, a lightweight classification network model Efficient Net is used to classify each picture block, to obtain the category label of each picture block, and at the same time, the orientation angle information of its head can be obtained. Preferably, using a lightweight network model can effectively reduce the computational overhead and improve the computational speed.

Therefore, the above-mentioned perceptual recognition method based on vehicle-road collaboration based on category re-recall uses the two-dimensional image of the camera to re-recall the three-dimensional missed detection objects of the lidar, effectively combining the complementary characteristics of 2D and 3D data, and using the 3D clustering results. , and then project it into 2D to locate the object position, and finally perform mandatory classification and recall, which improves the robustness and accuracy of the algorithm and enhances the safety factor of driving.

FIG. 2 is a schematic diagram of a perceptual recognition device for category recall based on vehicle-road coordination provided by an embodiment of the present application. The device includes a sensor module, a point cloud clustering module, a three-dimensional detection module, a calibration module, a calculation module and a classification module.

Specifically, the sensor module includes a road-side three-dimensional laser radar and a road-side camera, which are respectively used for scanning to obtain point cloud data and shooting to obtain two-dimensional pictures.

The point cloud clustering module is used for clustering the point cloud data scanned by the roadside three-dimensional lidar to obtain several point cloud clusters separated from each other.

The three-dimensional detection module is used to perform three-dimensional detection on the point cloud data scanned by the road-end three-dimensional laser radar, and record the point cloud clusters that are not identified by the three-dimensional detection algorithm.

The calibration module is used to calibrate the road-side 3D lidar and road-side camera, and obtain the transformation matrix from point cloud to image.

The calculation module is used to calculate the positions of all point cloud clusters in the two-dimensional picture that are not identified by the three-dimensional detection algorithm through the transformation matrix, and intercept them from the two-dimensional picture with the smallest rectangular bounding box to obtain several pictures. piece.

The classification module is used for classifying the picture blocks through a classification network model to obtain the category label of each picture block and the orientation angle information of its head.

Referring to FIG. 3 , this embodiment also provides a computer device, and the components of the computer device may include, but are not limited to: one or more processors or processing units, a system memory, and connecting different system components (including a system memory and a processing unit) the bus.

A bus represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect ( PCI) bus.

A computer system/server typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer system/server, including both volatile and non-volatile media, removable and non-removable media.

System memory may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and/or cache memory. The computer device may further include other removable/non-removable, volatile/non-volatile computer system storage media. For example only, the storage system may be used to read and write to non-removable, non-volatile magnetic media. Disk drives for reading and writing to removable non-volatile magnetic disks (eg "floppy disks") and removable non-volatile optical disks (eg CD-ROM, DVD-ROM or other optical media) may be provided CD-ROM. In these cases, each drive may be connected to the bus through one or more data media interfaces. The memory may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present invention.

A program/utility having a set (at least one) of program modules, which may be stored, for example, in a memory, such program modules including, but not limited to, an operating system, one or more application programs, other program modules, and program data , each or some combination of these examples may include an implementation of a network environment. Program modules generally perform the functions and/or methods of the described embodiments of the present invention.

The computer device may also communicate with one or more external devices such as keyboards, pointing devices, displays, etc.). This communication can take place through an input/output (I/O) interface. Also, the computer device may communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network such as the Internet) through a network adapter.

The processing unit executes the functions and/or methods in the described embodiments of the present invention by running programs stored in the system memory.

The above-mentioned computer program may be provided in a computer storage medium, that is, the computer storage medium is encoded with a computer program, which, when executed by one or more computers, causes one or more computers to execute the programs shown in the foregoing embodiments of the present invention. The method flow and/or device operation.

With the development of time and technology, the meaning of media has become more and more extensive, and the dissemination of computer programs is no longer limited to tangible media, and can also be downloaded directly from the network. Any combination of one or more computer-readable media may be employed. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (a non-exhaustive list) of computer readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), Erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signal in baseband or as part of a carrier wave, with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .

Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present invention may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional Procedural programming language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect).

In addition to the above-mentioned embodiments, the present invention may also have other embodiments; all technical solutions formed by equivalent replacement or equivalent transformation fall within the protection scope of the present invention.

Claims (8)

1. A perception identification method of category recall based on vehicle-road cooperation is characterized in that: comprises the steps of (a) preparing a mixture of a plurality of raw materials,

scanning through a road end three-dimensional laser radar to obtain point cloud data, and shooting through a road end camera to obtain a two-dimensional picture;

clustering the point cloud data to obtain a plurality of point cloud clusters which are separated from each other;

carrying out three-dimensional detection on the point cloud data by using a three-dimensional detection algorithm, and recording the point cloud cluster which is not identified by the three-dimensional detection algorithm;

calibrating a road end three-dimensional laser radar and a road end camera to obtain a conversion matrix from point cloud to picture;

obtaining the positions of all the point clouds which are not identified by the three-dimensional detection algorithm in the two-dimensional picture through the conversion matrix, and intercepting the point clouds from the two-dimensional picture by using a minimum rectangular surrounding frame to obtain a plurality of picture blocks;

and classifying the picture blocks through a classification network model to obtain the class label of each picture block and the orientation angle information of the head of the picture block.

2. The vehicle-road coordination based category recall perception identification method according to claim 1, wherein: the three-dimensional detection algorithm comprises a Center Point algorithm.

3. The vehicle-road coordination based category recall perception identification method according to claim 2, wherein: the three-dimensional detection of the point cloud data using a three-dimensional detection algorithm and recording the point cloud clusters not identified by the three-dimensional detection algorithm includes,

performing three-dimensional detection on the Point cloud data by using a Center Point algorithm to obtain a plurality of types of three-dimensional surrounding frames, and endowing each three-dimensional surrounding frame with a corresponding type label;

recording the cloud of points for which no category label is available.

4. The vehicle-road coordination based category recall perception identification method according to claim 1, wherein: the positions of all the point clouds which are not identified by the three-dimensional detection algorithm in the two-dimensional picture are obtained through the conversion matrix,

by the formula [ u, v,1]=M _t *[x,y,z,1]Calculating the position of the point cloud cluster in the two-dimensional picture, wherein u and v represent the positions of pixel points of the point cloud in the two-dimensional picture, x, y and z represent three-dimensional coordinates of the point cloud, and M is the position of the point cloud in the two-dimensional picture _t A transformation matrix representing the point cloud to the picture.

5. The vehicle-road coordination based category recall perception identification method according to claim 1, wherein: the classification network module comprises an efficiency Net network model.

6. The utility model provides a perception recognition device of category recall based on vehicle and road is cooperative which characterized in that: comprises the steps of (a) preparing a mixture of a plurality of raw materials,

the sensor module comprises a road end three-dimensional laser radar and a road end camera, and is respectively used for scanning to obtain point cloud data and shooting to obtain a two-dimensional picture;

the point cloud clustering module is used for clustering point cloud data obtained by scanning the road end three-dimensional laser radar to obtain a plurality of point cloud clusters which are separated from each other;

the three-dimensional detection module is used for carrying out three-dimensional detection on point cloud data obtained by scanning the road end three-dimensional laser radar and recording the point cloud cluster which is not identified by a three-dimensional detection algorithm;

the calibration module is used for calibrating the road end three-dimensional laser radar and the road end camera to obtain a conversion matrix from point cloud to picture;

the computing module is used for obtaining the positions of all the point clouds which are not identified by the three-dimensional detection algorithm in the two-dimensional picture through the conversion matrix computing, and intercepting the point clouds from the two-dimensional picture by using a minimum rectangular surrounding frame to obtain a plurality of picture blocks;

and the classification module is used for classifying the picture blocks through a classification network model to obtain the class label of each picture block and the orientation angle information of the head of the picture block.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: the processor, when executing the program, implements the method of any of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that: the program when executed by a processor implements a method as claimed in any one of claims 1 to 5.

Images