CN116701909A - Multimodal data fusion method, device, computer equipment and storage medium - Google Patents

Abstract

The present application relates to a multimodal data fusion method, device, computer equipment and storage medium. The method includes: obtaining multi-modal data to be processed; the multi-modal data to be processed includes point cloud data to be processed and image data to be processed; calling a pre-built feature extraction model; the feature extraction model is obtained by analyzing samples In-depth mining of multi-modal data is obtained by training according to the results of in-depth mining; feature extraction is performed on the point cloud data to be processed and the image data to be processed through the feature extraction model to obtain the point cloud data to be processed The corresponding first deep feature data and the second deep feature data corresponding to the image data to be processed; the first deep feature data and the second deep feature data are fused through the feature extraction model to obtain the The deep fusion features corresponding to the multimodal data to be processed. The method can improve the fusion accuracy of multimodal data.

Description

Multimodal data fusion method, device, computer equipment and storage medium

technical field

The present application relates to the technical field of automatic driving, in particular to a multimodal data fusion method, device, computer equipment, storage medium and computer program product.

Background technique

The rapid development of sensor technologies such as cameras, lidars, and millimeter-wave radars has promoted the advancement of autonomous driving perception capabilities. The data collected by different sensors corresponds to different modes, which also reflects the perception of the real world from different angles by the automatic driving system. The image collected by the camera contains more texture information such as color, while the point cloud collected by the radar contains more comprehensive spatial position information. The fusion method based on point cloud modality and image modality can further promote the development of autonomous driving perception technology.

In the traditional way, point cloud data and image data are superimposed on features within the visible range to achieve multi-modal data fusion.

However, traditional methods cannot obtain target and scene information closer to the real world, resulting in low accuracy of multimodal data fusion.

Contents of the invention

Based on this, it is necessary to provide a multi-modal data fusion method, device, computer equipment, computer-readable storage medium and computer program product capable of improving the fusion accuracy of multi-modal data for the above technical problems.

In a first aspect, the present application provides a multimodal data fusion method. The method includes:

Obtain multimodal data to be processed; multimodal data to be processed includes point cloud data to be processed and image data to be processed;

Call a pre-built feature extraction model;

Feature extraction is performed on the point cloud data to be processed and the image data to be processed respectively through the feature extraction model, and the first deep feature data corresponding to the point cloud data to be processed and the second deep feature data corresponding to the image data to be processed are obtained;

The first deep feature data and the second deep feature data are fused through a feature extraction model to obtain deep fusion features corresponding to the multimodal data to be processed.

In one of the embodiments, before acquiring the multimodal data to be processed, the method further includes:

Obtain sample multimodal data; sample multimodal data includes sample point cloud data and sample image data;

Input the sample multimodal data into the deep learning model to be trained; the deep learning model includes a feature extraction layer, a deep mining layer and a perception layer;

Extract the sample point cloud features corresponding to the sample point cloud data and the sample image features corresponding to the sample image data through the feature extraction layer, and fuse the sample point cloud features and sample image features to obtain sample fusion features;

Through the deep mining layer, the sample point cloud features and sample image features are deeply mined to obtain the deep mining results;

Through the perception layer, the deep learning model is trained according to the sample fusion features and the deep mining results, and a pre-built feature extraction model is obtained.

In one of the embodiments, the deep mining result is a relationship loss value; through the deep mining layer, the sample point cloud features and sample image features are deeply mined, and the deep mining results obtained include:

Through the deep mining layer, the sample point cloud features and sample image features are deeply mined to obtain the mined feature data;

Determine the relationship loss value based on the mined feature data.

In one of the embodiments, the perception layer includes a perception task layer and a training optimization layer; through the perception layer, according to the sample fusion features and depth mining results, training the deep learning model includes:

Through the perception task layer, the preset perception tasks are executed according to the sample fusion features, and the task execution results are obtained;

Through the training optimization layer, the overall loss value of the deep learning model is determined according to the task execution result and the relationship loss value, and the deep learning model is trained according to the overall loss value.

In one of the embodiments, the preset perception task is a target detection task; determining the overall loss value of the deep learning model according to the task execution result and the relationship loss value by training the optimization layer includes:

Determine the position loss value, direction loss value and category loss value of the deep learning model according to the task execution results by training the optimization layer;

The overall loss value of the deep learning model is determined by training the optimization layer according to the position loss value, direction loss value, category loss value and relationship loss value.

In one of the embodiments, feature extraction is performed on the point cloud data to be processed and the image data to be processed through the feature extraction model to obtain the first deep feature data corresponding to the point cloud data to be processed and the second deep feature data corresponding to the image data to be processed Data includes:

Feature extraction is performed on the point cloud data to be processed through the feature extraction model to obtain first deep feature data corresponding to the point cloud data to be processed; the first deep feature data includes deep point cloud features and associated features with the image data to be processed;

Perform semantic segmentation on the image data to be processed through the feature extraction model to obtain the semantic segmentation results, cluster the semantic segmentation results to obtain the clustering results;

The clustering result is up-sampled through the feature extraction model to obtain second deep feature data corresponding to the image data to be processed; the second deep feature data includes deep image features and associated features with the point cloud data to be processed.

In a second aspect, the present application also provides a multimodal data fusion device. The unit includes:

The data acquisition module is used to obtain multimodal data to be processed; the multimodal data to be processed includes point cloud data to be processed and image data to be processed;

Model calling module, used to call the pre-built feature extraction model;

The feature extraction module is used to perform feature extraction on the point cloud data to be processed and the image data to be processed respectively through the feature extraction model to obtain the first deep feature data corresponding to the point cloud data to be processed and the second deep feature data corresponding to the image data to be processed ;

The feature fusion module is used to fuse the first deep feature data and the second deep feature data through a feature extraction model to obtain deep fusion features corresponding to the multimodal data to be processed.

In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Obtain multimodal data to be processed; multimodal data to be processed includes point cloud data to be processed and image data to be processed;

Call a pre-built feature extraction model;

Feature extraction is performed on the point cloud data to be processed and the image data to be processed respectively through the feature extraction model, and the first deep feature data corresponding to the point cloud data to be processed and the second deep feature data corresponding to the image data to be processed are obtained;

The first deep feature data and the second deep feature data are fused through a feature extraction model to obtain deep fusion features corresponding to the multimodal data to be processed.

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

Obtain multimodal data to be processed; multimodal data to be processed includes point cloud data to be processed and image data to be processed;

Call a pre-built feature extraction model;

Feature extraction is performed on the point cloud data to be processed and the image data to be processed respectively through the feature extraction model, and the first deep feature data corresponding to the point cloud data to be processed and the second deep feature data corresponding to the image data to be processed are obtained;

The first deep feature data and the second deep feature data are fused through a feature extraction model to obtain deep fusion features corresponding to the multimodal data to be processed.

In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

Obtain multimodal data to be processed; multimodal data to be processed includes point cloud data to be processed and image data to be processed;

Call a pre-built feature extraction model;

Feature extraction is performed on the point cloud data to be processed and the image data to be processed respectively through the feature extraction model, and the first deep feature data corresponding to the point cloud data to be processed and the second deep feature data corresponding to the image data to be processed are obtained;

The first deep feature data and the second deep feature data are fused through a feature extraction model to obtain deep fusion features corresponding to the multimodal data to be processed.

The above multimodal data fusion method, device, computer equipment, storage medium, and computer program product, because the feature extraction model is obtained through deep mining of sample multimodal data and trained according to the deep mining results, can fully mine and extract multiple The non-intuitive and visible deep feature data of modal data is fused, and the obtained deep fusion feature is closer to the real-world target and scene information, which greatly improves the fusion accuracy of multi-modal data, thereby improving the automatic driving perception technology Awareness of the surrounding environment.

Description of drawings

Fig. 1 is an application environment diagram of a multimodal data fusion method in an embodiment;

FIG. 2 is a schematic flow diagram of a multimodal data fusion method in an embodiment;

Fig. 3 is a schematic flow chart of the training steps of the feature extraction model in one embodiment;

Fig. 4 is a schematic diagram of the network structure of the deep learning model in an embodiment;

Fig. 5 is a structural block diagram of a multimodal data fusion device in an embodiment;

Figure 6 is an internal block diagram of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

The multimodal data fusion method provided in the embodiment of the present application can be applied to the application environment shown in FIG. 1 . Wherein, the multi-modal data fusion method is mainly executed on the computer device 102, specifically, in the automatic driving environment, the vehicle is pre-installed with the vehicle sensor 102 and the vehicle computer device 104. The on-board computer equipment may be simply referred to as computer equipment. The on-board sensor 102 collects multimodal data to be processed, and transmits the collected multimodal data to the computer device 104, so that the computer device 104 invokes a pre-built feature extraction model, and the feature extraction model is obtained by performing multimodal data on the sample In-depth mining is obtained by training according to the results of in-depth mining, so that feature extraction is performed on the point cloud data to be processed and the image data to be processed through the feature extraction model, and the first deep feature data corresponding to the point cloud data to be processed and the corresponding image data to be processed are obtained. The second deep feature data, and then through the feature extraction model, the first deep feature data and the second deep feature data are fused to obtain the deep fusion features corresponding to the multi-modal data to be processed. Wherein, the on-vehicle sensor 102 may include various image acquisition devices and video acquisition devices for collecting image data to be processed, and various radar sensors for collecting point cloud data to be processed. The computer device 104 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and smart vehicle devices.

In one embodiment, as shown in FIG. 2 , a multimodal data fusion method is provided. The method is applied to the computer device in FIG. 1 as an example for illustration, including the following steps:

Step 202, acquiring multimodal data to be processed; the multimodal data to be processed includes point cloud data to be processed and image data to be processed.

Wherein, the multimodal data to be processed refers to multimodal perception data that needs to be fused.

Specifically, in the autonomous driving environment, various on-board sensors installed on the vehicle scan the surrounding environment of the vehicle to obtain point cloud data to be processed and image data to be processed. Since the data collected by different types of sensors correspond to different modalities, the point cloud data to be processed and the image data to be processed can be called multi-modal data to be processed. Optionally, the various on-board sensors may include a first sensor and a second sensor. The first sensor may include various image acquisition devices and video acquisition devices for acquiring image data to be processed, such as cameras, cameras, and the like. The second sensor may include various radar sensors for collecting point cloud data to be processed, such as lidar, millimeter-wave radar, and the like. Thus, the on-board sensor transmits the collected multi-modal data to the computer device.

Step 204, call the pre-built feature extraction model.

Among them, deep mining refers to the mining of non-intuitively visible deep feature data in sample multimodal data.

A pre-built feature extraction model is stored in the computer device, and the feature extraction model is used to extract deep feature data of the multimodal data. The feature extraction model is obtained through deep mining of a large number of sample multimodal data to obtain deep mining results, and then trained according to the deep mining results. The results of in-depth mining can include the deep semantic features and intrinsic relationships of sample multimodal data. The computer equipment calls the pre-built feature extraction model to perform deep feature extraction on the multi-modal data to be processed.

In step 206, feature extraction is performed on the point cloud data to be processed and the image data to be processed through the feature extraction model to obtain the first deep feature data corresponding to the point cloud data to be processed and the second deep feature data corresponding to the image data to be processed.

Wherein, the first deep feature data refers to non-intuitively visible deep feature data in the point cloud data to be processed. The second deep feature data refers to non-intuitive deep feature data in the image data to be processed. Both the first deep feature data and the second deep feature data are high-dimensional features.

Specifically, the feature extraction model may include a feature extraction layer. The feature extraction layer is used for feature extraction of point cloud data to be processed and image data to be processed, and to fuse the extracted features. Further, the feature extraction layer may include two network branches, the point cloud branch and the image branch. Feature extraction is performed on the point cloud data to be processed through the point cloud branch, and the first deep feature data corresponding to the point cloud data to be processed is obtained. Wherein, the first deep feature data may include deep semantic features corresponding to the number of point clouds to be processed, as well as an intrinsic relationship between the point cloud data to be processed and the image data to be processed. Feature extraction is performed on the image data to be processed through the image branch to obtain second deep feature data corresponding to the image data to be processed. Wherein, the second deep feature data may include the deep semantic features corresponding to the image data to be processed, and the intrinsic relationship between the image data to be processed and the point cloud data to be processed.

In one of the embodiments, feature extraction is performed on the point cloud data to be processed and the image data to be processed through the feature extraction model to obtain the first deep feature data corresponding to the point cloud data to be processed and the second deep feature data corresponding to the image data to be processed The data includes: feature extraction of the point cloud data to be processed through the feature extraction model to obtain the first deep feature data corresponding to the point cloud data to be processed; the first deep feature data includes the deep point cloud features and the association with the image data to be processed Features: use the feature extraction model to perform semantic segmentation on the image data to be processed to obtain the semantic segmentation results, cluster the semantic segmentation results to obtain the clustering results; use the feature extraction model to up-sample the clustering results to obtain the corresponding image data to be processed The second deep feature data; the second deep feature data includes deep image features and associated features with point cloud data to be processed.

Among them, the deep point cloud features refer to the deep semantic features in the point cloud data to be processed. The association feature with the image data to be processed refers to the intrinsic association between the point cloud data to be processed and the image data to be processed. Deep image features refer to the deep semantic features in the image data to be processed. The association feature with the point cloud data to be processed refers to the internal association relationship between the image data to be processed and the point cloud data to be processed.

Specifically, the feature extraction model may include a feature extraction layer, through the point cloud branch in the feature extraction layer to perform feature extraction on the point cloud data to be processed, to obtain features including deep point cloud features and associated features with the image data to be processed, the second A deep feature data. For the processing of the image data to be processed, the semantic segmentation of the image to be processed is carried out through the image branch in the feature extraction layer, and the semantic segmentation result is obtained. Wherein, the semantic segmentation result includes the category of each pixel in the image data to be processed. In this way, the semantic segmentation results are clustered, and each pixel is clustered according to the category of each pixel. Then, the clustering result is up-sampled, so that each pixel in the image data to be processed has the same feature dimension as the point cloud in the first deep feature data, and then obtained including the deep image features and the relationship between the point cloud data and the point cloud data to be processed. The correlation features between, the second deep feature data. For example, the deep point cloud features corresponding to the number of point clouds to be processed and the deep image features corresponding to the image data to be processed may be more abstract high-dimensional floating-point features. Key features can be similar features or identical features.

Furthermore, the point cloud branch and the image branch mainly use the encoder (Encoder), that is, the convolutional network, to extract deep feature data from the point cloud data to be processed and the image data to be processed respectively. The point cloud branch may include an encoder, through which the feature dimension of the point cloud data to be processed is increased to a preset dimension, thereby extracting high-dimensional and deep feature data in the point cloud data to be processed, that is, the first deep feature data. For example, the point cloud data to be processed can be represented as (P, [x, y, z, i]), and the feature dimension of (P, [x, y, z, i]) is changed by the encoder in the point cloud branch 4. Upgrade to C, and output the dimension of the first deep feature data F_pt is (P, C), wherein, P represents the number of points in the point cloud data to be processed, C represents the preset dimension, which is the number of intermediate features, such as 10 , 64, etc.

Different from the point cloud branch, the image branch can include two encoders. The image to be processed is semantically segmented through the first encoder, and the semantic segmentation results are clustered, so that the clustering results are clustered through the second encoder. Upsampling is performed to obtain the second deep feature data. For example, when the dimension of the first deep feature data F_pt is (P, C), each pixel in the second deep feature data F_img has the same feature dimension C as the point cloud in the first deep feature data, and the second The dimensions of the deep feature data F_img are (H, W, C), where H represents the height of the second deep feature data, W represents the width of the second deep feature data, and C represents the feature dimension of the second deep feature data.

In step 208, the first deep feature data and the second deep feature data are fused through the feature extraction model to obtain deep fusion features corresponding to the multimodal data to be processed.

The feature data of the first deep layer and the feature data of the second deep layer are fused through the feature extraction layer in the feature extraction model.

Further, the feature extraction layer also includes a fusion layer (Fusion layer), such as a 1×1 convolution kernel, and the fusion layer is respectively connected to the output end of the encoder in the point cloud branch and the image branch. The first deep feature data output by the encoder in the point cloud branch and the second deep feature data output by the second encoder in the image branch are used as the input of the fusion layer, and the second deep layer is integrated according to the preset external parameter matrix through the fusion layer Each pixel in the feature data is aligned with the point in the first deep feature data to obtain the aligned second deep feature data, so that the aligned second deep feature data is fused with the first deep feature data to obtain Process the deep fusion feature F_fuse corresponding to the multimodal data. The shape of F_fuse is (P, C1), where P represents the number of point cloud points, and C1 represents the feature dimension.

In the traditional way, point cloud data and image data are superimposed in the visible range through models such as PointPainting (image laser fusion model) and RoarNet (RegiOnApproximation Refinement Network, target detection network) to achieve multi-modal data fusion. Focusing on superimposing shallow features, that is, shapes, colors, and spatial positions that are visible to the naked eye and easy to understand intuitively, while ignoring the inherent deep semantic connections of various modal data, that is, high-level features that cannot be clearly represented by two-dimensional or three-dimensional features. Dimensional features, and only one-way superposition of multiple modalities, such as superimposing images into point clouds, without mutual supervision and mutual learning process between multiple modalities. Therefore, the traditional methods cannot fully exploit the deep semantic features and intrinsic relationships of multimodal data.

In the above multimodal data fusion method, by calling the pre-built feature extraction model, feature extraction is performed on the point cloud data to be processed and the image data to be processed in the multimodal data to be processed, and the corresponding point cloud data to be processed is obtained. The first deep feature data and the second deep feature data corresponding to the image data to be processed, so as to fuse the first deep feature data and the second deep feature data to obtain deep fusion features corresponding to the multimodal data to be processed. Since the feature extraction model is obtained through in-depth mining of sample multi-modal data and trained according to the results of deep mining, it can fully mine and extract non-intuitive and visible deep feature data of multi-modal data, and perform fusion to obtain deep fusion The features are closer to the target and scene information in the real world, which greatly improves the fusion accuracy of multi-modal data, thereby improving the ability of autonomous driving perception technology to perceive the surrounding environment.

In one embodiment, the above method further includes: performing feature extraction on deep fusion features to obtain target features; performing an automatic driving task according to the target features.

The feature extraction model can also include a perception task layer, which is used to perform autonomous driving tasks.

After the feature extraction layer in the feature extraction model outputs deep fusion features, the deep fusion features are input to the perception task layer. The perception task layer may include an extraction layer (Extract layer) and a perception task head (3DTaskHead). For example, the abstraction layer can be Voxel, Pillar, etc. The feature extraction is performed on the deep fusion feature through the extraction layer to obtain the target feature. The target feature refers to the feature extracted from the point cloud feature of the deep fusion feature. The target feature is used as the input of the perception task head, and the automatic driving task is performed according to the target feature through the perception task head. For example, autonomous driving tasks can be 3D object detection tasks, object segmentation and other tasks.

In this embodiment, since the deep fusion features can be closer to real-world scenes, feature extraction is performed on the deep fusion features to obtain target features, which can provide the required data for the perception task head, which is beneficial for the subsequent perception task head to perform automatic driving Task.

In one embodiment, as shown in Figure 3, before obtaining the multimodal data to be processed, the above method further includes: a step of training the feature extraction model, which step includes:

Step 302, acquiring sample multimodal data; the sample multimodal data includes sample point cloud data and sample image data.

Step 304, input the sample multimodal data into the deep learning model to be trained; the deep learning model includes a feature extraction layer, a depth mining layer and a perception layer.

Among them, the sample multimodal data refers to the multimodal data used to train the deep learning model.

During the model training process, the computer equipment can first obtain sample multimodal data. The sample multimodal data may include sample point cloud data and sample image data. Invoke the deep learning model to be trained, and input sample multimodal data into the deep learning model. The deep learning model may include a feature extraction layer, a deep mining layer, and a perception layer. Both feature extraction layer and deep mining layer include point cloud branch and image branch. The point cloud branch in the feature extraction layer is connected to the point cloud branch in the deep mining layer, and the image branch in the feature extraction layer is connected to the image branch in the deep mining layer. The point cloud data to be processed and the image data to be processed are processed through two different branches, that is, the point cloud data to be processed is processed through the point cloud branch in the deep learning model, and the image data to be processed is processed through the image data.

Step 306, extracting sample point cloud features corresponding to the sample point cloud data and sample image features corresponding to the sample image data through the feature extraction layer, and fusing the sample point cloud features and sample image features to obtain sample fusion features.

The sample point cloud features corresponding to the sample point cloud data are extracted through the point cloud branch in the feature extraction layer, and the sample image features corresponding to the sample image data are extracted through the image branch. Both sample point cloud features and sample image features are high-dimensional features. The sample point cloud feature and the sample image feature are fused through the feature extraction layer to obtain the sample fusion feature. Furthermore, the structure of the feature extraction layer of the deep learning model is the same as that of the feature extraction model, and will not be repeated here.

Step 308 , perform deep mining on the feature of the sample point cloud and the feature of the sample image through the deep mining layer to obtain a deep mining result.

The point cloud feature of the sample is deeply mined through the point cloud branch in the deep mining layer, and the sample image feature is deeply mined through the image branch in the deep mining layer to obtain the mined feature data. Mining feature data refers to the deep features of sample point cloud data and sample image data, and the intrinsic correlation between multimodal data such as sample point cloud data and sample image data. Therefore, the deep mining result is calculated according to the mining feature data through the deep mining layer.

In one of the embodiments, the result of deep mining is a relationship loss value; through the deep mining layer, the sample point cloud features and sample image features are deeply mined, and the deep mining results obtained include: using the deep mining layer to analyze the sample point cloud features and sample images The features are deeply mined to obtain the mining feature data; the relationship loss value is determined according to the mining feature data.

The deep mining layer includes a point cloud branch and an image branch. Through the point cloud branch in the deep mining layer, the feature of the sample point cloud is deeply mined to obtain the first mined feature. Wherein, the first mining feature refers to the deep feature corresponding to the sample point cloud data, and the intrinsic relationship with the sample image data. The sample image features are deeply mined through the image branch in the deep mining layer to obtain the second mined features. The second mining feature refers to the deep feature corresponding to the sample image data, and the intrinsic relationship with the sample point cloud data. Mining feature data is obtained according to the first mining feature and the second mining feature. Furthermore, the relationship loss value of the deep learning model is calculated according to the mining feature data, and the relationship loss value is determined as the result of deep mining.

Further, the point cloud branch and the image branch of the depth mining layer respectively include an encoder and a multilayer perceptron (Multilayer Perceptron, MLP), and the multilayer perceptron can also be called Projection. The encoder is used for deep mining of sample point cloud features and sample image features. The multi-layer perceptron is used to map the mining feature data to the sample label space, that is, to integrate the mining feature data into a value, which can reduce the influence of the feature position on the mining feature data and improve the robustness of the entire model. Through the deep mining layer, the relationship loss value of the deep learning model is calculated according to the first mining feature and the second mining feature, and the parameters of the encoder in the point cloud branch and the image branch in the feature extraction layer are driven according to the relationship loss value to mine deep features and multiple Update the direction of the intrinsic relationship between modal data.

Further, the sample multimodal data may be multimodal data of multiple targets, therefore, the mined feature data refers to the mined features of multiple targets. For example, the relationship loss function of the relationship loss value can be calculated using the InfoNCE loss function as follows:

Among them, L _relation represents the relationship loss value, N represents the number of images or point cloud frames in a batch, z _i represents the deep feature of a target in the first mining feature or the second mining feature, Indicates the feature of the target that is the same as _zi in another mode mining feature, z _j indicates any target in a batch, Sim( _zi , z _j ) indicates the cosine similarity between z _i and z _j , t indicates Temperature hyperparameter.

From the relational loss function, minimizing the relational loss means that zi _and The similarity of zi should be as large as possible, while the similarity between _zi and other different data should be as small as possible. The relational loss function makes the similarity of regional features of the same target between different modal data higher, and the similarity of regional features of different targets is lower. In this way, the model can mine the deeper semantic features of the data and the feature relationship between different modal data, which is more conducive to the improvement of the perception ability of automatic driving.

In step 310, the deep learning model is trained through the perception layer according to the fusion features of the samples and the results of deep mining to obtain a pre-built feature extraction model.

Through the perception layer, the preset perception tasks are performed according to the sample fusion features, and the task execution results are obtained. Wherein, the preset perception task refers to an automatic driving task, which may be a 3D target detection task, a target segmentation task, and the like. Then, the deep learning model is trained according to the task execution results and deep mining results until the preset conditions are met, and a pre-built feature extraction model is obtained. Wherein, the preset condition may be that the loss value of the deep learning model no longer decreases or reaches a preset number of iterations.

In one of the embodiments, the perception layer includes a perception task layer and a training optimization layer; through the perception layer, according to the sample fusion features and depth mining results, training the deep learning model includes: performing preset perception according to the sample fusion features through the perception task layer The task is to obtain the task execution result; the overall loss value of the deep learning model is determined according to the task execution result and the relationship loss value through the training optimization layer, and the deep learning model is trained according to the overall loss value.

The preset perception task is performed according to the sample fusion feature through the perception task layer in the perception layer. The perception layer structure in the training process is the same as the perception layer structure in the actual application process. Wherein, the perception task layer may include an extraction layer (Extract layer) and a perception task head (3DTaskHead). For example, the abstraction layer can be Voxel, Pillar, etc. Feature extraction is performed on the deep fusion features through the extraction layer, and then the preset perception tasks, that is, automatic driving tasks, are performed through the perception task head according to the extracted features, and the task execution results are obtained. For example, the preset perception tasks may be 3D target detection tasks, target segmentation and other tasks.

Through the training optimization layer in the perception layer, the perception task loss value of the deep learning model is calculated according to the task execution result and the corresponding perception task loss function. When the preset perceptual tasks are different, the corresponding perceptual task loss functions may also be different. For example, when the preset perception task is a 3D object detection task, its perception task loss function may include position loss, direction loss and category loss. When the preset perceptual task is an object segmentation task, its perceptual task loss function can include category loss, bounding box loss, mask loss and region focus loss. Then, the overall loss value of the deep learning model is calculated by training the optimization layer according to the perception task loss value and the relationship loss value. Train the parameters of a deep learning model by tuning the overall loss value. Wherein, the total loss value may be the minimum total loss value.

Further, the preset perception task is a target detection task; determining the overall loss value of the deep learning model through the training optimization layer according to the task execution result and the relationship loss value includes: determining the position loss value of the deep learning model according to the task execution result through the training optimization layer , direction loss value and category loss value; determine the overall loss value of the deep learning model according to the position loss value, direction loss value, category loss value and relationship loss value by training the optimization layer.

Specifically, when the preset perception task is an object detection task, that is, a 3D object detection task, the corresponding perception task loss function may include position loss, direction loss, and category loss. Through the training optimization layer, the position loss value, direction loss value and category loss value of the deep learning model are calculated according to the task execution result and the corresponding perception task loss function. Perceptual task loss functions may include position loss functions, orientation loss functions, and category loss functions. Among them, the position loss function can use the SmoothL1 function, the direction loss function can use the Softmax loss function, and the category loss can use the FocalLoss function. For example, the positional loss function can look like this:

L _loc =∑ _{b∈(x,y,z,w,h,l,θ)} SmoothL1(Δb) (3)

Among them, L _loc represents the position loss value, b represents the position of the target, expressed by 7 degrees of freedom (the space coordinates of the target center (x, y, z), the target width, length and height (w, l, h), and the heading angle θ), Δb represents the position deviation between the detection frame of target b predicted by the deep learning network and the real frame.

Therefore, the overall loss value of the deep learning model is calculated by training the optimization layer according to the position loss value, direction loss value, category loss value, relationship loss value and preset calculation relationship. The overall loss value may be to minimize the overall loss value, and the optimization goal is to minimize the overall loss value. For example, a preset calculation relationship can look like this:

Among them, L _total represents the overall loss value, N _positive represents the total number of positive samples, the determination of positive samples and threshold optimization, that is, the prediction score of sample multimodal data is greater than the positive sample threshold, then this sample is a positive sample, L _loc represents the location Loss value, β _loc represents the position weight corresponding to the position loss value, L _dir represents the direction loss value, β _dir represents the direction weight corresponding to the direction loss value, L _cls represents the category loss value, β _cls represents the category weight corresponding to the category loss value, L _relation represents the relation loss value, and β _relation represents the relation weight corresponding to the relation loss value.

By designing a new comprehensive optimization objective, the performance of autonomous driving perception tasks and the degree of multimodal fusion can be comprehensively characterized. Through this new optimization goal, the deep learning network can be driven to learn the deep features and associations of multi-modal data, and improve the perception performance of autonomous driving.

Exemplarily, when the preset perception task is the target detection task, a schematic diagram of the network structure of the deep learning model may be shown in FIG. 4 . Among them, the deep learning model includes a feature extraction layer, a deep mining layer, a perception task layer and a training optimization layer. When training a deep learning model, the feature extraction layer, deep mining layer, perception task layer, and training optimization layer participate in the calculation of various loss values throughout the process, and update and optimize model parameters to obtain a high-quality feature extraction model. In the actual application process, there is no need for optimization and loss value calculation, and all loss structures can be removed, and only the feature extraction layer and the perception task layer are applicable. Both feature extraction layer and deep mining layer include point cloud branch and image branch. The point cloud branch in the feature extraction layer is connected to the point cloud branch in the deep mining layer, and the image branch in the feature extraction layer is connected to the image branch in the deep mining layer.

Specifically, the image branch in the feature extraction layer includes two Encoders (encoders). The image to be processed is semantically segmented through the first Encoder, and the semantic segmentation results are clustered, and the clustering results are clustered through the second Encoder. Upsampling to get F_img (second deep feature data). The point cloud branch in the feature extraction layer includes an Encoder, which increases the feature dimension of the point cloud data to be processed to the preset dimension through the Encoder, thereby extracting high-dimensional and deep feature data in the point cloud data to be processed, namely F_pt (first deep feature data). The feature extraction layer also includes the Fusion layer. Through the fusion layer, each pixel in F_img is aligned with the point in F_pt according to the preset external parameter matrix to obtain the aligned F_img, so that the aligned F_img and F_pt are fused. Get the F_fuse (deep fusion feature) corresponding to the multimodal data to be processed.

The point cloud branch and the image branch of the deep mining layer include an Encoder and a Projection (multi-layer perceptron) respectively. The Encoder in the image branch is used to deeply mine the sample image features, and the mined image features are input to the Projection to obtain the second Mining features. Through the Encoder in the point cloud branch, the sample point cloud features are deeply mined, and the mined point cloud features are input to Projection to obtain the first mined feature. The Relation (relation loss value) of the deep learning model is calculated according to the first mining feature and the second mining feature through the deep mining layer.

The perception task layer includes the Extract layer (extraction layer) and 3DTaskHead (3D perception task head). F_fuse is input into the Extract layer for feature extraction, and then the target detection task is performed according to the extracted features through 3DTaskHead, and the task execution result is obtained. According to the task execution The result computes Loc (Location Loss), Dir (Direction Loss) and Cls (Category Loss).

Calculate the TotalLoss (overall loss value) of the deep learning model by training the optimization layer according to Loc, Dir, Cls and Relation. Use TotalLoss to optimize the parameters of the deep learning model until the loss value of the deep learning model no longer decreases or reaches the preset number of iterations to obtain a pre-built feature extraction model.

In this embodiment, the deep learning model to be trained includes a feature extraction layer, a depth mining layer, and a perception layer. The sample point cloud features corresponding to the sample point cloud data and the sample image features corresponding to the sample image data are extracted through the feature extraction layer, and the sample The point cloud features and the sample image features are fused to obtain the sample fusion features, so that the sample point cloud features and sample image features are deeply mined through the deep mining layer in the deep learning model to be trained, and the deep mining results are obtained. Sample fusion features and deep mining results are used to train the deep learning model to obtain a pre-built feature extraction model. The trained feature extraction model can fully extract the internal relationship between multi-modal data, integrate the features of similar targets, extract more effective scene information, and provide more accurate information for autonomous driving planning and decision-making.

In another embodiment, a method for multimodal data fusion is provided, the method comprising the following steps:

Obtain sample multimodal data; the sample multimodal data includes sample point cloud data and sample image data.

Input sample multimodal data into the deep learning model to be trained; the deep learning model includes a feature extraction layer, a deep mining layer and a perception layer; the perception layer includes a perception task layer and a training optimization layer.

The sample point cloud features corresponding to the sample point cloud data and the sample image features corresponding to the sample image data are extracted through the feature extraction layer, and the sample point cloud features and the sample image features are fused to obtain the sample fusion features.

Through the deep mining layer, the sample point cloud features and sample image features are deeply mined to obtain the mined feature data.

Determine the relationship loss value of the deep learning model based on the mined feature data.

Through the perception task layer, the preset perception tasks are executed according to the sample fusion features, and the task execution results are obtained.

When the preset perception task is the target detection task, the position loss value, direction loss value and category loss value of the deep learning model are determined according to the task execution results by training the optimization layer.

Through the training optimization layer, the overall loss value of the deep learning model is determined according to the position loss value, direction loss value, category loss value and relationship loss value, and the deep learning model is trained according to the overall loss value to obtain a pre-built feature extraction model.

Obtain multimodal data to be processed; the multimodal data to be processed includes point cloud data to be processed and image data to be processed.

Call a pre-built feature extraction model.

Feature extraction is performed on the point cloud data to be processed through a feature extraction model to obtain first deep feature data corresponding to the point cloud data to be processed; the first deep feature data includes deep point cloud features and associated features with image data to be processed.

The image data to be processed is semantically segmented through the feature extraction model to obtain the semantic segmentation results, and the semantic segmentation results are clustered to obtain the clustering results.

In this embodiment, the pre-built feature extraction model is used to fuse the multi-modal data to be processed. Since the feature extraction model is obtained through deep mining of sample multi-modal data and trained according to the results of deep mining, it can deeply mine multiple The deep semantic features of modal data and the internal correlation between multi-modal data make the deep fusion feature closer to the real world, improve the fusion accuracy of multi-modal data, and provide information for the planning and decision-making of autonomous driving. more accurate information.

In one embodiment, this method is compared with the current mainstream methods PointPainting and RoarNet on the KITTI dataset for vehicle categories, as shown in Table 1 below. Among them, the positive and negative sample thresholds are 0.6 and 0.45, respectively. The test uses the evaluation standard AP (Average Precision, detection accuracy) of the KITTI dataset as the index. For vehicle targets with different difficulties, the detection accuracy of this method is higher than the two current mainstream multi-modal fusion methods. Especially the vehicle target under difficult circumstances has a certain degree of difficulty in detection. However, the detection accuracy obtained by this method on such objects is 0.24% higher than that of PointPainting. The comparison test shows from the data that this method has a certain advantage in improving the perception ability compared with the two current mainstream methods. The improvement of perception ability also depends on the learning of data by the model. Therefore, the experiment also verifies that this method has a deeper mining ability for multi-modal data features.

Table 1 Comparative experiment

It should be understood that although the steps in the flow charts involved in the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flow charts involved in the above-mentioned embodiments may include multiple steps or stages, and these steps or stages are not necessarily executed at the same time, but may be performed at different times For execution, the execution order of these steps or stages is not necessarily performed sequentially, but may be executed in turn or alternately with other steps or at least a part of steps or stages in other steps.

Based on the same inventive concept, an embodiment of the present application further provides a multi-modal data fusion device for implementing the above-mentioned multi-modal data fusion method. The solution to the problem provided by the device is similar to the implementation described in the above method, so the specific limitations in one or more embodiments of the multi-modal data fusion device provided below can be referred to above for multi-modal The limitation of the data fusion method will not be repeated here.

In one embodiment, as shown in FIG. 5 , a multimodal data fusion device is provided, including: a data acquisition module 502, a model calling module 504, a feature extraction module 506, and a feature fusion module 508, wherein:

The data to be processed acquisition module 502 is configured to acquire multimodal data to be processed; the multimodal data to be processed includes point cloud data to be processed and image data to be processed.

The pre-built model calling module 504 is used to call the pre-built feature extraction model.

The feature extraction module 506 is used to perform feature extraction on the point cloud data to be processed and the image data to be processed respectively through the feature extraction model to obtain the first deep feature data corresponding to the point cloud data to be processed and the second deep feature data corresponding to the image data to be processed data.

The feature fusion module 508 is configured to fuse the first deep feature data and the second deep feature data through a feature extraction model to obtain deep fusion features corresponding to the multimodal data to be processed.

In one embodiment, the above-mentioned device also includes:

The sample data acquisition module is configured to acquire sample multimodal data; the sample multimodal data includes sample point cloud data and sample image data.

The model to be trained calls a module, which is used to input sample multimodal data into the deep learning model to be trained; the deep learning model includes a feature extraction layer, a deep mining layer and a perception layer.

The feature processing module is used to extract the sample point cloud features corresponding to the sample point cloud data and the sample image features corresponding to the sample image data through the feature extraction layer, and fuse the sample point cloud features and sample image features to obtain sample fusion features.

The feature mining module is used to perform deep mining on sample point cloud features and sample image features through the deep mining layer to obtain deep mining results.

The model optimization module is used to train the deep learning model through the perception layer according to the sample fusion features and the deep mining results to obtain a pre-built feature extraction model.

In one embodiment, the deep mining result is a relationship loss value; the feature mining module is also used to deeply mine the sample point cloud features and sample image features through the deep mining layer to obtain mining feature data; determine the relationship loss according to the mining feature data value.

In one embodiment, the perception layer includes a perception task layer and a training optimization layer; the model optimization module is also used to perform a preset perception task through the perception task layer according to the sample fusion feature to obtain a task execution result; through the training optimization layer according to the task execution The result and the relationship loss value determine the overall loss value of the deep learning model, and the deep learning model is trained according to the overall loss value.

In one embodiment, the preset perception task is a target detection task; the model optimization module is also used to determine the position loss value, direction loss value and category loss value of the deep learning model according to the task execution results through the training optimization layer; through training optimization The layer determines the overall loss value of the deep learning model based on the position loss value, direction loss value, category loss value, and relationship loss value.

In one embodiment, the feature extraction module 506 is also used to perform feature extraction on the point cloud data to be processed through the feature extraction model to obtain the first deep feature data corresponding to the point cloud data to be processed; the first deep feature data includes deep point cloud Features and the associated features with the image data to be processed; the image data to be processed is semantically segmented through the feature extraction model to obtain the semantic segmentation results, and the semantic segmentation results are clustered to obtain the clustering results; the clustering results are obtained through the feature extraction model The results are up-sampled to obtain second deep feature data corresponding to the image data to be processed; the second deep feature data includes deep image features and associated features with the point cloud data to be processed.

Each module in the above-mentioned multi-modal data fusion device can be fully or partially realized by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure may be as shown in FIG. 6 . The computer device includes a processor, a memory, an input/output interface (Input/Output, I/O for short), and a communication interface. Wherein, the processor, the memory and the input/output interface are connected through the system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs and databases. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store multimodal data to be processed and the like. The input/output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-modal data fusion method is realized.

Those skilled in the art can understand that the structure shown in FIG. 6 is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the computer equipment to which the solution of this application is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In one embodiment, a computer device is provided, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the foregoing method embodiments when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented.

In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented.

It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all It is information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above-mentioned embodiments can be completed by instructing related hardware through computer programs, and the computer programs can be stored in a non-volatile computer-readable memory In the medium, when the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any reference to storage, database or other media used in the various embodiments provided in the present application may include at least one of non-volatile and volatile storage. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive variable memory (ReRAM), magnetic variable memory (Magnetoresistive Random Access Memory, MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (Phase Change Memory, PCM), graphene memory, etc. The volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory. As an illustration and not a limitation, RAM can be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (Dynamic Random Access Memory, DRAM). The databases involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a blockchain-based distributed database, etc., but is not limited thereto. The processors involved in the various embodiments provided by this application can be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., and are not limited to this.

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above-mentioned embodiments only express several implementation modes of the present application, and the description thereof is relatively specific and detailed, but should not be construed as limiting the patent scope of the present application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be determined by the appended claims.

Claims (10)

1. A multimodal data fusion method, characterized in that the method comprises: Obtain multimodal data to be processed; the multimodal data to be processed includes point cloud data to be processed and image data to be processed; Call a pre-built feature extraction model; Feature extraction is performed on the point cloud data to be processed and the image data to be processed through the feature extraction model, and the first deep feature data corresponding to the point cloud data to be processed and the first deep feature data corresponding to the image data to be processed are obtained. The second deep feature data; The first deep feature data and the second deep feature data are fused through the feature extraction model to obtain deep fusion features corresponding to the multimodal data to be processed. 2. The method according to claim 1, wherein, before the acquisition of the multimodal data to be processed, the method further comprises: Obtain sample multimodal data; the sample multimodal data includes sample point cloud data and sample image data; The sample multimodal data is input into the deep learning model to be trained; the deep learning model includes a feature extraction layer, a depth mining layer and a perception layer; The sample point cloud features corresponding to the sample point cloud data and the sample image features corresponding to the sample image data are extracted through the feature extraction layer, and the sample point cloud features and the sample image features are fused to obtain a sample fusion. feature; performing deep mining on the sample point cloud features and the sample image features through the deep mining layer to obtain a deep mining result; The perception layer trains the deep learning model according to the sample fusion feature and the deep mining result to obtain the pre-built feature extraction model. 3. The method according to claim 2, wherein the deep mining result is a relationship loss value; the deep mining of the sample point cloud features and the sample image features by the deep mining layer, Get deep mining results including: performing deep mining on the sample point cloud features and the sample image features through the deep mining layer to obtain mining feature data; The relationship loss value is determined according to the mining feature data. 4. The method according to claim 3, wherein the perceptual layer comprises a perceptual task layer and a training optimization layer; the perceptual layer is used according to the sample fusion feature and the depth mining result to perform The training of the above-mentioned deep learning model includes: Executing a preset sensing task through the sensing task layer according to the sample fusion feature to obtain a task execution result; The overall loss value of the deep learning model is determined by the training optimization layer according to the task execution result and the relationship loss value, and the deep learning model is trained according to the overall loss value. 5. The method according to claim 4, wherein the preset perception task is a target detection task; the depth is determined by the training optimization layer according to the task execution result and the relationship loss value The overall loss values for the learned model include: Determine the position loss value, direction loss value and category loss value of the deep learning model according to the task execution result through the training optimization layer; The overall loss value of the deep learning model is determined by the training optimization layer according to the position loss value, the direction loss value, the category loss value and the relationship loss value. 6. The method according to claim 1, wherein the feature extraction is carried out to the point cloud data to be processed and the image data to be processed respectively by the feature extraction model to obtain the point cloud to be processed The first deep feature data corresponding to the data and the second deep feature data corresponding to the image data to be processed include: Feature extraction is performed on the point cloud data to be processed through the feature extraction model to obtain first deep feature data corresponding to the point cloud data to be processed; the first deep feature data includes deep point cloud features and the Correlation features between the image data to be processed; performing semantic segmentation on the image data to be processed by the feature extraction model to obtain a semantic segmentation result, and clustering the semantic segmentation result to obtain a clustering result; The clustering result is up-sampled by the feature extraction model to obtain second deep feature data corresponding to the image data to be processed; the second deep feature data includes deep image features and the point cloud to be processed Association features between data. 7. A multimodal data fusion device, characterized in that the device comprises: A data acquisition module, configured to acquire multimodal data to be processed; the multimodal data to be processed includes point cloud data to be processed and image data to be processed; The model calling module is used to call a pre-built feature extraction model; the feature extraction model is obtained by performing deep mining on sample multimodal data and training according to deep mining results; A feature extraction module, configured to perform feature extraction on the point cloud data to be processed and the image data to be processed through the feature extraction model, to obtain the first deep feature data corresponding to the point cloud data to be processed and the The second deep feature data corresponding to the image data to be processed; A feature fusion module, configured to fuse the first deep feature data and the second deep feature data through the feature extraction model to obtain deep fusion features corresponding to the multimodal data to be processed. 8. The device according to claim 7, further comprising: A sample data acquisition module, configured to acquire sample multimodal data; the sample multimodal data includes sample point cloud data and sample image data; The model calling module to be trained is used to input the sample multimodal data into the deep learning model to be trained; the deep learning model includes a feature extraction layer, a depth mining layer and a perception layer; A feature processing module, configured to extract the sample point cloud features corresponding to the sample point cloud data and the sample image features corresponding to the sample image data through the feature extraction layer, and extract the sample point cloud features and the sample image features Perform fusion to obtain sample fusion features; A feature mining module, configured to perform deep mining on the sample point cloud features and the sample image features through the deep mining layer to obtain a deep mining result; The model optimization module is used to train the deep learning model through the perception layer according to the sample fusion features and the deep mining results, so as to obtain the pre-built feature extraction model. 9. A computer device, comprising a memory and a processor, the memory stores a computer program, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program step. 10. A computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are realized.