CN115294539A - Multitask detection method and device, storage medium and terminal - Google Patents

Abstract

A multi-task detection method and device, storage medium and terminal, the method includes: acquiring input data, and inputting the input data into a multi-task detection model obtained by training, the multi-task detection model includes: a single feature extraction network, multiple Adjust the network and multiple prediction networks, wherein the prediction network corresponds to the task one-to-one, and the adjustment network corresponds to the prediction network one-to-one; the feature extraction network is used to extract features from the input data to obtain a general feature vector; the kth adjustment is used The network performs feature extraction and/or dimension transformation on the general feature vector to obtain the feature vector corresponding to the k-th task, denoted as the k-th feature vector, where 1≤k≤M, k, M are positive integers, M is the task The kth prediction network is used to calculate the kth feature vector to obtain the detection result of the kth task. Through the above solution, the calculation amount of the multi-task detection process can be reduced, and the detection efficiency can be improved.

Description

Multi-task detection method and device, storage medium, terminal

technical field

The present invention relates to the technical field of deep learning, in particular to a multi-task detection method and device, a storage medium, and a terminal.

Background technique

In recent years, autonomous driving technology has developed rapidly. At present, the automatic driving system is mainly composed of environment perception system, positioning and navigation system, path planning system, speed control system and motion control system. Among them, the environmental perception system is an important part of the automatic driving system. As the upstream link of the entire automatic driving, its detection performance directly affects the subsequent links of automatic driving such as planning and decision-making.

As a perception device, radar has good stability and adaptability. In the prior art, radar is usually deployed on self-driving vehicles for 3D target detection to realize environment perception. Environmental perception for autonomous driving involves multiple detection tasks such as obstacle detection, lane line detection, and drivable area segmentation. In existing solutions, these detection tasks are usually processed independently. Considering limited computing resources, these tasks are processed separately. It greatly increases the consumption of on-board computing resources, and the detection efficiency is low.

Therefore, there is an urgent need for a multi-task detection method, which can reduce the calculation amount of the multi-task detection process and improve detection efficiency.

Contents of the invention

The technical problem solved by the invention is how to reduce the calculation amount in the multi-task detection process and improve the detection efficiency.

In order to solve the above-mentioned technical problems, an embodiment of the present invention provides a multi-task detection method, the method comprising: acquiring input data, and inputting the input data into a trained multi-task detection model, the multi-task detection model includes : a single feature extraction network, a plurality of adjustment networks and a plurality of prediction networks, wherein the prediction network corresponds to the task one-to-one, and the adjustment network corresponds to the prediction network; the feature extraction network is adopted Perform feature extraction on the input data to obtain a general feature vector; use the kth adjustment network to perform feature extraction and/or dimension transformation on the general feature vector to obtain the feature vector corresponding to the kth task, which is denoted as k eigenvectors, wherein, 1≤k≤M, k and M are positive integers, and M is the number of tasks; use the kth prediction network to calculate the kth eigenvectors to obtain the kth task Test results.

Optionally, the input data is point cloud data collected by radar, and the feature extraction network includes a convolutional layer, and the convolutional layer performs sparse convolution calculation.

Optionally, the multi-task detection model is obtained by using training data to train a preset model in advance, and the preset model includes: a single initial feature extraction network, multiple initial adjustment networks, and multiple initial prediction networks, so The training data includes: sample data corresponding to the kth task. Before obtaining the input data, the method further includes: Step 1: Using the training data to perform supervised training on the preset model, when the first preset is satisfied conditions, an intermediate detection model is obtained, wherein the intermediate detection model includes the feature extraction network, a plurality of intermediate adjustment networks and a plurality of intermediate prediction networks; Step 2: use the sample data corresponding to the kth task to The intermediate adjustment network and the kth intermediate prediction network perform supervised training, and when the second preset condition is met, the multi-task detection network is obtained; wherein, the first preset condition includes: the total loss is less than or equal to the first The preset loss, the second preset condition includes: the loss of each task is less than or equal to the second preset loss, and the total loss is calculated based on the losses of M tasks.

Optionally, the total loss is calculated using the following formula:

L is the total loss, σ _k is the weight of the k-th initial adjustment network, σ _k is a learnable parameter, and L _k is the loss of the k-th task.

Optionally, the learning rate used in step 1 is a first learning rate, the learning rate used in step 2 is a second learning rate, and the first learning rate is greater than the second learning rate.

Optionally, the sample data set further includes: unlabeled sample data. Before step 1, the method further includes: using the unlabeled sample data to perform unsupervised training on the preset model, when the third preset When the conditions are set, a preset model for performing the supervised training is obtained.

Optionally, the unlabeled sample data includes multiple frames of first point cloud data, and the constraints of the unsupervised training include: the minimum feature distance between matching points and/or the maximum feature distance between non-matching points Before performing unsupervised training on the preset model using the unlabeled sample data, the method further includes: performing data enhancement processing on the first point cloud data of each frame, so as to obtain the first point cloud data corresponding to the frame. The second point cloud data, wherein, the points in the first point cloud data and the points in the second point cloud data correspond one-to-one; in each frame of the first point cloud data and its corresponding second point cloud Matching points and non-matching points are determined in the data; wherein, the matching points refer to points with a corresponding relationship, and the non-matching points refer to points without a corresponding relationship.

In order to solve the above-mentioned technical problems, an embodiment of the present invention also provides a multi-task detection method device, the device includes: an acquisition module, configured to acquire input data, and input the input data to the trained multi-task detection model, The multi-task detection model includes: a single feature extraction network, multiple adjustment networks and multiple prediction networks, wherein the prediction network corresponds to the task one by one, and the adjustment network corresponds to the prediction network one by one ; The feature extraction module is used to perform feature extraction on the input data using a feature extraction network to obtain a general feature vector; the adjustment module is used to perform feature extraction and/or dimensionality on the general feature vector using the kth adjustment network Transform to obtain the feature vector corresponding to the kth task, which is recorded as the kth feature vector, wherein, 1≤k≤M, k and M are positive integers, and M is the number of tasks; the detection module is used to adopt the kth task The k prediction networks calculate the kth feature vector to obtain the detection result of the kth task.

An embodiment of the present invention also provides a storage medium on which a computer program is stored, and when the computer program is run by a processor, the steps of the above-mentioned multitasking detection method are executed.

An embodiment of the present invention also provides a terminal, including a memory and a processor, the memory stores a computer program that can run on the processor, and the processor executes the above-mentioned multitasking detection when running the computer program method steps.

Compared with the prior art, the technical solutions of the embodiments of the present invention have the following beneficial effects:

In the solution of the embodiment of the present invention, the input data is acquired and input into the trained multi-task detection model. Wherein, the multi-task detection model includes: a single feature extraction network, multiple adjustment networks and multiple prediction networks, wherein the prediction network corresponds to the tasks one by one, and the adjustment network corresponds to the prediction network one by one. Further, the feature extraction network is used to perform feature extraction on the input data to obtain a general feature vector; the kth adjustment network is used to perform feature extraction and/or dimension transformation on the general feature vector to obtain the kth task corresponding to The eigenvector of is denoted as the kth eigenvector; the kth prediction network is used to calculate the kth eigenvector to obtain the detection result of the kth task. Therefore, in the solution of the embodiment of the present invention, only a single feature extraction can be performed on the input data of each frame, and then the detection results of multiple detection tasks can be obtained based on the extracted common feature vector, thereby completing multiple detection tasks. That is, multiple detection tasks share the feature extraction network. On the one hand, compared with the existing multi-task detection model, multiple detection tasks share the feature extraction network, which reduces the model capacity and computational complexity. On the other hand, a corresponding tuned network is adopted for each detection task to perform domain-adaptive transformation on the common feature vectors to ensure the detection performance of each prediction network. Therefore, the solutions in the embodiments of the present invention can reduce computational complexity and improve detection efficiency while ensuring detection effects.

Further, in the solution of the embodiment of the present invention, the training data is used to carry out supervised training on the preset model, and when the first preset condition is met, an intermediate detection model is obtained; then the sample data corresponding to the kth task is used to test the kth task The intermediate adjustment network and the kth intermediate prediction network perform supervised training, and when the second preset condition is met, a multi-task detection network is obtained; wherein, the first preset condition includes: the total loss is less than or equal to the first preset loss, The second preset condition includes: the loss of each task is less than or equal to the second preset loss, and the total loss is calculated based on the losses of M tasks. Compared with the prior art, each task detection model is trained independently of each other. Using the above scheme, the joint training method can be used to train the feature extraction network, which can dig out the correlation information between each detection task, which is conducive to improving feature extraction. The ability of the network to extract features, thereby improving the accuracy and generalization ability of multi-task detection.

Furthermore, in the solutions of the embodiments of the present invention, before using supervised training, unsupervised training is performed first to obtain a preset model for supervised training. Using such a scheme, combining unsupervised training with supervised training will help reduce the need for labeled sample data and improve the accuracy and generalization capabilities of the network.

其中，σ_k为第k个初始调整网络的权重，且σ_k为可学习的参数。采用这样的方案，对每个调整网络采用可学习的自适应权重，可以减少人为超参数设定，有利于网络自动优化权重，从而提高特征提取网络的训练效果。Further, in the solution of the embodiment of the present invention, the following formula is used to calculate the total loss:

Among them, σ _k is the weight of the kth initial adjustment network, and σ _k is a learnable parameter. Adopting such a scheme, adopting learnable adaptive weights for each adjustment network can reduce artificial hyperparameter settings, which is conducive to the automatic optimization of weights by the network, thereby improving the training effect of the feature extraction network.

Description of drawings

Fig. 1 is a schematic structural diagram of a multi-task detection model in the prior art;

Fig. 2 is a schematic flow chart of a multi-task detection method in an embodiment of the present invention;

Fig. 3 is a schematic structural diagram of a multi-task detection model in an embodiment of the present invention;

4 is a schematic flow diagram of a training method for a multi-task detection model in an embodiment of the present invention;

Fig. 5 is a schematic structural diagram of a multi-task detection device in an embodiment of the present invention.

Detailed ways

As mentioned in the background art, there is an urgent need for a multi-task detection method that can reduce the amount of calculation in the multi-task detection process and improve detection efficiency.

In the application scenario of the automatic driving perception system, due to the real-time requirements of the automatic driving perception system, it is usually necessary to simultaneously perform multiple detection tasks on the same frame of point cloud data. Existing multi-task detection models usually include multiple independent detection models. As shown in FIG. 1 , the first detection model 11 , the second detection model 12 and the third detection model 13 can be used to perform different detection tasks respectively. For example, the same frame of point cloud data can be input into the first detection model 11 , the second detection model 12 and the third detection model 13 to obtain detection results of different detection tasks, thereby completing multi-task detection.

In addition, when performing multi-task detection, it is necessary to read multiple detection models into the memory and perform task detection simultaneously. Therefore, the existing solutions need to occupy a large amount of memory, and the amount of calculation is also very large, which is easy to cause freezes and low detection efficiency.

In order to solve the above technical problems, an embodiment of the present invention provides a multi-task detection method. In the solution of the embodiment of the present invention, only a single feature extraction can be performed on the input data of each frame, and then based on the extracted general feature vector The detection results of multiple detection tasks are obtained, so as to complete multiple detection tasks. That is, multiple detection tasks share the feature extraction network. On the one hand, compared with the multi-task detection model in the prior art, multiple detection tasks in the solution of the embodiment of the present invention can share the feature extraction network, which reduces the model capacity and computational complexity. On the other hand, a corresponding tuned network is adopted for each detection task to perform domain-adaptive transformation on the common feature vectors to ensure the detection performance of each prediction network. Therefore, the solutions in the embodiments of the present invention can reduce computational complexity and improve detection efficiency while ensuring detection effects.

In order to make the above objects, features and beneficial effects of the present invention more comprehensible, specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

Referring to FIG. 2 , FIG. 2 is a schematic flowchart of a multi-task detection method in an embodiment of the present invention. The method shown in FIG. 2 can be executed by a terminal, and the terminal can be various existing devices with data receiving and data processing functions.

In an application scenario, the terminal may be a vehicle-mounted terminal, for example, it may be an electronic control unit (Electronic Control Unit, ECU) of a vehicle, wherein the vehicle may be equipped with a radar, and the embodiment of the present invention does not apply to the type of the vehicle Without limitation, for example, it may be automatic guided vehicles (Automated Guided Vehicles, AGV), automatic driving vehicles (Autonomous vehicles), etc., but is not limited thereto.

In another application scenario, the terminal can also be a server, for example, the server is connected to a vehicle in communication, the vehicle can be equipped with a radar, and the server can receive point cloud data collected by the radar from the vehicle and execute the embodiment of the present invention A multi-task detection method is provided to obtain detection results.

It should be noted that, in the solution of the embodiment of the present invention, the radar may be a laser radar or a millimeter wave radar.

It should also be noted that multi-task detection in this embodiment of the present invention refers to processing multiple detection tasks at the same time. Specifically, the same input data (for example, the same frame of point cloud data) is processed to obtain the detection results of multiple detection tasks, instead of performing the next detection task after obtaining the detection results of one detection task , and it is not to perform different detection tasks on different point cloud data.

It should also be noted that the method provided by the embodiment of the present invention can be applied to multiple technical fields, such as automatic driving, augmented reality, virtual reality and intelligent robot, and the embodiment of the present invention only takes the field of automatic driving as an example. The description does not constitute a limitation on the application scenarios of the methods provided in the embodiments of the present invention.

The multitasking detection method shown in Figure 2 may comprise the following steps:

Step S11: Acquiring input data, and inputting the input data into the trained multi-task detection model;

Step S12: Using the feature extraction network to perform feature extraction on the input data to obtain a general feature vector;

Step S13: Use the kth adjustment network to perform feature extraction and/or dimension transformation on the general feature vector to obtain the feature vector corresponding to the kth task, which is denoted as the kth feature vector;

Step S14: Using the kth prediction network to calculate the kth feature vector to obtain the detection result of the kth task.

It can be understood that, in a specific implementation, the method can be implemented in the form of a software program, and the software program runs in a processor integrated inside the chip or chip module; or, the method can be implemented by using hardware or a combination of hardware and software way to achieve.

Referring to FIG. 3 , FIG. 3 is a schematic structural diagram of a multi-task detection model in an embodiment of the present invention. As shown in FIG. 3 , the multi-task detection model may include: a single feature extraction network 31 , multiple adjustment networks 32 and multiple prediction networks 33 .

The multi-task detection method provided by the embodiment of the present invention will be described in a non-limiting manner below with reference to FIG. 2 and FIG. 3 .

In the specific implementation of step S21, the input data to be processed at the current moment may be obtained, and the input data may be point cloud data collected by the radar. During the driving process of the vehicle, the radar can sense the driving environment to obtain point cloud data. That is, the input data may be point cloud data.

In other embodiments, the input data may also be images collected by a camera, which is not limited in this embodiment of the present invention.

When the input data is point cloud data, before inputting the point cloud data into the multi-task detection model, the point cloud data needs to be preprocessed to obtain the processed point cloud data, and the processed point cloud data Input to the multi-task detection model.

Specifically, the point cloud data is N×C dimensional data, where N is the number of points in the point cloud data, and C is the dimension of the point cloud data. Preprocessing the point cloud data may include: dividing the three-dimensional space into voxel blocks to obtain multiple voxel grids, and then projecting the points in the point cloud data into the voxel grids to obtain a voxel feature map; The cloud data is projected along the bird's-eye view direction to obtain a bird's-eye view (Bird-Eye View, BEV); the point cloud data is projected along the depth direction to obtain a depth map (Range Image). Thus, the processed point cloud data can include: voxel feature map, bird's eye view map and depth map.

Correspondingly, the projection relationship between the point cloud data and the processed point cloud data can also be saved together, so that subsequent processing can perform inverse transformation according to the projection relationship. Wherein, the projection relationship may include a projection relationship between point cloud data and a voxel feature map, a projection relationship between point cloud data and a bird's-eye view image, and a projection relationship between point cloud data and a depth map. More specifically, in the adjustment network corresponding to the semantic segmentation task, the inverse transformation can be performed according to the projection relationship, so as to calculate the prediction result of each point.

Further, the input data can be input into the pre-trained multi-task detection model. That is, the input data can be input into the multi-task detection model shown in FIG. 3 .

The multi-task detection model may include: a single feature extraction network 31, and the feature extraction network 31 is used to extract features from input data.

The multi-task detection model may also include: multiple adjustment networks 32 and multiple prediction networks 33 . Wherein, the number of the adjustment network 32 and the prediction network 33 are the same, and the adjustment network 32 and the prediction network 33 correspond one to one. More specifically, the input end of each adjustment network 32 is connected to the output end of the feature extraction network 31 , and the output end is connected to the input end of its corresponding prediction network 33 .

Wherein, the number M of the adjustment network 32 and the prediction network 33 may be determined according to actual application requirements. More specifically, the numbers of the adjustment network 32 and the prediction network 33 are determined according to the number of detection tasks, and the prediction network 33 is in one-to-one correspondence with the detection tasks, that is, the adjustment network 32 is also in one-to-one correspondence with the detection tasks.

Wherein, the structure of the adjustment network 32 and the structure of the prediction network 33 are determined according to the corresponding detection tasks.

In the specific implementation of step S22, the feature extraction network in the multi-task detection model can be used to perform feature extraction on the input data to obtain a general feature vector.

In a specific implementation, the structure of the feature extraction network 31 may be determined by the type of input data. In a specific example, when the input data is point cloud data, the feature extraction network includes a convolutional layer, and the convolutional layer performs sparse convolution (Sparse Convolution) calculation. Further, the general feature vector output by the feature extraction network may be four-dimensional feature data, wherein the channel direction of the general feature vector may be used to indicate the attribute information of the points in the point cloud data.

In another specific example, when the input data is an image, the convolutional layer of the feature extraction network 31 can perform a fully convolution (Fully Convolution) calculation.

More specifically, compared with the multiple feature extraction networks shown in FIG. 1 , the feature extraction network shown in FIG. 3 may be a structurally identical part of the multiple feature extraction networks in FIG. 1 .

In the specific implementation of step S23, the general feature vector output by the feature extraction network can be transmitted to multiple adjustment networks, and each adjustment network is used to perform corresponding calculations to obtain the feature vector required for the detection task corresponding to the adjustment network .

More specifically, the kth adjustment network can be used to perform feature extraction and/or dimension change on the general feature vector to obtain the feature vector corresponding to the kth task, which is denoted as the kth feature vector. Among them, 1≤k≤M, k and M are positive integers, and M is the number of detection tasks.

On the one hand, the kth adjustment network can be used to perform feature extraction on the general feature vector to obtain the kth feature vector. It should be noted that, compared with feature extraction performed by the feature extraction network, the k-th adjustment network performs further feature extraction on the general feature vectors to obtain deeper feature vectors.

In a specific implementation, the multiple detection tasks may include: a target detection task. Correspondingly, the adjustment network corresponding to the target detection task may include: Feature Pyramid Network (FPN) and Cross Stage Partial Networks (CSPNet). The general feature vector can be input to the adjustment network corresponding to the target detection task, and FPN and CSPNet can further extract features from the general feature vector to obtain the feature vector output by the adjustment network, which can be a two-dimensional feature map. Furthermore, the prediction network corresponding to the target detection task can include a classification network and a regression network. The classification network can obtain the position of the center point of the target according to the feature vector output by the adjustment network, and the regression network can obtain the length and width of the target according to the feature vector output by the adjustment network. Height and heading angle to get the detection result (x, y, z, dx, dy, dz, heading) of each target. Among them, x, y, and z are the three-dimensional positions of the center point of the target, dx, dy, and dz are the length, width, and height of the three-dimensional rectangular frame circumscribing the target, respectively, and heading is the rotation angle of the target (that is, the above-mentioned orientation angle).

On the other hand, the kth adjustment network can be used to change the dimension of the general feature vector to obtain the kth feature vector.

In a specific example, multiple detection tasks may also include: a semantic segmentation task. Correspondingly, the adjustment network corresponding to the semantic segmentation task can include a feature projection network, which can be used to project the general feature vector output by the feature extraction network to each point in the point cloud data. Using the adjustment network to perform dimension transformation on the general feature vector may refer to using a feature projection network to project the general feature vector to each point in the point cloud data.

In the specific implementation, the feature projection network can inversely transform the general feature vector according to the projection relationship used when preprocessing the point cloud data, so as to project the general feature vector to each point in the point cloud data, so as to obtain the kth Feature vector. Wherein, the kth eigenvector includes the eigenvectors of N points. Furthermore, the prediction network corresponding to the semantic segmentation task can be used to calculate the predicted category of the point according to the feature vector of each point. Furthermore, the prediction network corresponding to the semantic segmentation task can be a 2-layer fully connected network, and the prediction network can obtain the category of each point according to the feature vector output by the adjustment network. The category can be any of the following: mist, dust, twigs, and others.

In the specific implementation of step S24, the kth prediction network may be used to calculate the kth feature vector to obtain the detection result of the kth task. Wherein, the specific structure of the k-th prediction network may be determined according to the corresponding detection task, which is not limited in this embodiment.

From the above, in the solution provided by the embodiment of the present invention, only a single feature extraction can be performed on the input data (for example, point cloud data) of each frame, and then the detection results of multiple detection tasks can be obtained based on the extracted general feature vector , so as to complete multiple detection tasks. On the one hand, compared with the existing multi-task detection model shown in FIG. 1 , multiple detection tasks in the solution of the embodiment of the present invention can share a feature extraction network, which reduces model capacity and computational complexity. On the other hand, in the solution of the embodiment of the present invention, a corresponding adjustment network is used for each detection task to perform domain-adaptive transformation on the general feature vector, so as to improve the training effect and detection effect of each prediction network. Therefore, the solutions in the embodiments of the present invention can reduce computational complexity and improve detection efficiency while ensuring detection effects.

In practical applications, in the process of unmanned driving, the vehicle-mounted lidar will acquire point cloud data in real time, input it into the multi-task detection model shown in Figure 3, and obtain the detection results of each task. Furthermore, the detection results can be format-converted and then input to downstream modules such as the prediction module and the planning module.

The training process of the multi-task detection model shown in FIG. 3 is described below in a non-limiting manner.

Specifically, training data can be used to train the prediction model, wherein the preset model is a model before training, and the prediction model can include: a single initial feature extraction network, multiple initial adjustment networks, and multiple initial prediction networks. The training data may include sample data corresponding to multiple tasks. Among them, the initial feature extraction network is the feature extraction network before training, the initial adjustment network corresponds to the adjustment network one by one, the initial adjustment network is the adjustment network before training, the initial prediction network corresponds to the prediction network, and the initial prediction network is before training prediction network.

Referring to FIG. 4 , FIG. 4 is a schematic flowchart of a training method for a multi-task detection model in an embodiment of the present invention. The training process of the multi-task detection model shown in FIG. 3 will be described in a non-limiting manner below with reference to FIG. 4 . The training method shown in Fig. 4 can comprise the following steps:

Step S41: Perform unsupervised training on the preset model using unlabeled sample data, and obtain a preset model for supervised training when the third preset condition is met;

Step S42: Using the training data to perform supervised training on the preset model, and when the first preset condition is met, an intermediate detection model is obtained;

Step S43: Perform supervised training on the k-th intermediate adjustment network and the k-th intermediate prediction network by using the sample data corresponding to the k-th task, and obtain the multi-task detection network when the second preset condition is met.

In the training method shown in FIG. 4 , the training data used for training may include: sample data corresponding to multiple tasks, more specifically, the sample data corresponding to each task may include: unlabeled sample data and labeled sample data. More specifically, the labeled sample data corresponding to each task can be obtained by labeling the unlabeled sample data corresponding to the task, or can be obtained by labeling other sample data than the unlabeled sample data corresponding to the task , which is not limited in this embodiment.

In the specific implementation of step S41, matching points and non-matching points may be determined first.

Specifically, the unlabeled sample data may include multiple frames of first point cloud data, and data enhancement processing may be performed on each frame of first point cloud data to obtain second point cloud data corresponding to the frame of point cloud. Wherein, the data enhancement processing may include one or more of the following: adding random noise, random flipping, random noise and random scale transformation (for example, random scaling).

Since the second point cloud data is generated by the first point cloud data, the points in the first point cloud data correspond to the points in the second point cloud data, therefore, the first point cloud data and its Matching points and non-matching points are determined in the corresponding second point cloud data. Wherein, a matching point may refer to a point having a corresponding relationship, and a non-matching point refers to a point not having a corresponding relationship.

Furthermore, unsupervised training can be performed on the preset model. Wherein, the loss function of the unsupervised training may be a triplet loss function, and the constraint conditions of the unsupervised training may include: minimizing a feature distance between matching points and/or maximizing a feature distance between non-matching points. Wherein, the characteristic distance may be cosine distance, Chebyshev distance, Manhattan distance, etc., which is not limited in this embodiment.

Further, when the third preset condition is met, unsupervised training can be stopped, and a preset model for supervised training can be obtained. Wherein, the third preset condition may include that the training loss of the unsupervised training is less than or equal to a preset value, which is not limited in this embodiment.

In the specific implementation of step S42, the training data may be used to perform supervised training on the preset model. Wherein, the training data may include labeled sample data corresponding to multiple tasks. It should be noted that the preset model in step S42 refers to the preset model obtained after step S41 is executed. In other words, supervised training can use the preset model obtained from unsupervised training as the initial model for training.

Further, in the training process, each batch of training data can include labeled sample data corresponding to multiple tasks, input the labeled sample data of all tasks to the initial feature extraction network, and then output each task from the initial feature extraction network The corresponding sample universal feature vectors are respectively input into the corresponding initial adjustment network and initial prediction network to obtain the sample prediction results of each task.

Further, for each detection task, the loss of the task can be calculated according to the label of the sample data corresponding to the task and the sample prediction result.

Furthermore, the total loss can be calculated according to the losses of multiple tasks, and the parameters of the preset network model can be updated according to the total loss.

In a non-limiting example, the total loss may be calculated using the following formula:

Wherein, L is the total loss, σ _k is the weight of the k-th initial adjustment network, σ _k is a learnable parameter, L _k is the loss of the k-th task, and M is the total number of tasks.

It should be noted that σ _k is a learnable parameter. Specifically, during the first supervised training (ie, step S42 ), each time the parameters of each network are updated, the weights σ _k of each adjustment network are also updated together. Wherein, the initial value of σ _k may be preset. It should be noted that σ _k is used to calculate the total loss, and the total loss is optimized by optimizing σ _k , thereby optimizing the training effect of the feature extraction network.

Further, during the training process of step S42, the stochastic gradient descent method (that is, the SGD optimizer) may be used to update parameters.

Further, when the first preset condition is satisfied, an intermediate detection model is obtained, wherein the intermediate detection model includes the feature extraction network, multiple intermediate adjustment networks and multiple intermediate prediction networks. Wherein, the first preset condition includes: the total loss is less than or equal to the first preset loss.

In step S42, adopting learnable adaptive weights for each adjustment network can reduce artificial hyperparameter settings, which is beneficial for the network to automatically optimize parameter gradient weights, thereby improving the training effect of the feature extraction network.

In the specific implementation of step S43, the sample data corresponding to the kth task is used to perform supervised training on the kth intermediate adjustment network and the kth intermediate prediction network. When the second preset condition is met, the multi-task Detect network.

It should be noted that when step S42 is completed (that is, when the first preset condition is met), the feature extraction network can be obtained, that is, the training of the feature extraction network is completed. In step S43, only the adjustment network and the prediction network are trained, and the feature extraction network is not trained. That is, the parameters of the feature extraction network are fixed when the first preset condition is satisfied, and only the parameters of the adjustment network and the prediction network are updated in step S43.

Further, since in step S43, the sample data corresponding to each task is used to train the adjustment network and prediction network corresponding to the task, therefore, in step S43, there is no need to calculate the total loss, but the loss of each task is calculated separately, And update the parameters of the intermediate adjustment network and intermediate prediction network corresponding to the task according to the loss of each task.

In a specific implementation, the learning rate adopted in step S42 is the first learning rate, the learning rate adopted in step S43 is the second learning rate, and the first learning rate is greater than the second learning rate.

Further, when the second preset condition is satisfied, the above-mentioned multi-task detection model can be obtained. Wherein, the second preset condition includes: the loss of each task is less than or equal to the second preset loss.

In a specific example, the verification set data can be used for verification after each training cycle, and the verification effect and loss changes can be observed. If the verification effect does not improve and the loss increases, the training is stopped immediately.

From the above, in the solution of the embodiment of the present invention, the method of combining unsupervised and supervised training can reduce the demand for sample data and help improve the accuracy and generalization ability of the network.

In the case where the training data is point cloud data, it is prone to problems that are difficult to label. Specifically, the existing annotations are usually artificial. In the autonomous driving scenario, the points in the point cloud data usually include points corresponding to water mist and dust. It is usually difficult to judge whether they correspond to water mist or dust point. For this reason, the above-mentioned method of combining unsupervised and supervised training can also effectively deal with the problem of lack of labeled samples.

It should be noted that, in other embodiments, only a supervised training method may also be used. That is, the multi-task detection model can be obtained by executing steps S42 and S43.

As mentioned above, the existing task detection models are independent of each other, and the training process is also independent of each other. However, in the perception scenario of autonomous driving, there are correlations between different detection tasks. For example, lanes are often the boundaries of drivable areas, which usually closely surround traffic objects. In the embodiment of the present invention, the multi-task detection model shown in FIG. 3 is constructed and jointly trained. Such a training process can learn better representations by sharing information between multiple tasks, and the trained feature extraction network can extract more information, thereby improving the performance of each detection task.

In practical applications, pytorch can be used for training. After the training is completed, the trained multi-task detection model can be quantified and accelerated through the tensorrt library, and the model can be implemented and deployed online in C++ language.

Referring to FIG. 5 , FIG. 5 is a schematic structural diagram of a multi-task detection device in an embodiment of the present invention. The device shown in Figure 5 may include:

The obtaining module 51 is used to obtain input data, and input the input data into the trained multi-task detection model, the multi-task detection model includes: a single feature extraction network, multiple adjustment networks and multiple prediction networks, wherein , the prediction network is in one-to-one correspondence with the tasks, and the adjustment network is in one-to-one correspondence with the prediction network;

A feature extraction module 52, configured to extract features from the input data using the feature extraction network to obtain a general feature vector;

The adjustment module 53 is configured to use the kth adjustment network to perform feature extraction and/or dimension transformation on the general feature vector to obtain the feature vector corresponding to the kth task, which is denoted as the kth feature vector, where 1≤k ≤M, k and M are positive integers, and M is the number of tasks;

The detection module 54 is configured to use the k-th prediction network to calculate the k-th feature vector, so as to obtain the detection result of the k-th task.

In a specific implementation, the above-mentioned multitasking detection device may correspond to a chip with data processing function in the terminal; or correspond to a chip module with data processing function in the terminal, or correspond to the terminal.

For more details about the working principle, working mode and beneficial effects of the multi-tasking detection device shown in FIG. 5 , reference may be made to the relevant descriptions in FIG. 1 to FIG. 4 above, which will not be repeated here.

An embodiment of the present invention also provides a storage medium on which a computer program is stored, and when the computer program is run by a processor, the steps of the above-mentioned multitasking detection method are executed. The storage medium may include ROM, RAM, magnetic or optical disks, and the like. The storage medium may also include a non-volatile memory (non-volatile) or a non-transitory (non-transitory) memory, and the like.

An embodiment of the present invention also provides a terminal, including a memory and a processor, the memory stores a computer program that can run on the processor, and the processor executes the above-mentioned multitasking detection when running the computer program method steps. The terminal may be a vehicle-mounted terminal.

An embodiment of the present invention also provides a vehicle, which may include the above-mentioned terminal, and the terminal may execute the above-mentioned multi-task detection method.

It should be understood that in the embodiment of the present application, the processor may be a central processing unit (CPU for short), and the processor may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP for short), dedicated Integrated circuit (application specific integrated circuit, referred to as ASIC), off-the-shelf programmable gate array (field programmable gate array, referred to as FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, and the like.

It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Wherein, the non-volatile memory can be read-only memory (read-only memory, referred to as ROM), programmable read-only memory (programmable ROM, referred to as PROM), erasable programmable read-only memory (erasable PROM, referred to as EPROM) , Electrically Erasable Programmable Read-Only Memory (electrically EPROM, referred to as EEPROM) or flash memory. The volatile memory may be random access memory (RAM for short), which is used as an external cache. By way of illustration and not limitation, many forms of random access memory (RAM) are available such as static random access memory (static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic Random access memory (synchronousDRAM, referred to as SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, referred to as DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, referred to as ESDRAM), synchronous connection Dynamic random access memory (synchlink DRAM, SLDRAM for short) and direct memory bus random access memory (direct rambus RAM, DR RAM for short).

The above-mentioned embodiments may be implemented in whole or in part by software, hardware, firmware or other arbitrary combinations. When implemented using software, the above-described embodiments may be implemented in whole or in part in the form of computer program products. The computer program product comprises one or more computer instructions or computer programs. When the computer instruction or computer program is loaded or executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer program can be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program can be downloaded from a website, computer, server or data center Wired or wireless transmission to another website site, computer, server or data center.

In the several embodiments provided in this application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are only illustrative; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software functional units. For example, for each device or product applied to or integrated into a chip, each module/unit contained therein may be realized by hardware such as a circuit, or at least some modules/units may be realized by a software program, and the software program Running on the integrated processor inside the chip, the remaining (if any) modules/units can be realized by means of hardware such as circuits; They are all realized by means of hardware such as circuits, and different modules/units can be located in the same component (such as chips, circuit modules, etc.) or different components of the chip module, or at least some modules/units can be realized by means of software programs, The software program runs on the processor integrated in the chip module, and the remaining (if any) modules/units can be realized by hardware such as circuits; /Units can be realized by means of hardware such as circuits, and different modules/units can be located in the same component (such as chips, circuit modules, etc.) or different components in the terminal, or at least some modules/units can be implemented in the form of software programs Realization, the software program runs on the processor integrated in the terminal, and the remaining (if any) modules/units can be implemented by means of hardware such as circuits.

It should be understood that the term "and/or" in this article is only an association relationship describing associated objects, which means that there may be three relationships, for example, A and/or B may mean: A exists alone, and A and B exist at the same time , there are three cases of B alone. In addition, the character "/" in this article indicates that the contextual objects are an "or" relationship.

"Multiple" appearing in the embodiments of the present application means two or more.

The first, second, etc. descriptions that appear in the embodiments of this application are only for illustration and to distinguish the description objects. Any limitations of the examples.

Although the present invention is disclosed above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, so the protection scope of the present invention should be based on the scope defined in the claims.

Claims (10)

1. A method of multi-tasking detection, the method comprising:

acquiring input data, and inputting the input data into a multi-task detection model obtained by training, wherein the multi-task detection model comprises: the task scheduling system comprises a single feature extraction network, a plurality of adjustment networks and a plurality of prediction networks, wherein the prediction networks correspond to the tasks one to one, and the adjustment networks correspond to the prediction networks one to one;

extracting the features of the input data by adopting the feature extraction network to obtain a universal feature vector;

performing feature extraction and/or dimension transformation on the general feature vector by adopting a kth adjusting network to obtain a feature vector corresponding to a kth task, and marking the feature vector as the kth feature vector, wherein k is more than or equal to 1 and is less than or equal to M, k and M are positive integers, and M is the number of the tasks;

and calculating the kth characteristic vector by adopting a kth prediction network to obtain a detection result of the kth task.

2. The multi-tasking detection method of claim 1, wherein the input data is radar-acquired point cloud data, the feature extraction network comprises convolutional layers, and the convolutional layers perform sparse convolution calculations.

3. The multitask detection method according to claim 2, wherein the multitask detection model is obtained by training a preset model in advance by using training data, and the preset model comprises: a single initial feature extraction network, a plurality of initial adjustment networks, and a plurality of initial prediction networks, the training data comprising: before the sample data corresponding to the kth task is obtained, the method further includes:

the method comprises the following steps: carrying out supervised training on the preset model by adopting the training data, and obtaining an intermediate detection model when a first preset condition is met, wherein the intermediate detection model comprises the feature extraction network, a plurality of intermediate adjustment networks and a plurality of intermediate prediction networks;

step two: carrying out supervised training on a kth intermediate adjusting network and a kth intermediate predicting network by adopting sample data corresponding to the kth task, and obtaining the multi-task detecting network when a second preset condition is met;

wherein the first preset condition comprises: the total loss is less than or equal to a first preset loss, and the second preset condition comprises that: and the loss of each task is less than or equal to a second preset loss, and the total loss is calculated according to the losses of the M tasks.

4. A method of multitasking detection according to claim 3, characterized in that the total loss is calculated using the following formula:

wherein L is the total loss, σ _k For the kth initial adjustment network weight, σ _k As a learnable parameter, L _k Is the loss of the k-th task.

5. The multitask detection method according to claim 3, wherein the learning rate used in the first step is a first learning rate, the learning rate used in the second step is a second learning rate, and the first learning rate is larger than the second learning rate.

6. The multitask detection method according to claim 3, characterized in that said sample data set further comprises: no labeled sample data, before the first step, the method further comprises:

and performing unsupervised training on the preset model by adopting the unlabeled sample data, and obtaining the preset model for performing the supervised training when a third preset condition is met.

7. The multitask detection method according to claim 6, wherein the unlabeled sample data comprises a plurality of frames of first point cloud data, and the unsupervised training constraints comprise: minimizing the feature distance between the matching points and/or maximizing the feature distance between the non-matching points, wherein before performing unsupervised training on the preset model by using the unlabeled sample data, the method further comprises:

performing data enhancement processing on each frame of first point cloud data to obtain second point cloud data corresponding to the frame of first point cloud data, wherein points in the first point cloud data correspond to points in the second point cloud data one to one;

determining matching points and non-matching points in each frame of first point cloud data and corresponding second point cloud data; the matching points refer to points with corresponding relations, and the non-matching points refer to points without corresponding relations.

8. A method and apparatus for multi-task detection, the apparatus comprising:

an obtaining module, configured to obtain input data and input the input data into a trained multi-task detection model, where the multi-task detection model includes: the system comprises a single feature extraction network, a plurality of adjustment networks and a plurality of prediction networks, wherein the prediction networks correspond to the tasks one to one, and the adjustment networks correspond to the prediction networks one to one;

the characteristic extraction module is used for extracting the characteristics of the input data by adopting the characteristic extraction network so as to obtain a universal characteristic vector;

the adjusting module is used for performing feature extraction and/or dimension transformation on the general feature vector by adopting a kth adjusting network to obtain a feature vector corresponding to a kth task, and the feature vector is marked as the kth feature vector, wherein k is more than or equal to 1 and less than or equal to M, k and M are positive integers, and M is the number of the tasks;

and the detection module is used for calculating the kth characteristic vector by adopting the kth prediction network so as to obtain a detection result of the kth task.

9. A storage medium having a computer program stored thereon, the computer program, when executed by a processor, performing the steps of the multitask detection method according to any one of claims 1 to 7.

10. A terminal comprising a memory and a processor, the memory having stored thereon a computer program operable on the processor, wherein the processor, when executing the computer program, performs the steps of the multitask detection method according to any one of claims 1-7.

Images