CN112052818A - Unsupervised domain adaptive pedestrian detection method, unsupervised domain adaptive pedestrian detection system and storage medium - Google Patents

Abstract

The embodiment of the application provides a pedestrian detection method and system adaptive to an unsupervised domain and a storage medium. According to the method and the device, only label-free image data in a new scene need to be acquired, and a large amount of image annotation is not needed, so that the manpower and material resource consumption caused by data annotation in the development process is greatly saved, and the efficiency is improved; the migration capability of the model is improved, and the method can be more suitable for the change of scenes.

Description

Pedestrian detection method, system and storage medium for unsupervised domain adaptation

technical field

The present application belongs to the technical field of pedestrian detection, and in particular, relates to an unsupervised domain adaptation pedestrian detection method, system and storage medium.

Background technique

With the increasing use of pedestrian detection in areas such as intelligent security and autonomous driving, the application scenarios of pedestrian detection are becoming more and more abundant. However, different scenes have different lighting conditions, backgrounds, camera angles, etc., which means that the data distribution in different scenes is usually different. The practical application scenarios of pedestrian detection are various, and the direct use of a network model trained in one scenario in another scenario will result in a significant drop in detection performance. The deep learning method relies on a large amount of labeled data to improve the generalization performance of the network model. Therefore, the existing technology mainly re-labels the large amount of data in each scene in a data-driven manner, and then relabels the newly labeled data. Retrain the network to get the network model in the new scene.

Specifically, in the existing methods, supervised transfer learning is mainly used to deal with such problems. That is, artificially label new data in a new scene, and use the re-labeled data to perform fine-tuning training on the original model to achieve model migration. The specific implementation process includes: 1. Collect data in the target domain scene; 2. Manually label the collected data; 3. Use the data and its corresponding label information to fine-tune the model trained on the source domain; 4. Use The trained model performs object detection in the target domain.

However, although this method can make the network model adapt to new scenarios, each time a scenario is switched, a large amount of new scenario data will be generated correspondingly, and a large amount of new scenario data will require a lot of manpower and material resources for labeling, which will bring great value to applications and developers. There is a great inconvenience, and at the same time, with the explosive growth of cameras, data tagging for each newly erected camera requires a very large cost. Finally, with the increasing number of changes in the scene, the model transfer ability gradually deteriorates. Therefore, the above methods for fine-tuning training based on labeled data encounter a relatively large bottleneck in practical applications, and a new pedestrian detection algorithm is urgently needed to solve the above problems.

SUMMARY OF THE INVENTION

The present invention proposes a pedestrian detection method, system and storage medium for unsupervised domain adaptation, aiming to solve the problem of time-consuming and laborious manual labeling of a large amount of data in new scenarios when performing fine-tuning training based on labeled data in the prior art. question.

According to a first aspect of the embodiments of the present application, an unsupervised domain adaptation pedestrian detection method is provided, including the following steps:

Randomly select labeled image data and unlabeled image data;

The labeled image data is enhanced with random data to obtain enhanced labeled image data; the unlabeled image labeled data is obtained through random data enhancement to obtain the first enhanced unlabeled image data and the second enhanced unlabeled image data respectively;

Input the enhanced labeled data to the first pedestrian detection network to obtain the first pedestrian prediction feature; input the first enhanced unlabeled data to the first pedestrian detection network to obtain the second pedestrian prediction feature; input the second enhanced unlabeled image The data is sent to the second pedestrian detection network to obtain the third pedestrian prediction feature;

According to the label feature of the labeled image data and the first pedestrian prediction feature, the supervised learning cost is obtained; according to the second pedestrian prediction feature and the third pedestrian prediction feature, the consistency cost is obtained;

The total cost is obtained by adding the supervised learning cost and the consistency cost;

According to the total cost, the weight parameters of the first pedestrian detection network are updated through the stochastic gradient descent algorithm;

According to the weight parameters of the first pedestrian detection network, the weight parameters of the second pedestrian detection network are updated through the exponential moving average algorithm.

In some embodiments of the present application, the pedestrian detection method for unsupervised domain adaptation further includes:

After repeating the above steps until the first pedestrian detection network and the second pedestrian detection network converge, the updated first pedestrian detection network and the second pedestrian detection network are obtained;

Input the unlabeled image data to be tested into the updated first pedestrian detection network to obtain the pedestrian detection result.

In some embodiments of the present application, the labeled image data is randomly selected from image data whose labeled data is known; the unlabeled image data is randomly selected from the unlabeled image data to be tested.

In some embodiments of the present application, the first pedestrian prediction feature, the second pedestrian prediction feature, and the third pedestrian prediction feature include size information, classification information, and location information of the image pedestrian.

In some embodiments of the present application, the supervised learning cost and the consistency cost include pedestrian classification loss, pedestrian center point offset loss, pedestrian border width and height loss.

In some embodiments of the present application, the first pedestrian detection network and the second pedestrian detection network initially use the same neural network architecture.

In some embodiments of the present application, random data augmentation includes image size or pixel random augmentation.

In some embodiments of the present application, the weight parameters of the second pedestrian detection network are obtained by calculating the weights of the first pedestrian detection network by exponential moving average in the training process.

According to a second aspect of the embodiments of the present application, an unsupervised domain adaptation pedestrian detection system is provided, which specifically includes:

Training data selection module: used to randomly select labeled image data and unlabeled image data;

Data enhancement module: used to enhance the labeled image data through random data enhancement to obtain enhanced labeled image data; used to enhance the unlabeled image labeled data through random data enhancement to obtain the first enhanced unlabeled image data and the second enhanced unlabeled image respectively. data;

Feature prediction network module: used to input the enhanced labeled data to the first pedestrian detection network to obtain the first pedestrian prediction feature; used to input the first enhanced unlabeled data to the first pedestrian detection network to obtain the second pedestrian prediction feature; used to input the second enhanced unlabeled image data to the second pedestrian detection network to obtain the third pedestrian prediction feature;

Supervised learning cost module: used to obtain the supervised learning cost according to the label features of the labeled image data and the first pedestrian prediction feature;

Consistency cost module: used to obtain the consistency cost according to the predicted features of the second pedestrian and the predicted features of the third pedestrian;

Total cost module: used to add the supervised learning cost and the consistency cost to obtain the total cost;

The first pedestrian detection network update module: used to update the weight parameters of the first pedestrian detection network through the stochastic gradient descent algorithm according to the total cost;

The second pedestrian detection network update module is used to update the weight parameters of the second pedestrian detection network through the exponential moving average algorithm according to the weight parameters of the first pedestrian detection network.

In some embodiments of the present application, the unsupervised domain-adapted pedestrian detection system further includes:

Training convergence module: used to repeat the above steps until the first pedestrian detection network and the second pedestrian monitoring network converge, and the updated first pedestrian detection network and the second pedestrian monitoring network are obtained;

Pedestrian detection module: used to input the unlabeled image data to be tested into the updated first pedestrian detection network to obtain pedestrian detection results.

According to a third aspect of the embodiments of the present application, there is provided a computer-readable storage medium on which a computer program is stored; the computer program is executed by a processor to implement a pedestrian detection method for unsupervised domain adaptation.

Using the pedestrian detection method, system and storage medium for unsupervised domain adaptation in the embodiments of the present application, labeled image data and unlabeled image data are randomly selected; labeled image data is enhanced by random data enhancement to enhance labeled image data; The image signature data is obtained by random data enhancement to obtain the first enhanced unlabeled image data and the second enhanced unlabeled image data; input the enhanced labeled data to the first pedestrian detection network to obtain the first pedestrian prediction feature; input the first enhanced The unlabeled data is sent to the first pedestrian detection network to obtain the second pedestrian prediction feature; the second enhanced unlabeled image data is input to the second pedestrian detection network to obtain the third pedestrian prediction feature; According to the pedestrian prediction feature, the supervised learning cost is obtained; according to the second pedestrian prediction feature and the third pedestrian prediction feature, the consistency cost is obtained; according to the supervised learning cost and the consistency cost, the total cost is obtained; according to the total cost, through the stochastic gradient The descending algorithm is used to update the weight parameters of the first pedestrian detection network; according to the weight parameters of the first pedestrian detection network, the exponential moving average algorithm is used to update the weight parameters of the second pedestrian detection network. This application adopts transfer learning to strengthen the transfer ability of the model, through the method of unsupervised domain adaptation, that is, the model is jointly trained by the labeled data in the existing scene and the unlabeled data in the new scene, so that the model can transfer the existing scene to the Data representation capabilities are migrated to new scene data. This application only needs to collect unlabeled image data in a new scene, and does not need to re-label a large number of images, which greatly saves the human and material consumption caused by data labeling in the development process, improves efficiency, and improves the migration ability of the model. It is more adaptable to changes in the scene.

Description of drawings

The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the attached image:

FIG. 1 shows a flowchart of steps of an unsupervised domain adaptation pedestrian detection method according to an embodiment of the present application;

FIG. 2 shows a schematic diagram of steps in an unsupervised domain adaptation pedestrian detection method according to another embodiment of the present application;

FIG. 3 shows a schematic flowchart of the pedestrian detection method for unsupervised domain adaptation according to an embodiment of the present application;

FIG. 4 shows a schematic structural diagram of an unsupervised domain-adapted pedestrian detection system according to an embodiment of the present application.

Detailed ways

In the process of realizing this application, the inventor found that when the existing processing of pedestrian detection data in a new scene is used, a supervised transfer learning method is used to manually label the image data in the new scene to obtain new data, and the original The detection network of the scene is fine-tuned, and finally, with the increasing number of scenes, the model migration ability gradually deteriorates, the detection results are inaccurate, and a large amount of manual data annotation is time-consuming and labor-intensive.

In order to solve the problems of the gradual deterioration of model transfer ability and the high cost of labeling data, this application proposes a transfer learning method for unsupervised domain adaptation, which only needs to collect unlabeled data in the target scene without manually labeling new data. The data.

This application builds two identical pedestrian detection network models, one as a student model and one as a teacher model. The teacher network generates pseudo-labels according to the target domain data, and uses the generated pseudo-labels to guide the student network that needs to be trained. By making the output of the student network as close as possible to the output of the teacher network, the student network is more adapted to the distribution of the target domain data.

In the pedestrian detection method, system and storage medium for unsupervised domain adaptation of the present application, labelled image data and unlabeled image data are randomly selected; labelled image data is enhanced by random data enhancement to enhance labelled image data; Random data enhancement obtains the first enhanced unlabeled image data and the second enhanced unlabeled image data; input the enhanced labeled data to the first pedestrian detection network to obtain the first pedestrian prediction feature; input the first enhanced unlabeled data to The first pedestrian detection network is used to obtain the second pedestrian prediction feature; the second enhanced unlabeled image data is input to the second pedestrian detection network to obtain the third pedestrian prediction feature; according to the label feature of the labeled image data and the first pedestrian prediction feature , the supervised learning cost is obtained; the consistency cost is obtained according to the second pedestrian prediction feature and the third pedestrian prediction feature; the total cost is obtained by adding the supervised learning cost and the consistency cost; according to the total cost, the stochastic gradient descent algorithm is used to update The weight parameters of the first pedestrian detection network; according to the weight parameters of the first pedestrian detection network, the weight parameters of the second pedestrian detection network are updated through the exponential moving average algorithm. This application adopts transfer learning to strengthen the transfer ability of the model. Through the method of unsupervised domain adaptation, the model is jointly trained by the labeled data in the existing scene and the unlabeled data in the new scene, so that the model can transfer the data of the existing scene. The performance capability is migrated to the new scene data. The present application only needs to collect unlabeled image data in a new scene, and does not need to re-label a large number of images, which greatly improves the efficiency and thus greatly improves the transfer performance of the model.

In order to make the technical solutions and advantages of the embodiments of the present application more clear, the exemplary embodiments of the present application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and Not all embodiments are exhaustive. It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict.

Example 1

FIG. 1 shows a flowchart of steps of an unsupervised domain adaptation pedestrian detection method according to an embodiment of the present application.

As shown in Figure 1, the pedestrian detection method for unsupervised domain adaptation of the present application specifically includes the following steps:

S101: Randomly select labeled image data and unlabeled image data.

Among them, the labeled image data is randomly selected from the image data whose labeled data is known; the unlabeled image data is randomly selected from the unlabeled image data to be tested.

S102: The labeled image data is enhanced with random data to obtain enhanced labeled image data; the unlabeled image labeled data is enhanced with random data to obtain first enhanced unlabeled image data and second enhanced unlabeled image data, respectively.

Among them, random data enhancement includes image size or pixel random enhancement.

S103: Input the enhanced labeled data to the first pedestrian detection network to obtain the first pedestrian prediction feature; input the first enhanced unlabeled data to the first pedestrian detection network to obtain the second pedestrian prediction feature; Label the image data to the second pedestrian detection network to obtain the third pedestrian prediction feature.

Among them, the first pedestrian detection network and the second pedestrian detection network initially use the same neural network architecture.

S104: Obtain the supervised learning cost according to the label feature of the labeled image data and the first pedestrian prediction feature; obtain the consistency cost according to the second pedestrian prediction feature and the third pedestrian prediction feature.

The first pedestrian prediction feature, the second pedestrian prediction feature, and the third pedestrian prediction feature include size information, classification information, and location information of the image pedestrian.

S105: The total cost is obtained by adding the supervised learning cost and the consistency cost.

S106: According to the total cost, the weight parameter of the first pedestrian detection network is updated through the stochastic gradient descent algorithm.

Among them, supervised learning cost and consistency cost include pedestrian classification loss, pedestrian center point offset loss, pedestrian border width and height loss.

S107: According to the weight parameters of the first pedestrian detection network, the weight parameters of the second pedestrian detection network are updated through an exponential moving average algorithm.

Wherein, the weight parameters of the second pedestrian detection network are obtained by calculating the weights of the first pedestrian detection network by exponential moving average in the training process.

FIG. 2 shows a schematic diagram of steps in an unsupervised domain adaptation pedestrian detection method according to another embodiment of the present application.

As shown in FIG. 2, in some embodiments of the present application, the pedestrian detection method for unsupervised domain adaptation further includes the following steps:

S108: Repeat the above steps S101 to S107 until the first pedestrian detection network and the second pedestrian detection network converge, and obtain the updated first pedestrian detection network and the second pedestrian detection network;

S109: Input the unlabeled image data to be tested into the updated first pedestrian detection network to obtain a pedestrian detection result.

FIG. 3 shows a schematic flowchart of a pedestrian detection method for unsupervised domain adaptation according to an embodiment of the present application.

In the specific implementation of the pedestrian detection method adapted to the supervised domain of the present application, it is first necessary to construct two identical pedestrian detection network models, namely the first pedestrian detection network and the second pedestrian detection network, and the first pedestrian detection network is used as a student model. , the second pedestrian monitoring network as a teacher model.

In this embodiment, the pedestrian detection network may use a network with anchor frames, such as SSD, YOLO-V3 and other network structures; or a pedestrian detection network without anchor frames, such as Center-Net, YOLO-V1 and other network structures.

As shown in Figure 3, the specific steps of the pedestrian detection method adapted to the supervised domain of the present application are as follows:

1) Randomly select training data from labeled image data in the existing scene and unlabeled image data in the new scene, labeled image data (x _s , B _s ) and unlabeled image data x _t , where B _s represents the label of the labeled image data x _s .

x_t做两次随机数据增强处理后，分别得到第一增强无标签图像数据

和第二增强无标签图像数据

2) Labeled image data (x _s , B _s ) are enhanced with labelled data

After _doing two random data enhancement processing, the first enhanced unlabeled image data are obtained respectively.

and the second augmented unlabeled image data

以及第一增强无标签图像数据

分别输入学生网络得到输出特征f_s以及f_t ^S，将第二增强无标签图像数据

输入教师网络得到输出特征f_t ^T，输出特征f_t ^T即伪标签。3) will enhance the labeled data

and the first enhanced unlabeled image data

Input the student network respectively to obtain the output features f _s and f _t ^S , and use the second enhanced unlabeled image data

Input the teacher network to get the output feature f _t ^T , and the output feature f _t ^T is the pseudo-label.

4) Calculate the supervision loss _lsupervised between the output features f _s and the label B _s , and calculate the consistency loss l _consist between the output features f _t ^S and f _t ^T.

5) The total cost l _total is obtained by summing the supervision loss l _supervised and the consistency cost l _consist .

6) Update the weight parameters of the student network through stochastic gradient descent (SGD).

7) Update the student network weights to the teacher network through the exponential moving average (EMA).

8) Go back to step 1) to cycle through this series of steps until the student network converges.

In the testing phase, the image to be detected in the target domain in the new scene is input to the trained student network, and the student network outputs the classification confidence, the frame offset value, and the width and height of the frame. The feature points with the pedestrian classification confidence greater than a certain threshold and their corresponding bounding boxes are used as the final output, and the threshold in this embodiment is set to 0.7.

Specifically, the total cost of the student network, that is, the loss function, is composed of two parts: the supervised loss of the source domain data in the existing scene and the consistency loss of the target domain data in the new scene.

Regarding the supervision loss of the source domain data in the existing scene, taking the Center-Net detection network as an example, the output of each feature point of the network includes the classification information and location information of the object to which it belongs. Among them, the classification information is expressed as the class confidence of each class in the detection task. The position information is expressed as the offset distance of each point from the center of the target, and the length and width of the target frame to which each point belongs.

Therefore, the supervision loss of the network consists of the accumulated pedestrian center point classification loss, pedestrian center point offset loss, and pedestrian bounding box width and height losses.

The total loss of supervision loss l _supervised is the weighted sum of three losses, as shown in formula (1):

L _supervised =L _k +λ _shape L _shape +λ _off L _off Formula (1)

Among them, L _k is the classification loss, L _shape is the width and height loss of the pedestrian frame, L _off is the offset loss of the pedestrian center point, λ _shape is the weight of the width and height loss of the pedestrian frame, and λ _off is the pedestrian center point. Weights for offset loss.

Regarding the consistency loss of the target domain data in the new scene, since the target domain data has no artificial label information, the label information used comes from the output of the teacher network, which is the same as the supervision loss of the source domain data in the existing scene. It also includes classification confidence loss and border position and width and height loss, which will not be repeated here.

Specifically, as shown in Fig. 3, the student network receives two kinds of input data: one input is image data of the source domain, and the other is the image data of the target domain.

The student network outputs features f _s from the image data of the source domain, including pedestrian locations and class confidences in the image data.

Since the labels of the image data of the source domain are known, the supervised loss L _supervised can be calculated by comparing the pedestrian locations and class confidences predicted by the network with the ground-truth labels. During the training process, the gradient descent makes L _supervised smaller and smaller, so that the student network can output more and more accurate pedestrian detection results in the source domain.

On the other hand, the student network outputs features f _t ^S through the image data of the target domain, including pedestrian locations and class confidences in the image data.

Since the target domain has no label information, it cannot be directly used to train the network. The present application proposes a method for constructing "pseudo-labels", inputting image data of different target domains into the teacher network to obtain features f _t ^T , and f _t ^T as "pseudo-labels".

Then, the output feature f _t ^S of the student network is compared with the pseudo-label f _t ^T and the loss L _consist is calculated. During the training process, the gradient descent makes L _consist smaller and smaller, so that the student network and the teacher network can output in Predict increasingly accurate pedestrian detection results on the target domain.

Among them, in order to obtain the above-mentioned "pseudo label", the teacher model constructed in this application is the same as the student model.

During the training process, the weights of the teacher network model are obtained by calculating the exponential moving average (EMA) of the student network weights during the training process. The parameters of the teacher network after this iteration are expressed as follows:

Y _t =λY _t-1 +(1-λ)X _t

Among them, Y _t is the parameter of the teacher network after this iteration, Y _t-1 is the parameter of the teacher network after the last update, X _t is the parameter of the student network after this update, and λ is the model assigned to the last iteration parameter weights.

When t=0, Y is initialized and consistent with X, ie, Y ₀ =X ₀ .

Therefore, the teacher network can be seen as the result of a weighted summation of a series of student networks at different iterative stages. Therefore, the weights of the teacher network are smoother over time. At the same time, it also has stronger generalization ability due to the collection of student networks at different stages.

The stochastic gradient descent method used in the update method of the student network in the embodiment of the present application may also use other optimization update methods.

The teacher network update method in the embodiment of the present application adopts an exponential sliding average weighting method, which can also be replaced by other weighting methods.

The pedestrian detection network in the embodiment of the present application is not limited to a specific deep network model.

In the pedestrian detection method for unsupervised domain adaptation in the embodiment of the present application, the unlabeled images in the target domain are respectively enhanced with different data, and then input to the student network and the teacher network respectively. The teacher network predicts information such as the size, location, and category confidence of pedestrians in the image, and uses this information as a pseudo-label for the student network to guide the student network to learn. After the student network updates the weights, the sliding average is used to update the weights of the teacher network.

At the same time, since the teacher network and the student network receive images enhanced by different data from the same data. By constraining the output of the two networks to be consistent, the student network can learn the potential similarity of the target domain data. Since the teacher network has stronger generalization ability than the student network, the generalization ability of the student network can also be enhanced by making the output of the student network consistent with the output of the teacher network.

Finally, as shown in Figure 3, by continuously iterating the above training process, the student model and the teacher model can promote each other's performance improvement, so the teacher model can predict better pseudo-labels, and the student model can also better adapt to the target domain data Distribution.

Using the pedestrian detection method for unsupervised domain adaptation in the embodiment of the present application, labeled image data and unlabeled image data are randomly selected; labeled image data is enhanced with random data to obtain enhanced labeled image data; The data enhancement obtains the first enhanced unlabeled image data and the second enhanced unlabeled image data; input the enhanced labeled data to the first pedestrian detection network to obtain the first pedestrian prediction feature; input the first enhanced unlabeled data to the first pedestrian detection network. A pedestrian detection network is used to obtain the second pedestrian prediction feature; the second enhanced unlabeled image data is input to the second pedestrian detection network to obtain the third pedestrian prediction feature; according to the label feature of the labeled image data and the first pedestrian prediction feature, Obtain the supervised learning cost; obtain the consistency cost according to the second pedestrian prediction feature and the third pedestrian prediction feature; add the supervised learning cost and the consistency cost to obtain the total cost; according to the total cost, use the stochastic gradient descent algorithm to update the first The weight parameters of the pedestrian detection network; according to the weight parameters of the first pedestrian detection network, the weight parameters of the second pedestrian detection network are updated through the exponential moving average algorithm. This application adopts transfer learning to strengthen the transfer ability of the model, through the method of unsupervised domain adaptation, that is, the model is jointly trained by the labeled data in the existing scene and the unlabeled data in the new scene, so that the model can transfer the existing scene to the Data representation capabilities are migrated to new scene data. This application only needs to collect unlabeled image data in a new scene, and does not need to re-label a large number of images, which greatly saves the human and material consumption caused by data labeling in the development process, improves efficiency, and improves the migration ability of the model. It is more adaptable to changes in the scene.

Example 2

This embodiment provides an unsupervised domain-adapted pedestrian detection system. For details not disclosed in the unsupervised domain-adapted pedestrian detection system of this embodiment, please refer to the unsupervised domain-adapted pedestrian detection method in other embodiments implementation content.

FIG. 4 shows a schematic structural diagram of an unsupervised domain-adapted pedestrian detection system according to an embodiment of the present application.

As shown in FIG. 4 , the pedestrian detection system for unsupervised domain adaptation according to the embodiment of the present application includes a training data selection module 10 , a data enhancement module 20 , a feature prediction network module 30 , a supervised learning cost module 40 , a consistency cost module 50 , and a total The cost module 60 , the first pedestrian detection network update module 70 and the second pedestrian detection network update module 80 .

specific:

Training data selection module 10: used to randomly select labeled image data and unlabeled image data.

Data enhancement module 20: used for enhancing labeled image data through random data enhancement to obtain enhanced labeled image data; for enhancing unlabeled image label data through random data enhancement to obtain first enhanced unlabeled image data and second enhanced unlabeled image data respectively image data.

Feature prediction network module 30: used to input the enhanced labeled data to the first pedestrian detection network to obtain the first pedestrian prediction feature; used to input the first enhanced unlabeled data to the first pedestrian detection network to obtain the second pedestrian Prediction feature; used to input the second enhanced unlabeled image data to the second pedestrian detection network to obtain the third pedestrian prediction feature.

Supervised learning cost module 40: used to obtain the supervised learning cost according to the label feature of the labeled image data and the first pedestrian prediction feature.

Consistency cost module 50: used to obtain the consistency cost according to the second pedestrian prediction feature and the third pedestrian prediction feature.

The total cost module 60 is used to add the supervised learning cost and the consistency cost to obtain the total cost.

The first pedestrian detection network updating module 70 is used for updating the weight parameter of the first pedestrian detection network through the stochastic gradient descent algorithm according to the total cost.

The second pedestrian detection network update module 80 is configured to update the weight parameters of the second pedestrian detection network through the exponential moving average algorithm according to the weight parameters of the first pedestrian detection network.

In some embodiments of the present application, the unsupervised domain-adapted pedestrian detection system further includes:

The training convergence module is used to repeat the above steps until the first pedestrian detection network and the second pedestrian monitoring network converge, and the updated first pedestrian detection network and the second pedestrian monitoring network are obtained.

Pedestrian detection module: used to input the unlabeled image data to be tested into the updated first pedestrian detection network to obtain pedestrian detection results.

FIG. 3 also shows a schematic diagram of an application flow of the pedestrian detection system for unsupervised domain adaptation according to an embodiment of the present application.

First, two identical pedestrian detection network models need to be constructed, namely the first pedestrian detection network and the second pedestrian detection network, the first pedestrian detection network is used as the student model, and the second pedestrian detection network is used as the teacher model.

In this embodiment, the pedestrian detection network may use a network with anchor frames, such as SSD, YOLO-V3 and other network structures; or a pedestrian detection network without anchor frames, such as Center-Net, YOLO-V1 and other network structures.

As shown in FIG. 3 , the specific steps of the application of the pedestrian detection system adapted to the supervised domain of the present application are as follows:

1) Randomly select training data from labeled image data in the existing scene and unlabeled image data in the new scene, labeled image data (x _s , B _s ) and unlabeled image data x _t , where B _s represents the label of the labeled image data x _s .

x_t做两次随机数据增强处理后，分别得到第一增强无标签图像数据

和第二增强无标签图像数据

2) Labeled image data (x _s , B _s ) are enhanced with labelled data

After _doing two random data enhancement processing, the first enhanced unlabeled image data are obtained respectively.

and the second augmented unlabeled image data

以及第一增强无标签图像数据

分别输入学生网络得到输出特征f_s以及f_t ^S，将第二增强无标签图像数据

输入教师网络得到输出特征f_t ^T，输出特征f_t ^T即伪标签。3) will enhance the labeled data

and the first enhanced unlabeled image data

Input the student network respectively to obtain the output features f _s and f _t ^S , and use the second enhanced unlabeled image data

Input the teacher network to get the output feature f _t ^T , and the output feature f _t ^T is the pseudo-label.

4) Calculate the supervision loss _lsupervised between the output features f _s and the label B _s , and calculate the consistency loss l _consist between the output features f _t ^S and f _t ^T.

5) The total cost l _total is obtained by summing the supervision loss l _supervised and the consistency cost l _consist .

6) Update the weight parameters of the student network through stochastic gradient descent (SGD).

7) Update the student network weights to the teacher network through the exponential moving average (EMA).

8) Go back to step 1) to cycle through this series of steps until the student network converges.

In the testing phase, the image to be detected in the target domain in the new scene is input to the trained student network, and the student network outputs the classification confidence, the frame offset value, and the width and height of the frame. The feature points with the pedestrian classification confidence greater than a certain threshold and their corresponding bounding boxes are used as the final output, and the threshold in this embodiment is set to 0.7.

Specifically, the total cost of the student network, that is, the loss function, is composed of two parts: the supervised loss of the source domain data in the existing scene and the consistency loss of the target domain data in the new scene.

Regarding the supervision loss of the source domain data in the existing scene, taking the Center-Net detection network as an example, the output of each feature point of the network includes the classification information and location information of the object to which it belongs. Among them, the classification information is expressed as the class confidence of each class in the detection task. The position information is expressed as the offset distance of each point from the center of the target, and the length and width of the target frame to which each point belongs.

Therefore, the supervision loss of the network consists of the accumulated pedestrian center point classification loss, pedestrian center point offset loss, and pedestrian bounding box width and height losses.

In the pedestrian detection system for unsupervised domain adaptation in the embodiment of the present application, the unlabeled images in the target domain are respectively enhanced with different data, and then input to the student network and the teacher network respectively. The teacher network predicts information such as the size, location, and category confidence of pedestrians in the image, and uses this information as a pseudo-label for the student network to guide the student network to learn. After the student network updates the weights, the sliding average is used to update the weights of the teacher network.

At the same time, since the teacher network and the student network receive images enhanced by different data from the same data. By constraining the output of the two networks to be consistent, the student network can learn the potential similarity of the target domain data. Since the teacher network has stronger generalization ability than the student network, the generalization ability of the student network can also be enhanced by making the output of the student network consistent with the output of the teacher network.

Finally, as shown in Figure 3, by continuously iterating the above training process, the student model and the teacher model can promote each other's performance improvement, so the teacher model can predict better pseudo-labels, and the student model can also better adapt to the target domain data Distribution.

Using the pedestrian detection system for unsupervised domain adaptation in the embodiment of the present application, the labeled image data and the unlabeled image data are randomly selected; the labeled image data is enhanced by random data to obtain enhanced labeled image data; the unlabeled image labeled data is randomly The data enhancement obtains the first enhanced unlabeled image data and the second enhanced unlabeled image data; input the enhanced labeled data to the first pedestrian detection network to obtain the first pedestrian prediction feature; input the first enhanced unlabeled data to the first pedestrian detection network. A pedestrian detection network is used to obtain the second pedestrian prediction feature; the second enhanced unlabeled image data is input to the second pedestrian detection network to obtain the third pedestrian prediction feature; according to the label feature of the labeled image data and the first pedestrian prediction feature, Obtain the supervised learning cost; obtain the consistency cost according to the second pedestrian prediction feature and the third pedestrian prediction feature; add the supervised learning cost and the consistency cost to obtain the total cost; according to the total cost, use the stochastic gradient descent algorithm to update the first The weight parameters of the pedestrian detection network; according to the weight parameters of the first pedestrian detection network, the weight parameters of the second pedestrian detection network are updated through the exponential moving average algorithm. This application adopts transfer learning to strengthen the transfer ability of the model, through the method of unsupervised domain adaptation, that is, the model is jointly trained by the labeled data in the existing scene and the unlabeled data in the new scene, so that the model can transfer the existing scene to the Data representation capabilities are migrated to new scene data. The present application only needs to collect unlabeled image data in a new scene, and does not need to re-label a large number of images, which greatly improves the efficiency and thus greatly improves the transfer performance of the model.

Example 3

This embodiment provides a computer-readable storage medium on which a computer program is stored; the computer program is executed by a processor to implement the pedestrian detection method for unsupervised domain adaptation in other embodiments.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The terminology used in the present invention is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in this specification and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, such information should not be limited by these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, without departing from the scope of the present invention. Depending on the context, the word "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determining."

While the preferred embodiments of the present application have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of this application.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (10)

1. An unsupervised domain adapted pedestrian detection method, comprising the steps of:

randomly selecting labeled image data and unlabeled image data;

the tagged image data is enhanced through random data to obtain enhanced tagged image data; the label-free image label data are subjected to random data enhancement to respectively obtain first enhanced label-free image data and second enhanced label-free image data;

inputting the enhanced labeled data to a first pedestrian detection network to obtain a first pedestrian prediction characteristic; inputting the first enhanced non-tag data to a first pedestrian detection network to obtain a second pedestrian prediction characteristic; inputting the second enhanced unlabeled image data to a second pedestrian detection network to obtain a third pedestrian prediction feature;

obtaining a supervised learning cost according to the label characteristics of the labeled image data and the first pedestrian prediction characteristics; obtaining consistency cost according to the second pedestrian prediction characteristic and the third pedestrian prediction characteristic;

adding the supervised learning cost and the consistency cost to obtain a total cost;

updating the weight parameter of the first pedestrian detection network through a random gradient descent algorithm according to the total cost;

and updating the weight parameter of the second pedestrian detection network through an exponential moving average algorithm according to the weight parameter of the first pedestrian detection network.

2. The unsupervised domain adapted pedestrian detection method of claim 1, further comprising:

repeating the steps until the first pedestrian detection network and the second pedestrian monitoring network are converged to obtain an updated first pedestrian detection network and an updated second pedestrian monitoring network;

and inputting the unlabeled image data to be detected into the updated first pedestrian detection network to obtain a pedestrian detection result.

3. The unsupervised domain adapted pedestrian detection method of claim 1, wherein the tagged image data is randomly selected from image data for which tag data is known; and randomly selecting the non-label image data from the to-be-detected non-label image data.

4. The unsupervised domain adapted pedestrian detection method of claim 1, wherein the first, second and third pedestrian prediction features comprise image pedestrian size information, classification information and location information.

5. The unsupervised domain adapted pedestrian detection method of claim 3, wherein the supervised learning cost and consistency cost comprise a pedestrian classification loss, a pedestrian center point offset loss, a pedestrian border width and a height loss.

6. The unsupervised domain adapted pedestrian detection method of claim 1, wherein the first and second pedestrian detection networks initially employ the same neural network architecture.

7. The unsupervised domain adapted pedestrian detection method of claim 1, wherein the weight parameters of the second pedestrian detection network are obtained by exponential moving average in a training process of weight calculation of the first pedestrian detection network.

8. A pedestrian detection system adaptive to an unsupervised domain is characterized by specifically comprising:

a training data selection module: the image processing device is used for randomly selecting image data with labels and image data without labels;

the data enhancement module: the image processing device is used for enhancing the labeled image data through random data to obtain enhanced labeled image data; the image enhancement module is used for enhancing the label-free image data through random data to respectively obtain first enhanced label-free image data and second enhanced label-free image data;

the characteristic prediction network module: the system is used for inputting the enhanced labeled data to a first pedestrian detection network to obtain a first pedestrian prediction characteristic; the first enhanced non-tag data is input to a first pedestrian detection network to obtain a second pedestrian prediction characteristic; the second enhanced unlabeled image data is input to a second pedestrian detection network to obtain a third pedestrian prediction characteristic;

a supervised learning cost module: the monitoring learning cost is obtained according to the label features of the labeled image data and the first pedestrian prediction features;

a consistency cost module: the system is used for obtaining consistency cost according to the second pedestrian prediction characteristic and the third pedestrian prediction characteristic;

a total cost module: the device is used for adding the supervised learning cost and the consistency cost to obtain a total cost;

a first pedestrian detection network update module: the weight parameter of the first pedestrian detection network is updated through a random gradient descent algorithm according to the total cost;

a second pedestrian detection network update module: and the weight parameter updating module is used for updating the weight parameter of the second pedestrian detection network through an exponential moving average algorithm according to the weight parameter of the first pedestrian detection network.

9. The unsupervised domain adapted pedestrian detection system of claim 8, further comprising:

training a convergence module: the pedestrian detection method comprises the steps of obtaining a first pedestrian detection network and a second pedestrian monitoring network after updating, and repeating the steps until the first pedestrian detection network and the second pedestrian monitoring network converge to obtain the updated first pedestrian detection network and the updated second pedestrian monitoring network;

a pedestrian detection module: and the pedestrian detection system is used for inputting the unlabeled image data to be detected into the updated first pedestrian detection network to obtain a pedestrian detection result.

10. A computer-readable storage medium, having stored thereon a computer program; a computer program for execution by a processor for implementing an unsupervised domain adapted pedestrian detection method according to any one of claims 1-7.

Images