CN117636086B - Passive domain adaptive target detection method and device - Google Patents

Abstract

The present invention provides a passive domain adaptive target detection method and device, comprising: constructing multiple feature prototypes of each type of target based on the first instance features of each type of target extracted from part of the images of the target domain data set by a teacher model; correcting the target detection results of each image in the target domain data set obtained by the teacher model according to the multiple feature prototypes of each type of target, and obtaining pseudo labels of each image; using each image of the target domain data set as a sample, using the pseudo labels of each image as a label to train a student model, and using the trained student model to detect targets in the image to be detected; the teacher model and the student model are obtained by pre-training a target detection model using a source domain data set. The present invention uses multiple feature prototypes of various types of targets in the target domain to guide the generation of more accurate pseudo labels as supervisory information for model training, thereby improving the accuracy of target detection.

Description

Passive domain adaptive target detection method and device

Technical Field

The present invention relates to the field of computer vision technology, and in particular to a passive domain adaptive target detection method and device.

Background Art

When performing target detection, if the target detection model trained with the training set is used to detect targets in a new environment, there is a problem of inconsistent data distribution between the training set (source domain) and the test set (target domain), resulting in poor performance of the trained target detection model in the new environment. Collecting and annotating a large enough dataset for a new environment is time-consuming, labor-intensive, and costly.

To solve this problem, passive domain adaptive target detection was proposed, which aims to transfer the knowledge of the target detection model trained in the source domain to the target domain. Without the need to label and access the target domain data, the detection performance of the target detection model in the target domain is improved, greatly reducing the labeling cost.

Existing passive domain adaptation object detection methods usually adopt a pseudo-label generation strategy. The object detection model pre-trained on the source domain generates pseudo-labels for the target domain dataset, which are used as supervision information on the target domain to fine-tune the object detector. In this type of method, the quality of the pseudo-labels affects the performance of the object detection model after adaptation. For example, the class imbalance problem will introduce noisy pseudo-labels because common categories receive more attention, affecting the performance of object detection for rare categories. Therefore, problems such as inaccurate pseudo-labels and the sensitivity of the object detection model to domain shift lead to low accuracy of object detection.

Summary of the invention

The present invention provides a passive domain adaptive target detection method and device, which are used to solve the defect that the quality of pseudo labels generated by the target detection model for the target domain data set in the prior art is poor, which affects the accuracy of target detection, and improves the quality of pseudo labels, thereby improving the accuracy of target detection.

The present invention provides a passive domain adaptive target detection method, comprising:

Based on the first instance features of various types of targets extracted by the teacher model from some images of the target domain dataset, multiple feature prototypes of the various types of targets are constructed;

According to the multiple feature prototypes of the various types of targets, the target detection results of each image in the target domain dataset obtained by the teacher model are corrected to obtain pseudo labels of each image;

Using each image of the target domain data set as a sample, using the pseudo label of each image as a label to train the student model, and using the trained student model to detect the target in the image to be detected;

The teacher model and the student model are obtained by pre-training the target detection model using a source domain dataset.

According to a passive domain adaptation target detection method provided by the present invention, the first instance features of various targets extracted from part of the images of the target domain data set based on the teacher model are used to construct multiple feature prototypes of the various targets, including:

Randomly extracting some images from the target domain dataset;

Detecting a first category label and a first detection frame of each target in the partial image based on the teacher model, and using an image area within the first detection frame of the target as the first instance feature;

Determining objects in the partial image that belong to the same category according to the first category labels of the objects;

According to the first instance features of each type of target, a plurality of feature prototypes of each type of target are constructed.

According to a passive domain adaptive target detection method provided by the present invention, the step of constructing multiple feature prototypes of each type of target based on first instance features of each type of target includes:

Clustering the first instance features of the various types of targets multiple times, with the number of clusters in each clustering being different;

The silhouette score of each clustering is determined, and the cluster center of each cluster in the cluster corresponding to the largest silhouette score is used as the feature prototype of each type of target.

According to a passive domain adaptive target detection method provided by the present invention, the target detection result of each image in the target domain data set obtained by the teacher model is corrected according to the multiple feature prototypes of the various types of targets to obtain a pseudo label of each image, including:

Detecting a second category label, a second detection frame, and a confidence score that each target belongs to the second category label of each target from each image of the target domain dataset based on the teacher model;

Using the image area within the second detection frame of each target as the second instance feature of each target;

Correcting the second category label of each target and the confidence score that each target belongs to the second category label according to the similarity between the second instance feature of each target and the plurality of feature prototypes of each type of target;

A pseudo label for each target is determined according to the second detection frame, the corrected second category label, and the corrected confidence score of each target.

According to a passive domain adaptive target detection method provided by the present invention, the second category label of each target and the confidence score that each target belongs to the second category label are corrected according to the similarity between the second instance feature of each target and the multiple feature prototypes of each type of target, including:

Determine a first maximum value among similarities between the second instance feature of each target and a plurality of feature prototypes of each type of target;

According to the first category label corresponding to the maximum value among the first maximum values corresponding to all categories of targets, as the second category label after correction of each target;

The maximum value among the first maximum values corresponding to all categories of targets is used as the corrected confidence score of each target.

According to a passive domain adaptation target detection method provided by the present invention, the step of using the image area within the second detection frame of each target as the second instance feature of each target includes:

Comparing the confidence score of each target belonging to the second category label with a preset threshold, where the preset threshold is determined according to the number of preset detection categories corresponding to the target detection model;

When the confidence score is less than or equal to the preset threshold, the image area within the second detection frame of each target is used as the second instance feature of each target.

According to a passive domain adaptation target detection method provided by the present invention, each image of the target domain data set is used as a sample, and the pseudo label of each image is used as a label to train the student model, including:

Detecting a third category label, a third detection frame, and a confidence score that each target belongs to the third category label of each target from each image of the target domain dataset based on the student model;

Using the image area within the third detection frame of each target as the third instance feature of each target;

Determine a first loss for each target according to a loss between a third category label of each target in each image of the target domain dataset and the corrected second category label, and a loss between a third detection frame and a second detection frame;

Determine the second loss of each target according to the similarity between the second instance feature and the third instance feature of each target and the plurality of feature prototypes of each type of target;

Determining a third loss for each target according to the confidence score that each target belongs to the third category label and the corrected confidence score;

The student model is trained according to the first loss, the second loss and the third loss.

According to a passive domain adaptive target detection method provided by the present invention, the second loss of each target is determined according to the similarity between the second instance feature and the third instance feature of each target and the plurality of feature prototypes of each type of target, including:

Determine a first maximum value among similarities between the second instance feature of each target and a plurality of feature prototypes of each type of target;

Determine a second maximum value of similarities between the third instance feature of each target and a plurality of feature prototypes of each type of target;

A second loss of each of the targets is determined according to the first maximum value and the second maximum value.

According to a passive domain adaptation target detection method provided by the present invention, while taking each image of the target domain data set as a sample and taking the pseudo label of each image as a label to train the student model, it also includes:

Determining new cluster centers of each class of targets according to the second instance features of each target newly generated in each preset number of iterations during the training process;

Selecting the feature prototypes that are most similar to the new cluster centers of the various types of targets from the feature prototypes of the various types of targets;

The new cluster centers of the various types of targets are used to update the selected feature prototypes of the various types of targets.

The present invention also provides a passive domain adaptive target detection device, comprising:

A construction module, configured to construct a plurality of feature prototypes of each type of target based on first instance features of each type of target extracted by the teacher model from a portion of images of the target domain dataset;

A generation module, configured to correct the target detection result of each image in the target domain dataset obtained by the teacher model according to the multiple feature prototypes of the various types of targets, so as to obtain a pseudo label of each image;

The training module is used to use each image of the target domain data set as a sample, use the pseudo label of each image as a label to train the student model, and use the trained student model to detect the target in the image to be detected.

The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the passive domain adaptive target detection method as described above is implemented.

The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the passive domain adaptive target detection method as described above is implemented.

The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the passive domain adaptive target detection method as described above is implemented.

The passive domain adaptive target detection method and device provided by the present invention constructs multiple feature prototypes of each type of target by using the first instance features of each type of target extracted from part of the images of the target domain data set by the teacher model, thereby providing more representative category information for the target domain. After the target detection results of each image in the target domain data set predicted by the teacher model are corrected by the multiple feature prototypes of each type of target, more accurate pseudo labels are obtained as supervision information for the student model training, thereby improving the accuracy of target detection.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram of a flow chart of a passive domain adaptive target detection method provided by the present invention;

FIG2 is a schematic diagram of a framework of a passive domain adaptive target detection method provided by the present invention;

FIG3 is a schematic diagram of the structure of a passive domain adaptive target detection device provided by the present invention;

FIG. 4 is a schematic diagram of the structure of an electronic device provided by the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

A passive domain adaptive target detection method of the present invention is described below in conjunction with FIG1 , including:

Step 101, constructing multiple feature prototypes of various types of targets based on first instance features of various types of targets extracted by the teacher model from some images of the target domain dataset;

Target domain dataset The dataset contains N _t images, and the images in the target domain dataset are not labeled. Extract part of the image.

The extracted partial image is input into the teacher model θ _tea for target detection, and the first instance feature f _t of the target in the partial image is obtained, that is, the image area within the detection frame of the target and the first category label of the target.

According to the first category label of the target, each type of target in the partial image is counted. The first instance features of each type of target are analyzed to obtain multiple feature prototypes of each type of target. The multiple feature prototypes of each type of target are used to represent the features of each type of target.

The cluster centers adaptively generated according to the distribution of the first instance features of each type of target can be used as multiple feature prototypes of each type of target, thereby providing more representative category information for the target domain. The migration of knowledge from the source domain to the target domain is guided by constructing category-specific multi-feature prototypes. This embodiment does not limit the construction method of the feature prototype.

Step 102, according to multiple feature prototypes of various types of targets, correct the target detection results of each image in the target domain data set obtained by the teacher model to obtain a pseudo label for each image;

Each image in the target domain dataset is input into the teacher model to obtain the object detection results of each image.

Before each image in the target domain dataset is input into the teacher model, each image can be weakly enhanced, including random horizontal flipping. That is, the teacher model inputs a weakly enhanced image.

Since the teacher model is trained using the source domain dataset, the pseudo labels predicted by the teacher model for each image are inaccurate. Multiple feature prototypes specific to each type of target in the target domain dataset are introduced to correct the target detection results of each image predicted by the teacher model, so as to assign more accurate pseudo labels to the targets in each image in the presence of intra-class variations.

More accurate pseudo labels can be obtained based on the distance between the second instance features of the target extracted by the teacher model from each image in the target domain dataset and multiple feature prototypes of each type of target.

Step 103, using each image of the target domain data set as a sample, using the pseudo label of each image as a label to train the student model, using the trained student model to detect the target in the image to be detected; using the target detection model trained using the source domain data set as the teacher model and the student model.

The source domain dataset includes multiple images, and the objects in each image are annotated with category labels and detection boxes.

The target detection model may adopt a two-stage target detector Faster-RCNN (Faster Region Convolutional Neural Networks), but is not limited to this model.

The target detection model is trained on the source domain dataset to obtain an initial target detection model θ. The initial target detection model θ is used as the initial teacher model θ _tea and the initial student model θ _tea . After the target detection model is trained on the source domain dataset, the source domain dataset is no longer used.

Each image of the target domain dataset is input as a sample into the student model θ _stu to obtain the target detection result predicted by the student model θ _stu . According to the difference between the target detection result predicted by the student model θ _stu and the corresponding pseudo label, the parameters of the student model θ _stu are adjusted so that the difference between the target detection result predicted by the student model θ _stu and the corresponding pseudo label is less than the set value.

The parameters of the teacher model are updated using the exponential moving average method based on the parameters of the student model, and the formula is as follows:

θ _tea =η·θ _tea +(1-η)·θ _stu ;

Among them, η is the coefficient of the exponential moving average method, which can be set to 0.99.

The updated teacher model continues forward reasoning, and the trained student model can detect objects in new environments.

Before each image in the target domain data set is input into the student model, each image may be strongly enhanced, including one or more of random grayscale, Gaussian blur and color jitter. That is, the student model inputs a strongly enhanced image. The framework diagram of the passive domain adaptation target detection method provided in this embodiment is shown in FIG2.

Experiments are conducted on multiple domain adaptation object detection benchmarks, including Cityscapes, Foggy Cityscapes, KITTI, Sim10k, BDD100k, PASCAL VOC, and Watercolor datasets. The experimental results show that the method provided in this embodiment outperforms other passive domain adaptation object detection methods in terms of performance.

This embodiment constructs multiple feature prototypes of each type of target by using the first instance features of each type of target extracted by the teacher model from some images of the target domain dataset, thereby providing more representative category information for the target domain. After the target detection results of each image in the target domain dataset predicted by the teacher model are corrected by the multiple feature prototypes of each type of target, more accurate pseudo-labels are obtained as supervision information for student model training, thereby improving the accuracy of target detection.

On the basis of the above embodiment, in this embodiment, based on the first instance features of various types of targets extracted by the teacher model from some images of the target domain data set, multiple feature prototypes of various types of targets are constructed, including:

Randomly extract some images from the target domain dataset;

Detecting a first category label and a first detection frame of each target in a partial image based on the teacher model, and using an image region within the first detection frame of the target as a first instance feature;

According to the first category label of each target, determine the targets belonging to the same category in some images;

According to the first instance features of each category of targets, multiple feature prototypes of each category of targets are constructed.

From the target domain dataset Randomly select some images from , such as randomly sampling 500 images.

The teacher model is used to perform target detection on each image in some images to obtain the first category label, the first detection frame and the confidence score of each target belonging to the first category label for each target in each image.

When determining the first category label of the target, the teacher model first predicts the confidence score that the target belongs to each preset category label, and takes the preset category label with the highest confidence score as the first category label of the target.

For each type of target in each image of the extracted partial images, only the instance feature of the target with the highest confidence score in each type of target can be retained as the first instance feature with category representativeness

According to the first instance features of the target with the same first category label in the partial image Build multiple feature prototypes for various types of targets.

On the basis of the above embodiment, in this embodiment, multiple feature prototypes of various types of targets are constructed according to the first instance features of various types of targets, including:

The first instance features of each type of target are clustered multiple times, with a different number of clusters each time;

Determine the silhouette score of each clustering, and take the cluster center of each cluster in the cluster corresponding to the largest silhouette score as the feature prototype of each type of target.

The first instance features of the i-th category target are clustered multiple times. Each clustering divides the first instance features of the i-th category target into multiple groups, so that the difference between the first instance features in each group is small, and the difference between the first instance features of different groups is large. The number of groups into which the first instance features are divided is the number of clusters.

Calculate the silhouette score of each clustering. The silhouette score is used to characterize the compactness of each cluster in the clustering. The larger the value, the better. The average of the silhouette scores of each cluster in each clustering can be used as the silhouette score S of each clustering. Keep the clustering with the largest silhouette score, and use the number of clusters n in this clustering as the number of characteristic prototypes of the i-th target.

The cluster center of each cluster in this clustering is used as the characteristic prototype of the i-th type of target in is the j-th feature prototype of the i-th category target, |G _ij | is the number of first instance features of the j-th cluster G _ij belonging to the i-th category target.

For example, the first instance feature of the i-th target is clustered three times, and the first instance feature of the i-th target is divided into 2, 3, and 4 clusters respectively. The silhouette score of the first clustering is 0.82, the silhouette score of the second clustering is 0.92, and the silhouette score of the third clustering is 0.88. Then the number of feature prototypes of the i-th target is the number of clusters divided in the second clustering, 2. The feature prototype of the i-th target is the cluster center of the two clusters in the second clustering.

On the basis of the above embodiment, in this embodiment, according to multiple feature prototypes of various types of targets, the target detection results of each image in the target domain data set obtained by the teacher model are corrected to obtain pseudo labels of each image, including:

Based on the teacher model, detect the second category label, the second detection box and the confidence score that each target belongs to the second category label of each target from each image of the target domain dataset;

Using the image area within the second detection frame of each target as the second instance feature of each target;

According to the similarity between the second instance feature of each target and multiple feature prototypes of each target, the second category label of each target and the confidence score of each target belonging to the second category label are corrected;

A pseudo label of each target is determined according to the second detection frame of each target, the corrected second category label and the corrected confidence score.

The i-th image in the target domain dataset Input the teacher model and get the target detection results predicted by the teacher model:

in, is the second category label of the target in the i-th image in the predicted target domain dataset, is the second detection box of the target in the i-th image in the predicted target domain dataset, is the confidence score that the target output by the classification layer such as softmax belongs to the second category label, and F is the teacher model.

Due to the domain shift between the source domain and the target domain, the object detection results contain noisy samples, such as misclassified positive samples.

In order to further correct the misclassified positive samples, the target detection results are corrected using the similarity between the second instance features of the target and multiple feature prototypes of each type of target, and the pseudo labels of each target are obtained.

On the basis of the above embodiment, in this embodiment, according to the similarity between the second instance feature of each target and multiple feature prototypes of each type of target, the second category label of each target and the confidence score of each target belonging to the second category label are corrected, including:

Determine a first maximum value among similarities between a second instance feature of each target and a plurality of feature prototypes of each class of targets;

The first category label corresponding to the maximum value among the first maximum values corresponding to all categories of targets is used as the corrected second category label of each target;

The maximum value among the first maximum values corresponding to all class targets is taken as the corrected confidence score of each target.

According to the similarity between the second instance features of each target and multiple feature prototypes of each type of target, the formula for correcting and filtering the target detection results predicted by the teacher model can be expressed as follows:

Among them, ft _′ is the second instance feature corresponding to the second detection box _Bt , is the nth feature prototype of the i-th target, and _ft ′ is The similarity between them can be expressed by the dot product of the two, Nc is the total number of target categories, is the nth feature prototype of the jth target, is the corrected second category label, exp represents the exponential function, is the corrected confidence score, τ is the set constant, is the final pseudo label.

On the basis of the above embodiment, in this embodiment, the image area within the second detection frame of each target is used as the second instance feature of each target, including:

Compare the confidence score of each target belonging to the second category label with a preset threshold, where the preset threshold is determined according to the number of preset detection categories corresponding to the target detection model;

When the confidence score is less than or equal to a preset threshold, the image area within the second detection frame of each target is used as the second instance feature of each target.

Due to the domain offset between the source domain and the target domain, the target detection results contain noise samples, such as negative samples. In order to filter out negative samples, we filter them by confidence score:

Where C is the number of preset detection categories corresponding to the target detection model. When performing target detection, the target detection model outputs the confidence score that the target belongs to each preset detection category, and takes the preset detection category with the highest confidence score as the category label of the target.

According to the target detection result after filtering out the negative samples, the second instance feature of the target is determined, and the target detection result after filtering out the negative samples is further corrected according to the second instance feature of the target.

Based on the above embodiment, in this embodiment, each image of the target domain data set is used as a sample, and the pseudo label of each image is used as a label to train the student model, including:

Based on the student model, a third category label, a third detection box, and a confidence score that each target belongs to the third category label are detected from each image of the target domain dataset;

Using the image area within the third detection frame of each target as the third instance feature of each target;

Determine a first loss for each target according to a loss between a third category label of each target in each image of the target domain dataset and a corrected second category label, and a loss between a third detection frame and a second detection frame;

Determine the second loss of each target according to the similarity between the second instance feature, the third instance feature of each target and multiple feature prototypes of each type of target;

Determine the third loss of each target based on the confidence score of each target belonging to the third category label and the corrected confidence score;

The student model is trained based on the first loss, the second loss, and the third loss.

For the consistency learning strategy of object detection, the object detection model reduces the sensitivity to domain changes by maintaining the consistency between the original image and the style perturbation image. Existing studies apply consistency regularization at a single prediction level, which has limited performance improvement, or introduce auxiliary branches to construct consistency regularization, which increases the number of model parameters.

This embodiment designs multi-level consistency regularization, including prototype-based consistency regularization and prediction consistency regularization, which further improves the performance of consistency learning in alleviating the domain shift problem and introduces fewer additional model parameters.

The pseudo labels of each image in the target domain dataset are used as the supervision information of the student model to construct the self-training loss, i.e. the first loss. The self-training loss function is constructed as:

in, and They represent the classification loss and the detection box regression loss of the target in each image. The classification loss is obtained based on the third category label of the target predicted by the student model and the corrected second category label in the pseudo label. The detection box regression loss is obtained based on the third detection box of the target predicted by the student model and the second detection box in the pseudo label.

By utilizing the more accurate pseudo labels generated by multiple prototypes as supervision signals for the student model in the unlabeled target domain, it helps to improve the detection performance of the object detector in the target domain.

For prototype-based consistency regularization, it can be achieved by using multi-category prototypes to calculate the category probability distribution of instance features. Specifically, the category probability distribution of the second instance feature can be determined based on the similarity between the second instance feature extracted by the teacher model and multiple feature prototypes of each category of targets. The category probability distribution of the third instance feature can be determined based on the similarity between the third instance feature extracted by the student model and multiple feature prototypes of each category of targets. Based on the category probability distribution of the second instance feature and the category probability distribution of the third instance feature, a prototype-based consistency loss is constructed. That is the second loss.

Training according to the second loss can minimize the difference between the category probability distribution of instance features extracted by the teacher model and the student model, effectively reducing the sensitivity of the target detection model to changes in the data domain.

The confidence scores predicted by the teacher model and the student model are used to construct the prediction consistency regularization loss, i.e., the third loss. For each image in the target domain dataset, keeping the predictions of weakly enhanced images and strongly enhanced images consistent helps the object detection model learn better transferable features.

Denote the confidence scores of the teacher model and the student model predictions as P _stu and P _stu , respectively, and define the prediction consistency regularization loss as:

Use the constructed first loss, second loss, and third loss to supervise the student model training. Update the parameters of the student model θ _stu through the objective function shown below:

in, is the self-training loss of the student model supervised by pseudo labels, is the prototype-based consistency regularization loss, is the prediction consistency regularization loss.

On the basis of the above embodiment, in this embodiment, the second loss of each target is determined according to the similarity between the second instance feature and the third instance feature of each target and multiple feature prototypes of each type of target, including:

Determine a first maximum value among similarities between a second instance feature of each target and a plurality of feature prototypes of each class of targets;

Determine the second maximum value of the similarities between the third instance feature of each target and the plurality of feature prototypes of each class of targets;

A second loss for each target is determined based on the first maximum value and the second maximum value.

The first maximum value among the similarities between the second instance feature f _t extracted by the teacher model and the multiple feature prototypes of each class of targets can be determined as the similarity between the second instance feature and the feature prototypes of each class of targets. The sum of the first maximum values corresponding to all class targets is determined, and the ratio between the first maximum value corresponding to each class of targets and the sum of the first maximum values corresponding to all class targets is used as the category probability distribution Z _tea of the second instance feature.

The third instance feature extracted by the student model can be determined The second maximum value among the similarities between the multiple feature prototypes of each class of targets is used as the similarity between the third instance feature and the feature prototypes of each class of targets. The sum of the second maximum values corresponding to all class targets is determined, and the ratio between the second maximum values corresponding to each class of targets and the sum of the second maximum values corresponding to all class targets is used as the category probability distribution Z _stu of the third instance feature, and the formula is as follows:

Among them, sim is a similarity function, which can be a cosine distance.

By minimizing the prototype-based consistency regularization loss, i.e., the second loss To force consistency between Z _stu and Z _twa :

in, is the Kullback-Leibler divergence, which measures the degree of difference between the probability distributions of two classes.

On the basis of the above embodiment, in this embodiment, while taking each image of the target domain data set as a sample and taking the pseudo label of each image as a label to train the student model, the following is also included:

Determine new cluster centers of each type of target according to the second instance features of each target newly generated at each preset number of iterations during the training process;

Select the feature prototype that is most similar to the new cluster center of each type of target from the feature prototypes of each type of target;

Using the new cluster centers of each class of targets, the feature prototypes of each class of selected targets are updated.

The feature prototypes of various types of targets are dynamically updated according to the second instance features extracted by the teacher model. The second instance features extracted by the teacher model can be stored in a memory bank.

During the training of the student model, the teacher model extracts the second instance features of an image in each iteration to generate a pseudo label. For each preset number of iterations, such as 100 iterations, the teacher model generates the second instance features of the targets in 100 images. The average value of the second instance features of each class of targets is used as the new cluster center of each class of targets. C represents the number of target categories and clears the memory bank.

Select the prototype that is most similar to the new cluster center _vi among the feature prototypes of the same type of target Prototypes can be updated with momentum strategies

Where α is the momentum coefficient and can be set to 0.99.

The passive domain adaptive target detection device provided by the present invention is described below. The passive domain adaptive target detection device described below and the passive domain adaptive target detection method described above can be referred to each other.

As shown in FIG3 , the device includes a construction module 301, a generation module 302 and a training module 303, wherein:

The construction module 301 is used to construct multiple feature prototypes of various types of targets based on the first instance features of various types of targets extracted by the teacher model from the partial images of the target domain data set;

The generation module 302 is used to correct the target detection results of each image in the target domain data set obtained by the teacher model according to multiple feature prototypes of various types of targets, and obtain pseudo labels for each image;

The training module 303 is used to use each image of the target domain data set as a sample, use the pseudo label of each image as a label to train the student model, and use the trained student model to detect the target in the image to be detected;

The teacher model and the student model are obtained by pre-training the object detection model using the source domain dataset.

This embodiment constructs multiple feature prototypes of each type of target by using the first instance features of each type of target extracted by the teacher model from some images of the target domain dataset, thereby providing more representative category information for the target domain. After the target detection results of each image in the target domain dataset predicted by the teacher model are corrected by the multiple feature prototypes of each type of target, more accurate pseudo-labels are obtained as supervision information for student model training, thereby improving the accuracy of target detection.

FIG4 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG4 , the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the passive domain adaptation target detection method, which includes: constructing multiple feature prototypes of each type of target based on the first instance features of each type of target extracted from part of the images of the target domain data set by the teacher model; correcting the target detection results of each image in the target domain data set obtained by the teacher model according to the multiple feature prototypes of each type of target, and obtaining pseudo labels of each image; using each image of the target domain data set as a sample, using the pseudo labels of each image as a label to train the student model, and using the trained student model to detect the target in the image to be detected; the teacher model and the student model are obtained by pre-training the target detection model using the source domain data set.

In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the passive domain adaptive target detection method provided by the above methods, which includes: constructing multiple feature prototypes of each type of target based on the first instance features of each type of target extracted by a teacher model from part of the images of a target domain data set; correcting the target detection results of each image in the target domain data set obtained by the teacher model according to the multiple feature prototypes of each type of target, and obtaining a pseudo label for each image; using each image of the target domain data set as a sample, and using the pseudo label of each image as a label to train a student model, and using the trained student model to detect targets in the image to be detected; the teacher model and the student model are obtained by pre-training the target detection model using the source domain data set.

On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the passive domain adaptive target detection method provided by the above-mentioned methods, the method comprising: constructing multiple feature prototypes of each type of target based on the first instance features of each type of target extracted by a teacher model from part of the images of a target domain data set; correcting the target detection results of each image in the target domain data set obtained by the teacher model according to the multiple feature prototypes of each type of target, and obtaining a pseudo label for each image; using each image of the target domain data set as a sample and the pseudo label of each image as a label to train a student model, and using the trained student model to detect targets in the image to be detected; the teacher model and the student model are obtained by pre-training the target detection model using the source domain data set.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (9)

1. A passive domain adaptive target detection method, comprising:

constructing a plurality of feature prototypes of various targets based on first instance features of the various targets extracted by a teacher model from partial images of a target domain data set;

Correcting target detection results of all images in the target domain data set acquired by the teacher model according to a plurality of feature prototypes of all the targets to obtain pseudo labels of all the images;

Training a student model by taking each image of the target domain data set as a sample and taking a pseudo tag of each image as a tag, and detecting a target in an image to be detected by using the trained student model;

The teacher model and the student model are obtained by training a target detection model by using a source domain data set in advance; the first example features of various targets extracted from partial images of a target domain data set based on a teacher model are used for constructing a plurality of feature prototypes of the various targets, and the method comprises the following steps:

Randomly extracting a partial image from the target domain dataset;

detecting a first class label and a first detection frame of each target in the partial image based on the teacher model, and taking an image area in the first detection frame of the target as the first example characteristic;

according to the first class labels of the targets, determining targets belonging to the same class in the partial image;

Constructing a plurality of feature prototypes of various targets according to the first example features of the various targets; the plurality of feature prototypes are used to represent features of the various types of objects.

2. The passive domain adaptive object detection method of claim 1, wherein said constructing a plurality of feature prototypes for each of said objects from first instance features of said each of said objects comprises:

Clustering the first example features of the targets for multiple times, wherein the number of clusters in each cluster is different;

and determining the contour score of each cluster, and taking the cluster center of each cluster in the clusters corresponding to the maximum contour score as the feature prototype of each type of target.

3. The passive domain adaptive target detection method according to claim 1, wherein correcting the target detection result of each image in the target domain data set obtained by the teacher model according to the feature prototypes of the various targets to obtain the pseudo tag of each image comprises:

Detecting a second class label, a second detection frame and confidence scores of the targets belonging to the second class label from each image of the target domain data set based on the teacher model;

taking the image area in the second detection frame of each target as a second example characteristic of each target;

Correcting the second class labels of the targets and confidence scores of the targets belonging to the second class labels according to the similarity between the second example features of the targets and the feature prototypes of the targets;

and determining the pseudo tags of the targets according to the second detection frames of the targets, the corrected second class tags and the corrected confidence scores.

4. The passive domain adaptive object detection method according to claim 3, wherein correcting the second class labels of the objects and the confidence scores of the objects belonging to the second class labels according to the similarity between the second instance features of the objects and the feature prototypes of the objects comprises:

Determining a first maximum value in similarity between the second example feature of each object and a plurality of feature prototypes of each object;

According to the first class labels corresponding to the maximum values in the first maximum values corresponding to all the class targets, the first class labels are used as second class labels after correction of the targets;

and taking the maximum value in the first maximum values corresponding to all the category targets as the corrected confidence score of each target.

5. A passive domain adaptive object detection method as defined in claim 3, wherein said characterizing an image area within a second detection frame of each object as a second instance of each object comprises:

comparing the confidence scores of the targets belonging to the second class labels with a preset threshold, wherein the preset threshold is determined according to the number of preset detection classes corresponding to the target detection model;

and taking the image area in the second detection frame of each target as a second example characteristic of each target under the condition that the confidence score is smaller than or equal to the preset threshold value.

6. The passive domain adaptive object detection method of claim 3, wherein training the student model using each image of the object domain dataset as a sample and a pseudo tag of each image as a tag comprises:

Detecting a third class label, a third detection frame and confidence scores of the targets belonging to the third class label from each image of the target domain dataset based on the student model;

taking the image area in the third detection frame of each target as a third example characteristic of each target;

Determining a first loss of each target according to the loss between the third class label of each target in each image of the target domain data set and the corrected second class label and the loss between the third detection frame and the second detection frame;

determining a second loss of each target according to the similarity between the second example feature and the third example feature of each target and the feature prototypes of each target;

Determining a third loss of each target according to the confidence score of each target belonging to the third category label and the corrected confidence score;

Training the student model based on the first loss, the second loss, and the third loss.

7. The passive domain adaptive object detection method of claim 6, wherein said determining the second loss of each object based on the similarity between the second instance feature, the third instance feature, and the plurality of feature prototypes of each object comprises:

Determining a first maximum value in similarity between the second example feature of each object and a plurality of feature prototypes of each object;

Determining a second maximum value in similarity between the third instance feature of each object and a plurality of feature prototypes of each object;

And determining a second loss of each target according to the first maximum value and the second maximum value.

8. The passive domain adaptive object detection method according to claim 3, wherein training the student model using each image of the object domain data set as a sample and using a pseudo tag of each image as a tag further comprises:

determining a new clustering center of each target according to the second example characteristic of each target newly generated in each preset number of iterations in the training process;

Selecting a feature prototype most similar to a new cluster center of each type of target from the feature prototypes of each type of target;

and updating the feature prototype of the selected various targets by using the new clustering centers of the various targets.

9. A passive domain adaptive target detection apparatus, comprising:

the construction module is used for constructing a plurality of feature prototypes of various targets based on first example features of the various targets extracted from partial images of the target domain data set by the teacher model;

The generating module is used for correcting target detection results of all images in the target domain data set acquired by the teacher model according to a plurality of feature prototypes of all the targets to obtain pseudo labels of all the images;

the training module is used for taking each image of the target domain data set as a sample, taking the pseudo tag of each image as a tag to train a student model, and detecting a target in an image to be detected by using the trained student model;

the teacher model and the student model are obtained by training a target detection model by using a source domain data set in advance;

The construction module is specifically used for:

Randomly extracting a partial image from the target domain dataset;

detecting a first class label and a first detection frame of each target in the partial image based on the teacher model, and taking an image area in the first detection frame of the target as the first example characteristic;

according to the first class labels of the targets, determining targets belonging to the same class in the partial image;

Constructing a plurality of feature prototypes of various targets according to the first example features of the various targets; the plurality of feature prototypes are used to represent features of the various types of objects.