CN116416562A - Domain adaptive video classification method, device, device, medium and product - Google Patents

Abstract

The application relates to a field-adaptive video classification method, device, equipment, medium and product. The method comprises the following steps: acquiring video input samples of a source domain and at least one target domain, and classifying based on the characteristics of the video input samples of the source domain to obtain an initial video classification model; constructing at least two private networks, wherein the private networks are used for respectively acquiring semantic irrelevant information characteristics of video input samples in each field; acquiring feature data of a video input sample of a source domain and at least one target domain extracted by an initial video classification model, acquiring feature distribution distances of extracted features of the video classification model and each private network, performing maximum processing on the feature distribution distances, and calculating maximum mean value difference to obtain public semantic information features; and (3) iteratively training the initial video classification model and each private network, and obtaining a target video classification model universal to the field when the iteration stop condition is met, and carrying out video classification according to the target video classification model, thereby being beneficial to improving the field adaptability and accuracy.

Description

Domain-adaptive video classification method, device, equipment, medium and product

Technical Field

The present application relates to the field of computer technology, and in particular to a domain-adaptive video classification method, device, equipment, medium and product.

Background Art

In recent years, unsupervised domain adaptation has attracted a lot of research attention. Its goal is to learn a domain-independent feature representation so that the model trained on a labeled source domain dataset can still maintain good performance in an unlabeled and differently distributed target domain.

A video classification model can be obtained by training video classification on a labeled source domain dataset. The target domain videos and the source domain videos usually have different feature distributions and are unlabeled. It is impossible to obtain good video classification results on the trained video classification model. Therefore, transfer learning of videos from different fields is required during video classification.

At present, video classification methods for different fields are usually implemented through domain adaptation methods, which learn domain-independent feature representations based on adversarial learning in a neural network. The adversarial learning method sets a domain discriminator with a gradient reversal layer in the feature extraction network. The domain discriminator is used to determine the domain source of the extracted features, and the feature extractor is used to learn how to extract more common semantic information to confuse the domain discriminator.

However, for video data that contains most of the interference information that is irrelevant to semantics, it is very difficult to learn the common semantic information between the two domains, and the domain adaptation effect of matching sample-level feature distribution using traditional adversarial learning is poor.

Summary of the invention

Based on this, it is necessary to provide a video classification method, device, computer equipment, computer-readable storage medium and computer program product that can achieve domain adaptation in response to the above technical problems.

In a first aspect, the present application provides a domain-adaptive video classification method. The method comprises:

Obtain video input samples of a source domain and at least one target domain, and classify the video input samples of the source domain based on features to obtain an initial video classification model;

Construct at least two private networks, where the private networks are used to obtain semantically irrelevant information features of video input samples in various fields respectively;

Acquire feature data of video input samples of a source domain and at least one target domain extracted by an initial video classification model, obtain feature distribution distances of features extracted by the video classification model and each private network, maximize the feature distribution distances and calculate the maximum mean difference to obtain common semantic information features;

The initial video classification model and each private network are iteratively trained. When the iteration stopping condition is met, a domain-wide target video classification model is obtained, and video classification is performed according to the target video classification model.

In one embodiment, at least two private networks are constructed, including:

Obtain background data of video input samples, and use the background data as a supervisory signal for reconstruction training of the private network;

Reconstruction training of video input samples in various fields is performed through a private network to obtain reconstructed background data;

Obtaining the reconstruction loss between the background data and the reconstructed background data;

Minimize the reconstruction loss and obtain semantically irrelevant information features.

In one embodiment, the private network includes a video feature extractor and a reconstruction network, and the private network is used to perform reconstruction training of video input samples in various fields to obtain reconstructed background data, including:

Based on the video feature extractor, background features of video input samples in various fields are obtained;

Reconstructing the background features based on the reconstruction network to obtain reconstructed background data;

Obtain the reconstruction loss between the background data and the reconstructed background data, including:

Obtaining the distance between background data and reconstructed background data;

The reconstruction loss is calculated using a loss function based on a distance metric and the distance.

In one embodiment, the initial video classification model includes a feature extractor, a domain discriminator and a classifier, and the initial video classification model is obtained by classifying based on the features of the source domain video input sample, including:

Obtain features of source domain video input samples through a feature extractor;

Classify features through classifiers;

Obtain classification loss, which is used to iteratively train the initial video classification model and each private network;

Acquiring feature data of video input samples of a source domain and at least one target domain extracted by an initial video classification model, including:

Obtaining initial feature data of video input samples of a source domain and at least one target domain through a feature extractor;

Perform adversarial training on the initial feature data according to the domain discriminator to obtain the target feature data after adversarial training;

Video classification of target feature data obtained by the classifier;

Obtain the adversarial training loss of the domain discriminator, which is used to iteratively train the initial video classification model and each private network.

In one embodiment, the method further comprises:

Constructing a feature source classifier, and determining a source identifier of an input feature according to the feature source classifier, wherein the source identifier is used to determine whether the source of the input feature is an initial video classification model or a private network;

The source classification loss of the feature source classifier is obtained, and the source classification loss is used to iteratively train the initial video classification model and each private network.

In one embodiment, the initial video classification model and each private network are iteratively trained to obtain a domain-general target video classification model when an iteration stop condition is met, including:

Obtain training loss based on the loss function, and obtain the iteration stop condition based on the training loss;

Calculate the gradient of the loss function based on the training loss backpropagation and update the loss function;

When the training loss is stable and the iteration stopping condition is met, a domain-wide target video classification model is obtained.

In a second aspect, the present application also provides a video classification device capable of achieving domain adaptation. The device comprises:

A video classification module, used to obtain video input samples of a source domain and at least one target domain, and classify the video input samples of the source domain based on their features to obtain an initial video classification model;

A private network module, used to construct at least two private networks, where the private networks are used to obtain semantically irrelevant information features of video input samples in various fields respectively;

A mean difference module is used to obtain feature data of video input samples of a source domain and at least one target domain extracted by an initial video classification model, obtain feature distribution distances of features extracted by the video classification model and each private network, maximize the feature distribution distances and calculate the maximum mean difference to obtain common semantic information features;

The iterative training module is used to iteratively train the initial video classification model and each private network. When the iteration stop condition is met, a domain-wide target video classification model is obtained, and video classification is performed according to the target video classification model.

In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Obtain video input samples of a source domain and at least one target domain, and classify the video input samples of the source domain based on features to obtain an initial video classification model;

Construct at least two private networks, where the private networks are used to obtain semantically irrelevant information features of video input samples in various fields respectively;

Acquire feature data of video input samples of a source domain and at least one target domain extracted by an initial video classification model, obtain feature distribution distances of features extracted by the video classification model and each private network, maximize the feature distribution distances and calculate the maximum mean difference to obtain common semantic information features;

The initial video classification model and each private network are iteratively trained. When the iteration stopping condition is met, a domain-wide target video classification model is obtained, and video classification is performed according to the target video classification model.

In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

Obtain video input samples of a source domain and at least one target domain, and classify the video input samples of the source domain based on features to obtain an initial video classification model;

Construct at least two private networks, where the private networks are used to obtain semantically irrelevant information features of video input samples in various fields respectively;

Acquire feature data of video input samples of a source domain and at least one target domain extracted by an initial video classification model, obtain feature distribution distances of features extracted by the video classification model and each private network, maximize the feature distribution distances and calculate the maximum mean difference to obtain common semantic information features;

The initial video classification model and each private network are iteratively trained. When the iteration stopping condition is met, a domain-wide target video classification model is obtained, and video classification is performed according to the target video classification model.

In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

Obtain video input samples of a source domain and at least one target domain, and classify the video input samples of the source domain based on features to obtain an initial video classification model;

Construct at least two private networks, where the private networks are used to obtain semantically irrelevant information features of video input samples in various fields respectively;

Acquire feature data of video input samples of a source domain and at least one target domain extracted by an initial video classification model, obtain feature distribution distances of features extracted by the video classification model and each private network, maximize the feature distribution distances and calculate the maximum mean difference to obtain common semantic information features;

The initial video classification model and each private network are iteratively trained. When the iteration stopping condition is met, a domain-wide target video classification model is obtained, and video classification is performed according to the target video classification model.

The above-mentioned domain-adaptive video classification method, apparatus, computer equipment, storage medium and computer program product obtain video input samples from a source domain and at least one target domain, classify them based on the features of the source domain video input samples to obtain an initial video classification model, construct at least two private networks, the private networks are used to respectively obtain semantically irrelevant information features of video input samples from each domain, obtain feature data of video input samples from the source domain and at least one target domain extracted by the initial video classification model, obtain feature distribution distances between the video classification model and the features extracted by each private network, maximize the feature distribution distances and calculate the maximum mean difference to obtain common semantic information features, iteratively train the initial video classification model and each private network, and when the iteration stop condition is met, obtain a domain-universal target video classification model, and implement domain adaptation based on the target video classification model. Video classification, this method constructs a private network and extracts semantically irrelevant information features, and obtains the maximum mean difference between the semantically irrelevant information features and the feature data extracted by the initial video classification model. After maximizing the maximum mean difference, that is, after achieving the maximum feature difference between the semantically irrelevant information features and the feature data, it is beneficial for the video classification model to obtain public semantic information features and ignore semantically irrelevant information features during the video classification process, which is beneficial to reducing the impact of semantically irrelevant information features in the target domain video that causes the target domain video to be unable to obtain adaptive video classification in the initial video classification model, and improves the domain adaptability of video classification in domain migration. For the classification of video data containing most of the interference information irrelevant to semantics, the domain adaptation effect is good, the domain adaptive video classification accuracy is high, and it has high reliability.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a diagram of an application environment of a video classification method according to an embodiment;

FIG2 is a schematic diagram of a flow chart of a video classification method in one embodiment;

FIG3 is a schematic diagram of the maximum mean difference of a video classification method according to an embodiment;

FIG4 is a schematic diagram of the structure of a private network in one embodiment;

FIG5 is a schematic diagram of the structure of an initial video classification model in one embodiment;

FIG6 is a schematic flow chart of a video classification method in another embodiment;

FIG7 is a structural block diagram of a video classification device in one embodiment;

FIG8 is a structural block diagram of a video classification device in another embodiment;

FIG9 is an internal structure diagram of a computer device as a server in one embodiment;

FIG. 10 is a diagram showing the internal structure of a computer device that is a terminal in one embodiment.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

The domain-adaptive video classification method provided in the embodiment of the present application can be applied in the application environment shown in FIG. 1 . Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 obtains video input samples of the source domain and at least one target domain, classifies based on the features of the source domain video input samples to obtain an initial video classification model, constructs at least two private networks, and the private networks are used to obtain semantically irrelevant information features of video input samples in each domain, obtain feature data of video input samples of the source domain and at least one target domain extracted by the initial video classification model, obtain the feature distribution distance of the video classification model and the features extracted by each private network, maximize the feature distribution distance and calculate the maximum mean difference, obtain the common semantic information features, iteratively train the initial video classification model and each private network, and when the iteration stop condition is met, obtain the domain-general target video classification model, and perform video classification according to the target video classification model. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

In one embodiment, as shown in FIG. 2 , a domain-adaptive video classification method is provided, and the method is applied to the server 104 in FIG. 1 as an example for description, including the following steps:

Step 202: Obtain video input samples of a source domain and at least one target domain, and classify the video input samples of the source domain based on features to obtain an initial video classification model.

Among them, the source domain represents a domain different from the test sample and has rich supervised annotation information; the target domain represents the domain where the test sample is located, which has no labels or only a small number of labels. For example, in transfer learning, the original samples from the source domain contain video category labels, and the original samples from the target domain do not contain video category labels and have a different distribution from the source domain samples.

Exemplarily, the input source domain and target domain video data are processed by image frame extraction and downsampling. The downsampling method is to sample 16 frames starting from a random position at a fixed sampling frequency to obtain source domain and target domain video input samples of the same size. The source domain video input samples are iteratively trained for 3000 video classification tasks to obtain an initial video classification model.

Step 204: construct at least two private networks, where the private networks are used to obtain semantically irrelevant information features of video input samples in various fields.

Among them, semantically irrelevant information features refer to background information in video classification that is closely related to the domain but irrelevant to semantics, and manifests as interference features for video classification.

For example, for dynamic videos, static background features are the most important semantically irrelevant information features. After building a private network, different private networks can be trained through background reconstruction to obtain the background features of the source domain and target domain video input samples respectively.

Step 206, obtain feature data of video input samples of the source domain and at least one target domain extracted by the initial video classification model, obtain feature distribution distances of the features extracted by the video classification model and each private network, maximize the feature distribution distance and calculate the maximum mean difference to obtain common semantic information features.

Among them, the maximum mean difference (MMD) is used to measure the distance between different feature distributions. If the mean difference of the distance reaches the maximum, it means that the samples are from completely different distributions. Common semantic information features refer to features that are not related to the domain but are related to semantics, which are manifested as features related to the classification of video categories.

Exemplarily, the initial video model extracts feature data from the video input samples of the source domain and the target domain. The feature data and the background features of the source domain video input samples extracted by the private network and the background features of the target domain video input samples have different feature distributions. The feature distribution distance of the three input samples is maximized to obtain the maximum mean difference MMD, and the calculation formula is expressed as:

Among them, X _S , _XT are the source domain and target domain video input samples respectively, and φ(x _s ) and φ(x _t ) are the corresponding kernel functions.

For example, as shown in FIG3 , the maximum mean difference diagram of the video classification method, when the feature distribution distance is maximized, the difference between the features is the largest, which can achieve the effect of ignoring the background features of the source domain and the target domain in the feature data obtained by the initial video classification model, and obtain the common semantic information features of the source domain and the target domain. In the process of maximizing the feature distribution distance, the loss function calculation formula of the maximum mean difference of the initial video classification model is expressed as:

为最大均值差异损失，在源域的背景特征分布为d_Sp、目标域的背景特征分布为d_Tp和初始视频分类模型的特征数据分布为d_main的情况下，d_Sp与d_Tp之间的最大均值差异为MMD(d_Sp,d_Tp)，d_main与d_Sp之间的最大均值差异为MMD(d_main,d_Sp)，d_main与d_Tp之间的最大均值差异为MMD(d_main,d_Tp)。in,

is the maximum mean difference loss. When the background feature distribution of the source domain is d _Sp , the background feature distribution of the target domain is d _Tp and the feature data distribution of the initial video classification model is d _main , the maximum mean difference between d _Sp and d _Tp is MMD(d _Sp ,d _Tp ), the maximum mean difference between d _main and d _Sp is MMD(d _main ,d _Sp ), and the maximum mean difference between d _main and d _Tp is MMD(d _main ,d _Tp ).

Step 208, iteratively train the initial video classification model and each private network, and when the iteration stop condition is met, obtain a domain-wide target video classification model, and perform video classification according to the target video classification model.

Among them, iterative training is the process of training the video classification model according to the iterative algorithm. Iteration is the activity of repeating the feedback process. The iterative algorithm starts from a certain value and continuously calculates the result of the next step based on the result of the previous step.

Exemplarily, iterative training can be performed based on the loss function in the video classification model training process. When the loss function no longer changes, specifically, when the loss function no longer decreases or the loss function tends to be stable, the video classification model converges to obtain a target video classification model that can accurately classify target domain videos. Video classification can then be performed based on the target video classification model.

The above-mentioned video classification method capable of achieving domain adaptation obtains video input samples from a source domain and at least one target domain, classifies the video input samples based on the features of the source domain to obtain an initial video classification model, constructs at least two private networks, and the private networks are used to respectively obtain semantically irrelevant information features of video input samples from each domain, obtains feature data of video input samples from the source domain and at least one target domain extracted by the initial video classification model, obtains feature distribution distances between the video classification model and features extracted by each private network, maximizes the feature distribution distance and calculates the maximum mean difference to obtain common semantic information features, iteratively trains the initial video classification model and each private network, and when the iteration stop condition is met, obtains a domain-universal target video classification model, and implements domain-adaptive video classification according to the target video classification model. The method is By building a private network and extracting semantically irrelevant information features, and obtaining the maximum mean difference between the semantically irrelevant information features and the feature data extracted by the initial video classification model, the maximum mean difference is maximized, that is, after achieving the maximum feature difference between the semantically irrelevant information features and the feature data, it is beneficial for the video classification model to obtain public semantic information features and ignore semantically irrelevant information features during the video classification process, which is beneficial to reducing the influence of semantically irrelevant information features in the target domain video that causes the target domain video to be unable to obtain adaptive video classification in the initial video classification model, and improves the domain adaptability of video classification in domain migration. For the classification of video data containing most of the interference information irrelevant to semantics, the domain adaptation effect is good, the domain adaptive video classification accuracy is high, and it has high reliability.

In one embodiment, at least two private networks are constructed, including: obtaining background data of video input samples, using the background data as a supervisory signal for reconstruction training of the private network; performing reconstruction training of video input samples in various fields through the private network to obtain reconstructed background data; obtaining a reconstruction loss between the background data and the reconstructed background data; and minimizing the reconstruction loss to obtain semantically irrelevant information features.

Among them, the supervisory signal refers to the expected output value of the video input sample in supervised learning. Reconstruction is image reconstruction (IR), which aims to reconstruct the image based on various information extracted from the ground truth image. Groundtruth means that in supervised learning, the data is labeled and appears in the form of (x, t), where x is the input data, t is the label, and the correct t label is the ground truth. Reconstruction loss refers to the reconstruction loss function, that is, the operation function of the degree of difference between the predicted value and the true value of the private network image reconstruction. The loss function includes the loss function based on the distance metric and the loss function based on the probability distribution metric.

Exemplarily, as shown in the structural diagram of the private network in FIG4 , the background data of the source domain and target domain video input samples can be extracted by the temporal median filter TMF to obtain the source domain and target domain background data, the source domain background data and the source domain video input samples are input into the source domain private network, and the target domain background data and the target domain video input samples are input into the target domain private network. The background data is used as a supervisory signal and is the expected output value of the private network during the reconstruction training process, and is also the correct t-labeled ground truth. The output value of the private network after image reconstruction is compared with the background data to calculate the reconstruction loss function. In the process of reconstruction training, the reconstruction loss is minimized, and the difference between the background data and the reconstructed background data can be reduced, so that the private network can learn semantically irrelevant information, that is, semantically irrelevant information features.

In this embodiment, a temporal median filter is used to obtain background data of a video input sample. The temporal median filter is a simple, intuitive and fast method for extracting video background. The loss function is used in the training phase of the model. After obtaining the loss value between the predicted value and the difference value obtained by a single training, the parameters of the private network are updated in the direction of minimizing the loss value, thereby reducing the loss between the true value and the predicted value, and making the predicted value generated by the model move closer to the true value, thereby achieving the effect of the private network learning the semantically irrelevant information features of the source domain and target domain video input samples.

In one embodiment, a private network includes a video feature extractor and a reconstruction network. Reconstruction training of video input samples in various fields is performed through the private network to obtain reconstructed background data, including: obtaining background features of video input samples in various fields based on the video feature extractor; reconstructing the background features based on the reconstruction network to obtain reconstructed background data; obtaining a reconstruction loss between the background data and the reconstructed background data, including: obtaining the distance between the background data and the reconstructed background data; and calculating the reconstruction loss through a loss function based on a distance metric and the distance.

The video feature extractor is used to extract the features of the video input samples, and the reconstruction network is used to reconstruct the background based on the extracted features.

Exemplarily, as shown in the structural diagram of the private network in FIG4 , the source domain private network includes a source domain video feature extractor F _Sp and a source domain reconstruction network, and the target domain private network includes a target domain video feature extractor F _Tp and a target domain reconstruction network. The background features of the video input sample extracted by the video feature extractor are obtained, and the image can be reconstructed in the reconstruction network according to the background features to obtain the predicted reconstructed background data. The L ₂ loss function of the source domain and the target domain is calculated according to the distance between the reconstructed background data of the source domain and the target domain and the background data. The L ₂ loss function is also called the Euclidean distance, which is a commonly used distance measurement function, and is usually used to measure the similarity between data points. The calculation formula of the source domain reconstruction loss is expressed as:

The calculation formula of the target domain reconstruction loss is expressed as:

The calculation formula of the private network reconstruction loss is the sum of two reconstruction losses:

为源域的L₂损失函数，b_Tp为目标域重构背景数据，b_T为目标域背景数据，

为目标域的L₂损失函数，

为私有网络重构损失函数。Among them, _bSp is the source domain reconstruction background data, _bS is the source domain background data,

is the _L2 loss function of the source domain, _bTp is the reconstructed background data of the target domain, _bT is the background data of the target domain,

is the _L2 loss function of the target domain,

Reconstruct the loss function for private networks.

In this embodiment, the distance between the true value of the video input sample and the predicted value of the private network in the feature space is measured by a loss function based on distance measurement. The smaller the distance between two points in the feature space, the better the prediction performance of the private network can be obtained, and the curve of the _L2 loss function is sufficiently flat when approaching the target, so this feature can be used to gradually and slowly converge when approaching the target, which is suitable for image processing.

In one of the embodiments, the initial video classification model includes a feature extractor, a domain discriminator and a classifier, and the initial video classification model is obtained by classifying based on the features of the source domain video input samples, including: obtaining the features of the source domain video input samples through the feature extractor; classifying the features through the classifier; obtaining the classification loss, and the classification loss is used to iteratively train the initial video classification model and each private network; obtaining the feature data of the video input samples of the source domain and at least one target domain extracted by the initial video classification model, including: obtaining the initial feature data of the video input samples of the source domain and at least one target domain through the feature extractor; performing adversarial training on the initial feature data according to the domain discriminator to obtain target feature data after adversarial training; obtaining the video classification of the target feature data according to the classifier; obtaining the adversarial training loss of the domain discriminator, and the adversarial training loss is used to iteratively train the initial video classification model and each private network.

Among them, adversarial training means that in the training process of the initial video classification model, the inputs of the domain discriminator and the image classifier are all from the features extracted by the feature extractor. The domain discriminator is used to maximize the domain discrimination loss and confuse the target domain video input data with the source domain video input data. The image classifier is used to minimize the image classification loss to achieve accurate image classification. The domain discriminator includes a gradient reversal layer and two fully connected layers, which are used to discriminate whether the features extracted by the feature extractor are from the source domain or the target domain. The gradient of the domain discrimination loss function is opposite to the gradient of the image classification loss function. The gradient reversal layer can realize the automatic inversion of the gradient of the domain discrimination loss before backpropagating to the parameters of the feature extractor, thereby realizing adversarial training. The classifier refers to an image classifier, which is used to classify the video based on the features extracted from the video input samples.

Exemplarily, the structural diagram of the initial video classification model shown in FIG5 obtains the features of the source domain video input sample through the feature extractor, and the source domain video input sample is an input video sample containing a video classification label. The classifier performs 3000 video classification task trainings based on the extracted features to obtain the initial video classification model. The initial video classification model can achieve accurate video classification for source domain videos with video classification labels. The difference between the output value and the true value of the video input sample classification during the video classification task is obtained to obtain the video classification loss. The calculation formula for calculating the video classification loss function is expressed as:

为视频分类损失，x是输入样本，x∈X_S是输入样本为源域视频输入样本，y为源域的视频类别标签也是真实值，σ为softmax函数，C(F(x))为分类器对源域视频输入样本进行分类得到的输出经过softmax函数计算得到概率值。in,

is the video classification loss, x is the input sample, x∈X _S is the input sample is the source domain video input sample, y is the video category label of the source domain and also the true value, σ is the softmax function, C(F(x)) is the output obtained by the classifier to classify the source domain video input sample and the probability value is calculated by the softmax function.

The feature extractor extracts the initial feature data of the source domain and target domain video input samples. There is a gradient reversal layer of the domain discriminator between the feature extractor and the classifier. Maximizing the loss value of domain discrimination and minimizing the loss value of the video classification task can realize adversarial training of the initial feature data, and obtain the target feature data of the source domain and target domain domain discrimination confusion. The source domain and target domain input videos can be classified according to the target feature data, and the loss value in adversarial training can be obtained. The calculation formula for calculating the adversarial training loss function is expressed as:

为对抗训练损失，y_d是一个代表领域标签的二维向量，即输入视频样本所在领域的真实值，当输入x，x∈X_S是源域原始样本时，y_d＝<1,0>，或当输入x是目标域原始样本x∈X_T时，y_d＝<0,1>，σ为softmax函数，为(D(F(X)))域判别器对源域和目标域视频输入样本进行领域判别得到的输出经过softmax函数计算得到概率值。in,

is the adversarial training loss, _yd is a two-dimensional vector representing the domain label, that is, the true value of the domain where the input video sample is located. When the input x, _x∈XS is the original sample of the source domain, _yd ＝<1,0>, or when the input x is the original sample of the target domain _x∈XT , _yd ＝<0,1>, σ is the softmax function, and the output obtained by the (D(F(X))) domain discriminator for the source domain and target domain video input samples is calculated by the softmax function to obtain the probability value.

In this embodiment, video classification task training and adversarial training of the initial video classification model are implemented.

In one embodiment, the method further includes: constructing a feature source classifier, determining a source identifier of an input feature according to the feature source classifier, wherein the source identifier is used to determine whether the source of the input feature is an initial video classification model or a private network; obtaining a source classification loss of the feature source classifier, and the source classification loss is used to iteratively train the initial video classification model and each private network.

Exemplarily, features extracted by the feature extractor of the initial video classification model and the video feature extractor of the private network are obtained, and the features carry a source identifier, and the source identifier is used to determine whether the source of the input feature is the feature extractor F of the initial video classification model or the source domain video feature extractor F _Sp of the source domain private network or the target domain video feature extractor F _Tp of the target domain private network, and the extracted features are input to the feature source classifier, and the feature source classifier obtains that the features come from the initial video classification model or the private network according to the input extracted features, and calculates the source classification loss according to the output value and the true value. The calculation formula of the source classification loss function is expressed as:

为来源分类损失，y_N为特征提取器F、源域视频特征提取器F_Sp和目标域视频特征提取器F_Tp的来源标识，f为任一输入的提取特征，C_N(f)为特征来源分类器判断输入特征得到的输出经过softmax函数计算得到的概率值。in,

is the source classification loss, y _N is the source identifier of the feature extractor F, the source domain video feature extractor F _Sp and the target domain video feature extractor F _Tp , f is the extracted feature of any input, and C _N (f) is the probability value calculated by the softmax function after the output obtained by the feature source classifier judging the input feature.

In this embodiment, by adding a feature source classifier to distinguish the features extracted by the video classification model and the private network, the difference in the feature content obtained through training is enhanced.

In one embodiment, the method further includes: iteratively training the initial video classification model and each private network, and obtaining a domain-wide target video classification model when an iteration stop condition is met, including: obtaining a training loss based on a loss function, and obtaining an iteration stop condition according to the training loss; calculating the gradient of the loss function by back-propagation according to the training loss, and updating the loss function; when the training loss is stable, the iteration stop condition is met, and a domain-wide target video classification model is obtained.

Back propagation is short for "error back propagation", which is a common method used in combination with optimization methods to train artificial neural networks. This method calculates the gradient of the loss function for all weights in the network, and this gradient is fed back to the optimization method to update the weights to minimize the loss function.

对抗训练损失

重构损失

最大均值差异损失

来源分类损失

根据总损失函数对初始视频分类模型进行迭代训练，总损失函数计算公式为：For example, the video classification loss is obtained

Adversarial Training Loss

Reconstruction loss

Maximum mean difference loss

Source classification loss

The initial video classification model is iteratively trained according to the total loss function, and the total loss function calculation formula is:

The direction of the maximum value of the directional derivative on the surface of the function image represents the direction of the gradient. When doing gradient descent, it should be updated in the opposite direction of the gradient, and the gradient of the loss function is calculated by back propagation. Combined with the stochastic gradient descent (SGD) optimization algorithm, the initial training model can be iteratively trained to minimize the loss function. In other words, according to the total loss function obtained, the statement is calculated. After the gradient is calculated by back propagation, the total loss function approaches the direction of minimizing the loss function, and the gradient will be fed back to the stochastic gradient descent optimization algorithm. The optimization algorithm can update the model parameters of the initial video classification model according to the gradient, and iteratively train the initial video classification model. When the minimum value obtained by the total loss function is stable, the model gradually converges to obtain the target video classification model. The target video classification model shows the minimum total loss, that is, the accuracy of video classification is high, and the video classification of the target domain can also be accurately classified in the video classification model.

FIG6 is a flow chart of a domain-adaptive video classification method according to another embodiment. The domain-adaptive video classification method comprises the following steps:

Step 602: obtain original source domain and target domain video data, perform video frame extraction and downsampling processing on the video data, and obtain source domain and target domain video input samples.

For the original video data, image frame extraction is performed to obtain an RGB video frame sequence, and video frame sampling is performed. According to the original source domain and target domain videos for training the initial video classification model, the collected RGB frame sequence is obtained, and one frame is sampled every 4 frames starting from a random position as input data. Each sample samples t frames as the video input sample for training the initial video classification model.

Step 604, construct an initial video classification model, including a feature extractor, a domain discriminator and a classifier, obtain source domain features of the source domain video input sample according to the feature extractor, classify the video according to the classifier and the obtained source domain features, and obtain the video classification loss function

The initial video classification model is pre-trained using source domain video input samples with video classification labels. The initial video classification model uses the I3D video classification model. First, only the source domain data is used for pre-training for 3000 iterations. The pre-training uses the SGD (standard gradient descent) optimization algorithm with a learning rate of 0.001.

Step 606, constructing source domain and target domain private networks, the private networks including a video feature extractor and a reconstruction network, the private networks are used to obtain semantically irrelevant information features of video input samples in each domain.

Step 608: The background image of the source domain and the target domain video input samples is obtained through a temporal median filter, and the background image is used as a supervisory signal for reconstruction training of the private network.

A fixed-parameter temporal median filter is used to extract the background image of the video input sample. The dimension of the input sample RGB frame sequence is time × height × width × number of channels (t × h × w × c), and the dimension of the extracted background image is height × width × number of channels (h × w × c), without a time dimension.

Step 610, obtaining background features of source domain and target domain video input samples through the video feature extractors of each private network, and obtaining reconstructed background images of source domain and target domain video input samples through the reconstruction networks of each private network.

Step 612, respectively obtain the _L2 losses of the source domain and the target domain between the background images of the source domain and the target domain and the reconstructed background images of the source domain and the target domain, obtain the reconstruction losses of the source domain and the target domain, obtain the _L2 loss function of the reconstruction loss, minimize the _L2 loss function, and obtain the semantically irrelevant information features of the source domain and the target domain video input samples.

将目标域视频输入样本输入目标域私有网络，计算

求两个私有网络重构训练损失之和

The source domain video input samples are input into the source domain private network, and background reconstruction training is performed to learn the semantically irrelevant information features of the source domain.

Input the target domain video input sample into the target domain private network and calculate

Find the sum of the reconstruction training losses of the two private networks

Step 614, obtain the initial features of the source domain and target domain video input samples according to the feature extractor, perform adversarial training on the initial features according to the domain discriminator to obtain the target features, obtain the classification of the target features according to the classifier, and obtain the adversarial training loss function

The feature dimension of the initial video classification model is 1024. The domain discriminator consists of a gradient reversal layer and a two-layer fully connected layer classifier. The input is a 1024-dimensional feature vector, the hidden layer dimension is 100, and the output is a 2-dimensional vector. Using labeled source domain samples and unlabeled target domain samples as input data, the main network feature extractor is trained to extract domain-independent features and calculate

Step 616, maximize the feature distribution distance between the reconstructed background image and the target feature, obtain the common semantic information feature, and obtain the loss function that maximizes the feature difference value.

MMD中核函数采用多个高斯核。Maximize the MMD distance of the three feature distributions and calculate

The kernel function in MMD uses multiple Gaussian kernels.

Step 618, construct a feature source classifier, input the target feature and the background feature obtained by the private network video feature extractor into the feature source classifier, determine the input feature source initial video classification model or private network according to the feature source classifier, and obtain the source classification loss function.

Step 620, iteratively train the initial video classification model and each private network, and when the iteration stop condition is met, obtain a domain-wide target video classification model, and perform video classification according to the target video classification model.

表达式，反向传播计算总损失函数的梯度，采用SGD(标准梯度下降)优化算法，更新初始视频分类模型的参数，学习率设置为0.0001，重复重构训练、对抗训练的过程，迭代训练16000次，在总损失函数稳定不变的情况下，得到目标视频分类模型，根据目标视频分类模型可以实现源域和目标域的视频分类。The total loss function is obtained by adding all the loss function expressions obtained.

Expression, back-propagation calculates the gradient of the total loss function, uses the SGD (standard gradient descent) optimization algorithm, updates the parameters of the initial video classification model, sets the learning rate to 0.0001, repeats the reconstruction training and adversarial training processes, and iterates the training for 16,000 times. When the total loss function remains stable, the target video classification model is obtained. According to the target video classification model, video classification in the source domain and the target domain can be realized.

Step 622: Acquire test samples to perform test training on the target video classification model.

The original video of the test target video classification model is obtained, and samples are taken from 5 random positions in the RGB frame sequence as input data. The prediction results of the 5 samples are averaged as the final prediction result. The dimension of each sample RGB frame sequence is time × height × width × number of channels (t × h × w × c). In this embodiment, t is 16, and h and w are 224.

It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the present application also provides a domain-adaptive video classification device for implementing the domain-adaptive video classification method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the domain-adaptive video classification device embodiment provided below can refer to the limitations of the domain-adaptive video classification method above, and will not be repeated here.

In one embodiment, as shown in FIG. 7 , a domain-adaptive video classification device 700 is provided, including: a video classification module 702, a private network module 704, a mean difference module 706, and an iterative training module 708, wherein:

A video classification module 702 is used to obtain video input samples of a source domain and at least one target domain, and classify the video input samples of the source domain based on features to obtain an initial video classification model;

A private network module 704 is used to construct at least two private networks, and the private networks are used to obtain semantically irrelevant information features of video input samples in various fields respectively;

The mean difference module 706 is used to obtain feature data of the video input samples of the source domain and at least one target domain extracted by the initial video classification model, obtain the feature distribution distance of the video classification model and the features extracted by each private network, maximize the feature distribution distance and calculate the maximum mean difference to obtain the common semantic information feature;

The iterative training module 708 is used to iteratively train the initial video classification model and each private network. When the iteration stop condition is met, a domain-wide target video classification model is obtained, and video classification is performed according to the target video classification model.

In one embodiment, the private network module 704 is also used to construct at least two private networks, including: obtaining background data of video input samples, using the background data as a supervisory signal for reconstruction training of the private network; performing reconstruction training of video input samples in various fields through the private network to obtain reconstructed background data; obtaining the reconstruction loss between the background data and the reconstructed background data; minimizing the reconstruction loss to obtain semantically irrelevant information features.

In one embodiment, the private network module 704 is also used for a private network including a video feature extractor and a reconstruction network. Reconstruction training of video input samples in various fields is performed through the private network to obtain reconstructed background data, including: obtaining background features of video input samples in various fields based on the video feature extractor; reconstructing the background features based on the reconstruction network to obtain reconstructed background data; obtaining the reconstruction loss between the background data and the reconstructed background data, including: obtaining the distance between the background data and the reconstructed background data; calculating the reconstruction loss through a loss function based on a distance metric and the distance.

In one embodiment, the mean difference module 706 is also used for the initial video classification model including a feature extractor, a domain discriminator and a classifier, and the initial video classification model is obtained by classification based on the features of the source domain video input samples, including: obtaining the features of the source domain video input samples through the feature extractor; classifying the features through the classifier; obtaining the classification loss, and the classification loss is used for iterative training of the initial video classification model and each private network; obtaining the feature data of the video input samples of the source domain and at least one target domain extracted by the initial video classification model, including: obtaining the initial feature data of the video input samples of the source domain and at least one target domain through the feature extractor; performing adversarial training on the initial feature data according to the domain discriminator to obtain the target feature data after adversarial training; obtaining the video classification of the target feature data according to the classifier; obtaining the adversarial training loss of the domain discriminator, and the adversarial training loss is used for iterative training of the initial video classification model and each private network.

In one embodiment, as shown in FIG8 , the device further includes a source classification module 810 for constructing a feature source classifier, determining a source identifier of an input feature according to the feature source classifier, wherein the source identifier is used to determine whether the source of the input feature is an initial video classification model or a private network; and obtaining a source classification loss of the feature source classifier, wherein the source classification loss is used to iteratively train the initial video classification model and each private network.

In one embodiment, the iterative training module 708 is also used to iteratively train the initial video classification model and each private network, and when the iteration stop condition is met, a domain-wide target video classification model is obtained, including: obtaining the training loss based on the loss function, and obtaining the iteration stop condition according to the training loss; calculating the gradient of the loss function by backpropagation according to the training loss, and updating the loss function; when the training loss is stable, the iteration stop condition is met, and the domain-wide target video classification model is obtained.

Each module in the above-mentioned domain-adaptive video classification device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG9. The computer device includes a processor, a memory, an input/output interface (Input/Output, referred to as I/O) and a communication interface. The processor, the memory and the input/output interface are connected via a system bus, and the communication interface is connected to the system bus via the input/output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video classification data. The input/output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a field-adaptive video classification method is implemented.

In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input/output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input/output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input/output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a field-adaptive video classification method is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

Those skilled in the art will appreciate that the aforementioned structure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned method embodiments when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method embodiments are implemented.

In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above-mentioned method embodiments when executed by a processor.

It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the embodiments of the above-mentioned methods when executed. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims (10)

1. A method for domain-adaptive video classification, the method comprising:

acquiring video input samples of a source domain and at least one target domain, and classifying based on the characteristics of the video input samples of the source domain to obtain an initial video classification model;

constructing at least two private networks, and acquiring video input samples of the source domain and at least one target domain, wherein the private networks are used for respectively acquiring semantic independent information characteristics of the video input samples of each domain;

Acquiring feature data of video input samples of the source domain and at least one target domain extracted by the initial video classification model, acquiring feature distribution distances of extracted features of the initial video classification model and each private network, performing maximum processing on the feature distribution distances, and calculating maximum mean value difference to obtain public semantic information features;

and iteratively training the initial video classification model and each private network, and obtaining a target video classification model universal to the field when the iteration stop condition is met, and carrying out video classification according to the target video classification model.

2. The method of claim 1, wherein said constructing at least two private networks comprises:

obtaining background data of the video input sample, wherein the background data is used as a supervision signal for reconstruction training of the private network;

performing reconstruction training of the video input samples in each field through the private network to obtain reconstruction background data;

obtaining reconstruction loss between the background data and the reconstructed background data;

and minimizing the reconstruction loss to obtain the semantic independent information features.

3. The method of claim 2, wherein the private network includes a video feature extractor and a reconstruction network, wherein the reconstructing training of each of the domain video input samples through the private network results in reconstructed background data, comprising:

Obtaining background characteristics of each field video input sample based on the video characteristic extractor;

reconstructing the background features based on the reconstruction network to obtain the reconstructed background data;

the obtaining the reconstruction loss between the background data and the reconstructed background data comprises:

acquiring the distance between the background data and the reconstructed background data;

the reconstruction loss is calculated by a loss function based on a distance measure and the distance.

4. The method of claim 1, wherein the initial video classification model comprises a feature extractor, a domain arbiter, and a classifier, wherein the classifying based on the features of the source domain video input samples results in an initial video classification model comprising;

acquiring the characteristics of the source domain video input sample through the characteristic extractor;

classifying the features by the classifier;

obtaining classification loss, wherein the classification loss is used for iteratively training the initial video classification model and each private network;

the obtaining the feature data of the video input samples of the source domain and at least one target domain extracted by the initial video classification model includes:

Obtaining initial feature data of a video input sample of the source domain and at least one target domain by the feature extractor;

performing countermeasure training on the initial feature data according to the domain discriminator to obtain target feature data after the countermeasure training;

obtaining video classification of the target feature data according to the classifier;

and obtaining an countermeasure training penalty of the domain discriminator, wherein the countermeasure training penalty is used for iteratively training the initial video classification model and each private network.

5. The method according to claim 1, wherein the method further comprises:

constructing a feature source classifier, and determining a source identifier of an input feature according to the feature source classifier, wherein the source identifier is used for determining that the source of the input feature is an initial video classification model or a private network;

and obtaining source classification loss of the characteristic source classifier, wherein the source classification loss is used for iteratively training the initial video classification model and each private network.

6. The method of claim 1, wherein the iteratively training the initial video classification model and each of the private networks, when an iteration stop condition is satisfied, results in a domain-generic target video classification model, comprising:

Acquiring training loss based on a loss function, and acquiring the iteration stop condition according to the training loss;

calculating the gradient of the loss function according to the training loss back propagation, and updating the loss function;

and under the condition that the training loss is stable, the iteration stop condition is met, and the universal target video classification model in the field is obtained.

7. A domain adaptive device, the device comprising:

the video classification module is used for acquiring video input samples of a source domain and at least one target domain, and classifying the video input samples based on the characteristics of the video input samples of the source domain to obtain an initial video classification model;

the private network module is used for constructing at least two private networks, and the private networks are used for respectively acquiring semantic independent information characteristics of the video input samples in each field;

the average value difference module is used for acquiring the characteristic data of the video input samples of the source domain and at least one target domain extracted by the initial video classification model, acquiring the maximum average value difference of the characteristic distribution distances extracted by the video classification model and each private network, and carrying out maximum processing on the maximum average value difference to obtain public semantic information characteristics;

And the iteration training module is used for iteratively training the initial video classification model and each private network, obtaining a target video classification model universal to the field when the iteration stopping condition is met, and carrying out video classification according to the target video classification model.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.

9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.

Images