CN111832601A - State detection method, model training method, storage medium and electronic device - Google Patents

Abstract

The embodiment of the present invention discloses a state detection method, a model training method, a storage medium and an electronic device. By acquiring multiple feature vectors of an image sequence to be detected, multiple branch tasks and corresponding feature weight vector sets are determined. The feature vector and feature weight vector set determine the input vector corresponding to the branch task, and output the branch task state to finally determine the state detection result according to the multiple branch task states. In the embodiment of the present invention, the input vector is calculated according to the feature weight vector set corresponding to each branch task and a plurality of feature vectors, so that the input vector of each branch task can be determined in a targeted manner, and the processing of each branch task can be improved. Corresponding branch task state accuracy, thereby improving the accuracy of the final state detection result.

Description

State detection method, model training method, storage medium and electronic device

technical field

The present invention relates to the field of computer technology, and in particular, to a state detection method, a model training method, a storage medium and an electronic device.

Background technique

In the field of deep learning, the detection process of a data state usually uses multiple models. At present, there are mainly two methods for using multiple models to detect the data state. Among them, the single-task detection method in the prior art has poor effect on complex application scenarios, and the calculation process is complicated, which will occupy a large amount of computing resources in the actual application process, and is not suitable for most real use scenarios. Although the multi-task detection method can save computational overhead, since multiple models share one input feature, these features are forcibly instilled into the training of each model, which will cause the training of each model to become inefficient, making it difficult to achieve a better performance than a single model. The task model has better detection effect.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present invention provide a state detection method, a model training method, a storage medium, and an electronic device, aiming at improving the accuracy of the detection result of the state detection performed by the multi-task model.

In a first aspect, an embodiment of the present invention discloses a state detection method, the method comprising:

determining a sequence of images to be detected, the sequence of images to be detected including at least one image to be detected;

Each of the images to be detected is sequentially passed through N convolutional layers to obtain corresponding N feature vectors, where N is an integer greater than or equal to 2;

Identify multiple branch tasks;

Determine a set of feature weight vectors corresponding to each of the branch tasks, and each of the feature weight vectors corresponds to each of the feature vectors;

Determine the input vector of each of the branch tasks according to the N feature vectors corresponding to each of the to-be-detected images and the set of feature weight vectors;

Inputting each of the input vectors into a picture classifier for processing the corresponding branch task to determine the branch task state corresponding to the image sequence to be detected;

The state detection result is determined according to the branch task state corresponding to the image sequence to be detected.

Further, obtaining the corresponding N feature vectors of each of the to-be-detected images sequentially through N convolutional layers is specifically:

Each of the images to be detected is input into a pre-trained convolutional neural network, and the outputs of the N convolutional layers in the convolutional neural network are respectively used as N feature vectors corresponding to the input images to be detected.

Further, determining the input vector of each branch task according to the N feature vectors corresponding to each of the to-be-detected images and the feature weight vector set respectively includes:

Determine the target branch task;

The N feature vectors corresponding to each of the images to be detected are weighted according to the feature weight vector set corresponding to the target branch task to determine the input vector of the target branch task.

Further, according to the feature weight vector set corresponding to the target branch task, the N feature vectors corresponding to each of the to-be-detected images are weighted to determine that the input vector of the target branch task is specifically:

Determine the target image to be detected;

For each of the target branch tasks, weight each of the feature vectors corresponding to the target image to be detected according to each of the feature weights in the corresponding feature weight vector set;

Each of the weighted feature vectors is input into a nonlinear mapping function respectively, and the sum of each of the output results is calculated to determine the input vector of the target image to be detected for each of the target branch tasks.

Further, determining the state detection result according to the branch task state corresponding to the image sequence to be detected is specifically:

The branch task state corresponding to the image sequence to be detected is input into the state sub-model and the state detection result is output.

In a second aspect, an embodiment of the present invention discloses a method for training a state detection model, the method comprising:

Determine a training set, the training set includes a plurality of images to be detected and corresponding branch task states;

Each of the images to be detected is used as the input of the state detection model, and the corresponding branch task state is used as the output, and the undetermined model parameters in the state detection model are predicted. The state detection model includes a feature vector extraction sub-model and a classification sub-model. Model;

In an iterative manner, each of the to-be-detected images in the training set is input into a state detection model with each of the to-be-determined model parameters as model parameters to obtain the corresponding prediction branch task state, where the model parameters include feature vector extraction parameters, Feature weight vector parameters and classification model parameters;

Determine a loss coefficient according to the predicted branch task state and the branch task state corresponding to each of the to-be-detected images;

The pending model parameters are updated until the loss coefficient no longer decreases.

Further, the updating of the undetermined model parameters until the loss coefficient is no longer reduced is specifically:

Determine the pending model parameters to be updated according to the back-propagation algorithm;

The pending model parameters to be updated are updated to new pending model parameters until the loss coefficient is no longer reduced.

Further, the state detection model also includes an attention mechanism;

The feature vector extraction parameters are parameters of the feature vector extraction sub-model, the feature weight vector parameters are parameters of the attention mechanism, and the classification model parameters are parameters of the classification sub-model.

In a third aspect, an embodiment of the present invention discloses a computer-readable storage medium for storing computer program instructions, the computer program instructions, when executed by a processor, implement any one of the first aspect and the second aspect. method described.

In a fourth aspect, an embodiment of the present invention discloses an electronic device, including a memory and a processor, where the memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are processed by the The machine executes to implement the method of any one of the first and second aspects.

In the embodiment of the present invention, the input vector is calculated according to the feature weight vector set corresponding to each branch task and a plurality of feature vectors, so that the input vector of each branch task can be determined in a targeted manner, and the processing of each branch task can be improved. Corresponding branch task state accuracy, thereby improving the accuracy of the final state detection result.

Description of drawings

The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the present invention with reference to the accompanying drawings, in which:

1 is a flowchart of a state detection method according to an embodiment of the present invention;

2 is a schematic diagram of a system for implementing a state detection method according to an embodiment of the present invention;

3 is a schematic diagram of a data flow of a state detection method according to an embodiment of the present invention;

4 is a flowchart of a state detection model training method according to an embodiment of the present invention;

5 is a schematic diagram of a data flow of a state detection model training method according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

The present invention is described below based on examples, but the present invention is not limited to these examples only. In the following detailed description of the invention, some specific details are described in detail. The present invention can be fully understood by those skilled in the art without the description of these detailed parts. Well-known methods, procedures, procedures, components and circuits have not been described in detail in order to avoid obscuring the essence of the present invention.

Furthermore, those of ordinary skill in the art will appreciate that the drawings provided herein are for illustrative purposes and are not necessarily drawn to scale.

Unless clearly required by the context, words such as "including", "comprising" and the like in the specification should be construed in an inclusive rather than an exclusive or exhaustive sense; that is, in the sense of "including but not limited to".

In the description of the present invention, it should be understood that the terms "first", "second" and the like are used for descriptive purposes only, and should not be construed as indicating or implying relative importance. Also, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

FIG. 1 is a flowchart of a state detection method according to an embodiment of the present invention. As shown in FIG. 1 , the state detection method includes the following steps:

Step S100, determining the image sequence to be detected.

Specifically, the sequence of images to be detected includes at least one image to be detected, which is determined by the server. The method for the server to determine the sequence of images to be detected may be, for example, receiving images to be detected uploaded by a terminal device through a preset application program interface, wherein the terminal device is a general-purpose data processing terminal with a communication function capable of running a computer program, For example, smart phones, tablet computers, notebook computers, and other terminal equipment loaded with communication modules, etc. At least one to-be-detected image of the to-be-detected image sequence includes image information for detecting a corresponding state. For example, when the state detection method is used to detect a traffic flow state at a certain intersection, each image in the to-be-detected image sequence is The to-be-detected images all include intersection information; when the state detection method is used to detect a face state, the to-be-detected images in the to-be-detected image sequence include human face information.

FIG. 2 is a system schematic diagram of a state detection method according to an embodiment of the present invention. As shown in FIG. 2 , the system for performing the state detection method may only include a system for acquiring a sequence of images to be detected, and performing state detection on the sequence of images to be detected. The detected terminal device 21, or includes a server 20 and a terminal device 21 connected through a network, the terminal device 21 is used to upload the image sequence to be detected to the server 20, and the server 20 is used to perform status based on the image sequence to be detected. detection.

Specifically, the system of the state detection method described in the embodiment of the present invention can be applied to various state detection scenarios, such as driver fatigue detection, road congestion degree detection, user emotion detection, and the like. Taking the application scenario of detecting the driver's fatigue driving state as an example, in an optional implementation manner of the embodiment of the present invention, the terminal device 21 is a data processing device installed in the vehicle and having a camera function, which is used to periodically A sequence of images to be detected including a driver's facial state is collected, and driver facial features are extracted from the sequence of images to be detected to detect whether the driver is in a fatigued driving state. In another optional implementation manner of the embodiment of the present invention, the terminal device 21 may be a data processing device installed in the vehicle and having a communication function and a camera function, and is used to periodically collect the to-be-detected data including the driver's facial state. image sequence, and upload the to-be-detected image sequence to the server 20 . The server 20 is configured to extract the driver's facial features from the image sequence to be detected uploaded by the terminal device 21 to detect whether the driver is in a fatigued driving state.

Taking the application scenario of detecting the degree of road congestion as an example, in an optional implementation manner of the embodiment of the present invention, the terminal device 21 is a data processing device installed at an intersection and having a camera function, which is used to periodically collect data including: A to-be-detected image sequence of the state of vehicles passing through the intersection, and vehicle motion features are extracted from the to-be-detected image sequence to detect whether the intersection is in a congested state. In another optional implementation manner of the embodiment of the present invention, the terminal device 21 may be a data processing device installed at an intersection and having a communication function and a camera function, and is used to periodically collect images to be detected including vehicle status at the intersection. sequence, and upload the to-be-detected image sequence to the server 20 . The server 20 is configured to extract vehicle motion features from the image sequence to be detected uploaded by the terminal device 21 to detect whether the intersection is in a congested state.

Further, the state detection method described in the embodiment of the present invention and the scene in which the state detection method is applied are also applicable to the detection of other types of data states, such as state detection through audio recording content, state detection through text recording content. Wait.

Step S200 , passing each of the to-be-detected images sequentially through N convolutional layers to obtain corresponding N feature vectors.

Specifically, after determining the sequence of images to be detected, the server sequentially performs feature extraction on each of the images to be detected included in the sequence of images to be detected. In the embodiment of the present invention, the feature extraction process is to input each of the to-be-detected images in the to-be-detected image sequence into N convolutional layers in turn, and use the output of each convolutional layer as a corresponding feature vector. In an optional implementation manner of the embodiment of the present invention, each of the convolutional layers is a common image classifier feature extraction structure, including but not limited to algorithms such as Alexnet, GoogleNet, VGG, and ResNet. The feature vectors are obtained based on the feature vectors output by the previous layer.

FIG. 3 is a schematic diagram of a data flow of a state detection method according to an embodiment of the present invention. As shown in FIG. 3 , the N convolutional layers may belong to the same convolutional neural network 30, and the convolutional neural network 30 includes N Or more than N number of convolutional layers. When the server performs feature extraction on an image to be detected, the image to be detected is input into the convolutional neural network 30, and the image to be detected passes through a plurality of convolutional layers in sequence in the convolutional neural network 30, wherein The output of each convolutional layer is used as the input of the next convolutional layer. The server determines, in the convolutional neural network 30, N outputs f _i (1≤i≤N) corresponding to the convolutional layers 1 to N corresponding to the convolutional layers N as feature vectors corresponding to the image to be detected. Wherein, the convolutional neural network 30 can be obtained by training multiple labeled image information and multiple feature vectors corresponding to each of the image information, that is, the image information is used as input, and multiple corresponding feature vectors are used as each The output training of the convolutional layer trains the convolutional neural network 30 .

Taking the state detection method for detecting the driver's fatigue driving state as an example for illustration, after the server obtains the image sequence to be detected, it sequentially inputs each image to be detected in the sequence of images to be detected into a pre-trained volume. A convolutional neural network 30, the first convolutional layer of the convolutional neural network 30 outputs the feature vector f1 corresponding to the portrait information in each _of the images to be detected, and the second convolutional layer outputs each of the portrait information. The feature vector f ₂ corresponding to the face information in the third convolution layer outputs the feature vector f ₃ corresponding to the eye information in each of the face information.

Step S300, determining a plurality of branch tasks.

Specifically, the server may preset a plurality of branch tasks, and each branch task is used to determine a corresponding branch task state, so that the server can determine the image sequence to be detected according to the branch task state of each branch task. state. As shown in FIG. 3 , in this embodiment of the present invention, each of the branch tasks corresponds to a pre-trained image classifier 32 , so that each of the image classifiers 32 can use the image classifier 32 according to the feature vector corresponding to each of the images to be detected. f _i (1≤i≤N) determines the corresponding branch task state.

Taking the state detection method for detecting a driver's fatigue driving state as an example for illustration, the branch tasks may include a face detection task, an eye opening and closing detection task, and a yawning detection task. Each of the branch tasks respectively corresponds to the image classifier 1 for detecting fatigue, the image classifier 2 for detecting blinking, and the image classifier 3 for detecting yawning. The branch task states output by each of the picture classifiers are fatigue, blinking, and yawning.

Step S400: Determine a feature weight vector set corresponding to each branch task.

Specifically, since each branch task is used to detect different branch task states, in order to improve the accuracy of each branch task state, corresponding input vectors need to be generated for different branch tasks. In the embodiment of the present invention, the input vector corresponding to each branch task needs to be determined according to the feature weight vector set corresponding to each branch task, wherein each feature weight vector corresponds to each feature vector through pre-training. The training method of the feature weight vector may be obtained by training the attention mechanism through the server. For example, the server may construct an input and output training set of the attention mechanism in advance, train the attention mechanism through the training set, and adjust various parameters of the attention mechanism during the training process until the loss reaches a minimum When the value is set, each parameter of the attention mechanism at this time is used as a feature weight vector to construct the feature weight vector set.

Step S500: Determine the input vector of each branch task according to the N feature vectors corresponding to each of the to-be-detected images and the feature weight vector set.

Specifically, after determining the N feature vectors corresponding to each of the to-be-detected images, each of the branch tasks, and a set of feature weight vectors corresponding to each of the branch tasks, the server determines each of the branch tasks according to the above parameters the corresponding input vector. In this embodiment of the present invention, the server may determine a target branch task in the branch task, use each element in the feature weight vector set corresponding to the target branch task as a parameter of the attention mechanism, and obtain an image corresponding to the target branch task. The N feature vectors of are used as the input of the attention mechanism, and the output corresponds to the target branch task and is used to represent the input vector of the image to be detected. The specific method for determining the input vector by the attention mechanism is to weight the N feature vectors corresponding to each of the to-be-detected images according to the feature weight vector set corresponding to the target branch task, and then based on the N weighted feature vectors obtained. Determine the input vector for the target branch task. That is, first determine the target image to be detected, and for each target branch task, weight each of the feature vectors corresponding to the target to-be-detected image according to each of the feature weights in the corresponding feature weight vector set, and finally weight each of the The weighted feature vectors are respectively input to the nonlinear mapping function, and the sum of each of the output results is calculated to determine the input vector of the target to-be-detected image for each of the target branch tasks. The target to-be-detected image is determined in the to-be-detected image sequence, and after each input vector corresponding to the target to-be-detected image is determined, the server may re-determine the next target in the to-be-detected image sequence The image to be detected to determine the corresponding input vector again.

Further, the input vector of the target branch task can be determined by the following formula:

Wherein, F _j is the input vector corresponding to the target branch task, M is the number of branch tasks, N is the number of the feature vectors, f _i is the feature vector, α _ji is the target branch task and the feature vector The corresponding feature weight vector, Φ is a nonlinear mapping function. As shown in FIG. 3, the server inputs the image to be detected into the convolutional neural network 30, and outputs N feature vectors f _i (1≤i≤N) including the features of the image to be detected, and each of the feature vectors f _i ( 1≤i≤N) is input to the attention mechanism 31 to pass the feature weight vector set α _ji (1≤i≤N, 1≤j≤M) corresponding to the target branch task and each of the feature vectors f _i (1 ≤i≤N) to determine the input vector F _j (1≤j≤M) corresponding to the target branch task. For example, when the target branch task is processed by the image classifier 1, the server assigns the feature weight vector set α _1i (1≤i≤N) and the feature vector f _i (1≤i) corresponding to the image classifier 1 ≤N) Input the above formula to obtain the input vector F ₁ corresponding to the target branch task. Further, after determining the input vector corresponding to a target branch task, the server determines that the branch tasks processed by other image classifiers are the target branch task, and then determines the input vector corresponding to the new target branch task, until the server obtains all the target branch tasks. Input vector for branching tasks.

Step S600: Input each of the input vectors into a picture classifier for processing the corresponding branch task to determine the branch task state corresponding to the image sequence to be detected.

Specifically, for each image to be detected, after the server determines the input vector corresponding to each branch task according to the corresponding N feature vectors and the feature weight vector set corresponding to each branch task, the server determines the input vector corresponding to each branch task. The input vectors corresponding to the branch tasks are respectively input to a picture classifier for processing the corresponding branch tasks, so as to output the corresponding branch task states. After determining the branch task states of all the images to be detected in the image sequence to be detected, the server determines the branch task state corresponding to the image sequence to be detected based on the branch task states of all the images to be detected. Wherein, each of the picture classifiers is pre-trained by the server.

The state detection method is used to detect the driver's fatigue driving state, and the branch tasks include the face detection task, the eye opening and closing detection task, and the yawning detection task as an example for description. For each of the to-be-detected images in the to-be-detected image sequence, the server inputs the input vector corresponding to each of the branch tasks into the corresponding image classifier, so as to determine the faces in the to-be-detected images respectively according to the output. Branch task status for fatigue, blinking, and yawning. Further, the server also determines the branch task state corresponding to the image sequence to be detected based on the branch task states of all the images to be detected, such as the number of images judged to be fatigued, the number of images judged to be blinking, and the number of images judged to be yawning. number of images.

Step S700: Determine a state detection result according to the branch task state corresponding to the image sequence to be detected.

Specifically, after the server determines the branch task state corresponding to the image sequence to be detected, the server determines the state detection result corresponding to the image sequence to be detected according to each of the score task states and preset rules. In an optional implementation manner of the embodiment of the present invention, the preset rule may be, for example, that each branch task state is used as a state sub-model obtained by pre-training, and a corresponding state detection result is output. In another optional implementation manner of the embodiment of the present invention, the preset rule may be, for example, setting conditions corresponding to each branch task state, and determining the final decision according to whether each branch task state meets the corresponding condition status detection result.

The state detection method is used to detect the driver's fatigue driving state, and the branch tasks include a face detection task, an eye opening and closing detection task, and a yawning detection task, and each branch task state corresponding to the image sequence to be detected corresponds to The number of images of fatigue state, the number of images of blinking, and the number of images of yawning are taken as examples for description. In this embodiment of the present invention, the server may set a fatigue state image probability threshold, a blinking image probability threshold, and a yawning image probability threshold, and the server determines the state of each branch task corresponding to the image sequence to be detected. Then, calculate the probability of fatigue state, blinking and yawning images in all the images to be detected. When the P probabilities are greater than the corresponding probability thresholds, the server determines that the state detection result corresponding to the image sequence to be detected is fatigue driving, and the P is a constant preset by the server. As shown in FIG. 3 , the specific process of the state detection method is that the server sequentially inputs each of the to-be-detected images in the to-be-detected image sequence into a convolutional neural network 30 to determine corresponding N feature vectors f _i (1≤i≤N), and then input the N feature vectors f _i (1≤i≤N) into the attention mechanism 31 with different parameters, so as to output the input vectors F _j corresponding to the branch tasks respectively ( 1≤j≤M). Wherein, when determining the input vector corresponding to the branch task j, the parameter of the attention mechanism is the feature weight vector set α _ji corresponding to the branch task j (1≤i≤N, 1≤j≤M). After the server determines each of the input vectors F _j (1≤j≤M), then each of the input vectors F _j (1≤j≤M) is respectively input into the picture classifier 32 for processing the corresponding branch task to obtain: The corresponding branch task status. After the server has determined the branch task states corresponding to all the images to be detected, it further obtains the branch task states corresponding to the image sequence to be detected and inputted to the state detection module 33 to determine the final state detection result according to a predetermined rule.

Further, in the embodiment of the present invention, the convolutional neural network 30, the attention mechanism 31, and the image classifier 32 applied in the process of determining the state detection result by the server may be obtained by pre-training respectively. In an implementation manner, the convolutional neural network 30, the attention mechanism 31 and the image classifier 32 can also be trained as a whole as a sub-model in a state detection model.

4 is a flowchart of a state detection model training method according to an embodiment of the present invention. The training process of the state detection model may be completed in the above-mentioned server for performing the state detection process, or may be completed in other servers or other terminal devices . As shown in Figure 4, the state detection model training method includes the following steps:

Step S800, determining a training set.

Specifically, the training set includes a plurality of images to be detected and corresponding branch task states, which are used as the input and output of the state detection model. The to-be-detected image may correspond to one or more branch task states, and the branch task states are obtained by pre-marking and fusion. Further, the training set may also include multiple sub-training sets of different types of branch task states, and the images to be detected in each sub-training set correspond to the branch task states one-to-one. The state detection model may include a plurality of input and output sub-paths, each of the sub-training sets corresponds to a sub-path in the state detection model, and is used for training the sub-paths. For example, when the state detection model is used to detect the driver's fatigue driving state, and the state detection model includes sub-paths for detecting fatigue, blinking, and yawning, the training set includes sub-paths for training each driver. The fatigue sub-training set, the blinking sub-training set, and the yawning sub-training set of the sub-model.

Step S900 , taking each of the images to be detected as the input of the state detection model, and the corresponding branch task state as the output, to predict the undetermined model parameters in the state detection model.

Specifically, after the training set is determined, each image to be detected in the training set is used as the input of the state detection model, and the corresponding branch task state is used as the output to predict the undetermined model parameters in the state detection model. Further, the state detection model includes a feature vector extraction sub-model, an attention mechanism and a classification sub-model, wherein the model parameters of each of the sub-models are respectively a feature vector extraction parameter, a feature weight vector parameter and a classification model parameter. In an optional implementation manner of the embodiment of the present invention, the state detection model includes only one input and output path, and the training process is to sequentially input the images to be detected in the training set into the feature vector extraction sub-model, pay attention to The force mechanism and the classification sub-model take the corresponding branch task state as an output to predict the pending model parameters in the state detection model.

In another optional implementation manner of the embodiment of the present invention, the state detection model includes a plurality of input and output sub-paths. The attention mechanism may correspond to a plurality of feature weight vector parameters, the classification sub-model may include multiple, and each of the feature weight vector parameters and each of the classification sub-models corresponds to a sub-model extracted from the feature vector , attention mechanism, and sub-paths composed of sequential connections of classification sub-models. When training the sub-paths, the images to be detected in the sub-training set corresponding to the sub-paths are sequentially input into the feature vector extraction sub-model, the attention mechanism and the classification sub-model, and the corresponding branch task state is used as the output to predict the corresponding The undetermined model parameters of each sub-model in the sub-path.

Step S1000 , input each of the to-be-detected images in the training set into a state detection model with each of the to-be-determined model parameters as model parameters in an iterative manner, to obtain a corresponding prediction branch task state.

Specifically, after the undetermined model parameters are obtained through prediction, each of the to-be-detected images in the training set is input into a state detection model whose model parameters are the undetermined model parameters, and the corresponding prediction branch task state is output. That is, when the state detection model includes only one input and output path, the image to be detected is input into the feature vector extraction sub-model, attention mechanism and classification sub-model in sequence, and the corresponding prediction branch task state is output. When the state detection model includes multiple input and output sub-paths, input the images to be detected in each of the sub-training sets into the sub-paths composed of the corresponding feature vector extraction sub-model, attention mechanism and classification sub-model, and output the corresponding sub-paths. Subpredict branch task state.

Step S1100: Determine a loss coefficient according to the predicted branch task state and the branch task state corresponding to each image to be detected.

Specifically, the predicted branch task state is compared with the branch task state corresponding to the image to be detected input by the state detection model to determine the corresponding loss coefficient. For example, when the output of the state detection model is 0.5 and the branch task state corresponding to the to-be-detected image is 1, the loss coefficient is 0.5 of the difference between the branch task state and the predicted branch task state. When the state detection model includes multiple input and output sub-paths, the loss coefficient is determined to be the sum of the sub-loss coefficients corresponding to each of the sub-paths.

Step S1200: Update the undetermined model parameters until the loss coefficient is no longer reduced.

Specifically, after the loss coefficient is determined, the undetermined model parameters are re-determined, and the new undetermined model parameters are used as the parameters of the state detection model. The above steps S1000 and S1100 are repeated until the loss coefficient is no longer reduced. In this embodiment of the present invention, the undetermined model parameters may be determined through a back-propagation algorithm. Further, the condition for stopping updating the parameters of the undetermined model may also be setting a loss coefficient threshold, and when the loss coefficient reaches the threshold, the training of the model is stopped, and so on.

5 is a schematic diagram of a data flow of a state detection model training method according to an embodiment of the present invention, which is used to illustrate the data flow during training of a state detection model including only one input and output path, or a state detection model including multiple input and output sub-paths The data flow of a subpath training process in . As shown in Figure 5, the training method of the state detection model is:

Obtain an image to be detected in the training set and input the feature vector extraction sub-model 50, attention mechanism 51 and classification sub-model 52 in turn, take the corresponding branch task state as output, and predict the feature vector extraction sub-model 50, attention mechanism 51 and the undetermined feature vector extraction parameters, undetermined feature weight vector parameters and undetermined classification model parameters corresponding to the classification sub-model 52 . Then, each of the images to be detected in the training set is sequentially input into the feature vector extraction sub-model 50, the attention mechanism 51 and the classification sub-model 52, and the corresponding prediction branch task state is output, wherein the model parameters of the above sub-models are respectively Extracting parameters, undetermined feature weight vector parameters and undetermined classification model parameters for the undetermined feature vector. After the predicted branch task state is obtained, the predicted branch task state is compared with the branch task state corresponding to the to-be-detected image to obtain a corresponding loss coefficient. When the loss coefficient does not meet the conditions preset by the server, the undetermined feature vector extraction parameters, undetermined feature weight vector parameters and undetermined classification model parameters to be updated are re-determined, and the to-be-updated model parameters are respectively updated to new ones model parameters, and then re-determine the loss coefficient until the loss coefficient meets the preset condition. The pending model parameters to be updated may be determined by a back-propagation algorithm. In an optional implementation manner of the embodiment of the present invention, the preset condition may be that the loss function is no longer reduced during multiple adjustment of the model parameters. In another optional implementation manner of the embodiment of the present invention, the preset condition may also be to set a loss coefficient threshold, and stop training the model when the loss coefficient reaches the threshold.

Further, the feature vector extraction sub-model 50, the attention mechanism 51 and the classification sub-model 52 may also pre-determine corresponding model parameters, that is, when training the state detection model, the state detection model may include The trained submodel. Since the model parameters of the already trained sub-model have been determined, there is no need to participate in the training process again, and the server only adjusts the undetermined model parameters when training the state detection model. For example, when the to-be-determined feature vector extraction sub-model and classification sub-model are pre-trained models, the server considers that the feature vector extraction parameters and classification model parameters have been determined, and in the process of training the state detection model , only the parameter of the feature weight vector is adjusted until the loss function satisfies the preset condition.

The embodiment of the present invention determines multiple branch tasks and corresponding feature weight vector sets by acquiring multiple feature vectors of the image sequence to be detected, determines the input vector corresponding to the branch task according to each of the feature vectors and feature weight vector sets, and outputs the output Branching task states to finally determine the state detection result according to the plurality of branching task states. The method calculates the input vector respectively according to the feature weight vector set corresponding to each of the branch tasks and a plurality of feature vectors, which can realize the targeted determination of the input vector of each of the branch tasks, and improve the corresponding processing of the branch tasks. The accuracy of the branch task status is improved, thereby improving the accuracy of the final status detection result.

FIG. 6 is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device shown in FIG. 6 is a general-purpose data processing apparatus, which includes a general-purpose computer hardware structure, which at least includes a processor 60 and a memory 61 . The processor 60 and the memory 61 are connected by a bus 62 . The memory 61 is adapted to store instructions or programs executable by the processor 60 . The processor 60 may be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 60 executes the commands stored in the memory 61 to execute the above-described method flow of the embodiments of the present invention to process data and control other devices. The bus 62 connects the above-mentioned various components together, while connecting the above-mentioned components to the display controller 63 and the display device and the input/output (I/O) device 64 . Input/output (I/O) device 64 may be a mouse, keyboard, modem, network interface, touch input device, somatosensory input device, printer, and other devices known in the art. Typically, input/output (I/O) devices 64 are connected to the system through input/output (I/O) controllers 65 .

Among them, the memory 61 can store software components, such as an operating system, a communication module, an interaction module, and an application program. Each of the modules and applications described above corresponds to a set of executable program instructions that perform one or more functions and methods described in embodiments of the invention.

The foregoing flowchart and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention describe various aspects of the present invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine such that the instructions (executed via the processor of the computer or other programmable data processing apparatus) create for implementing Flowchart and/or block diagram blocks or means for the functions/acts specified in the blocks.

Also, as will be appreciated by those skilled in the art, various aspects of the embodiments of the present invention may be implemented as a system, method or computer program product. Accordingly, various aspects of embodiments of the present invention may take the form of an entirely hardware implementation, an entirely software implementation (including firmware, resident software, microcode, etc.), or may be generally referred to herein as "circuits," "modules," ” or “system” that combines software aspects with hardware aspects. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable media having computer readable program code embodied thereon.

Any combination of one or more computer-readable media may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media would include the following: electrical connections with one or more wires, portable computer floppy disks, hard disks, random access memory (RAN), read only memory ( RON), erasable programmable read only memory (EPRON or flash memory), optical fiber, portable compact disk read only memory (CD-RON), optical storage, magnetic storage, or any suitable combination of the foregoing. In the context of embodiments of the present invention, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, device, or apparatus.

A computer-readable signal medium may include a propagated data signal having computer-readable program code embodied therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, and communicate a program for use by or in conjunction with the instruction execution system, apparatus, or apparatus. or transmission.

Computer program code for carrying out operations directed to aspects of the present invention may be written in any combination of one or more programming languages including: object-oriented programming languages such as Java, SNalltalk, C++, PHP, Python etc.; and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package; partly on the user's computer and partly on a remote computer; or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (eg, by using an Internet service provider's Internet) .

The present invention also relates to a computer-readable storage medium for storing a computer-readable program, the computer-readable program being used for a computer to execute some or all of the above method embodiments.

That is, those skilled in the art can understand that all or part of the steps in the method for implementing the above embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium and includes several instructions to make a device ( It may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, read-only memory (RON, Read-Only NeNory), random access memory (RAN, Random Access NeNory), magnetic disk or optical disk and other media that can store program codes.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A method of condition detection, the method comprising:

determining an image sequence to be detected, wherein the image sequence to be detected comprises at least one image to be detected;

sequentially passing each image to be detected through N convolutional layers to obtain corresponding N characteristic vectors, wherein N is an integer greater than or equal to 2;

determining a plurality of branch tasks;

determining a feature weight vector set corresponding to each branch task, wherein each feature weight vector corresponds to each feature vector;

respectively determining the input vector of each branch task according to the N characteristic vectors corresponding to each image to be detected and the characteristic weight vector set;

inputting each input vector into a picture classifier for processing a corresponding branch task to determine a branch task state corresponding to the image sequence to be detected;

and determining a state detection result according to the branch task state corresponding to the image sequence to be detected.

2. The method according to claim 1, wherein the obtaining of the corresponding N eigenvectors from each of the images to be detected sequentially through the N convolutional layers is specifically:

inputting each image to be detected into a convolutional neural network obtained by pre-training, and respectively taking the output of N convolutional layers in the convolutional neural network as N characteristic vectors corresponding to the input image to be detected.

3. The method according to claim 1, wherein the determining the input vector of each of the branch tasks according to the N feature vectors corresponding to each of the images to be detected and the feature weight vector set respectively comprises:

determining a target branch task;

and weighting the N eigenvectors corresponding to the images to be detected according to the characteristic weight vector set corresponding to the target branch task to determine the input vector of the target branch task.

4. The method according to claim 3, wherein the weighting the N feature vectors corresponding to each image to be detected according to the feature weight vector set corresponding to the target branch task to determine the input vector of the target branch task specifically comprises:

determining an image to be detected of a target;

for each target branch task, weighting each feature vector corresponding to the target image to be detected according to each feature weight in the corresponding feature weight vector set;

and respectively inputting each weighted feature vector into a nonlinear mapping function, and calculating the sum of each output result to determine the input vector of the target image to be detected for each target branch task.

5. The method according to claim 1, wherein the determining a state detection result according to the branch task state corresponding to the image sequence to be detected specifically comprises:

and inputting the branch task state corresponding to the image sequence to be detected into a state sub-model and outputting a state detection result.

6. A method for training a state detection model, the method comprising:

determining a training set, wherein the training set comprises a plurality of images to be detected and corresponding branch task states;

taking each image to be detected as the input of a state detection model, taking the corresponding branch task state as the output, and predicting the model parameters to be determined in the state detection model, wherein the state detection model comprises a feature vector extraction sub-model and a classification sub-model;

inputting each image to be detected in the training set into a state detection model taking each model parameter to be detected as a model parameter in an iterative manner so as to obtain a corresponding predicted branch task state, wherein the model parameter comprises a feature vector extraction parameter, a feature weight vector parameter and a classification model parameter;

determining a loss coefficient according to the predicted branch task state and the branch task state corresponding to each image to be detected;

updating the pending model parameters until the loss factor is no longer reduced.

7. The method of claim 6, wherein said updating the pending model parameters until the loss factor no longer decreases is specifically:

determining undetermined model parameters to be updated according to a back propagation algorithm;

and updating the undetermined model parameters to be updated into new undetermined model parameters until the loss coefficient is not reduced any more.

8. The method of claim 6, further comprising an attention mechanism in the state detection model;

the feature vector extraction parameters are parameters of the feature vector extraction submodel, the feature weight vector parameters are parameters of the attention mechanism, and the classification model parameters are parameters of the classification submodel.

9. A computer readable storage medium storing computer program instructions, which when executed by a processor implement the method of any one of claims 1-8.

10. An electronic device comprising a memory and a processor, wherein the memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any of claims 1-8.

Images