CN109426793A - A kind of image behavior recognition methods, equipment and computer readable storage medium - Google Patents

Abstract

The invention discloses an image behavior recognition method, equipment and computer-readable storage medium. The method includes: dividing the sub-components in the target area in the to-be-recognized image, and determining the area where each sub-component is located; extracting the feature of each sub-component from the area where each sub-component is located, and determining the target belongs to according to the feature of each sub-component category of behavior. The invention performs image recognition through the local area full convolution network LRFCN model, and can effectively improve the recognition accuracy under the condition of only increasing the calculation overhead.

Description

Image behavior recognition method, device and computer-readable storage medium

technical field

The present invention relates to the technical field of image processing, and in particular, to an image behavior recognition, a device and a computer-readable storage medium.

Background technique

In recent years, with the continuous popularization of surveillance electronic equipment in various fields, the demand for more efficient monitoring of valuable information from surveillance video has become increasingly prominent. The traditional monitoring method is to use manual monitoring, but the method of manual monitoring of video is inefficient and the accuracy is difficult to guarantee, so there is an urgent need for a method that can intelligently discriminate the behavior in the video, and can detect the behavior of interest in the video. .

SUMMARY OF THE INVENTION

The present invention provides an image behavior recognition method, device and computer-readable storage medium, which are used to realize the problem of power consumption control of a terminal under the condition of constant battery capacity.

In order to realize the above-mentioned purpose of the invention, the present invention adopts the following technical scheme:

According to one aspect of the present invention, there is provided an image behavior recognition method, the method comprising:

Divide the sub-components in the area where the target in the image to be recognized is located, and determine the area where each sub-component is located;

The feature of each subcomponent is extracted from the area where each subcomponent is located, and the behavior category to which the target belongs is determined according to the feature of each subcomponent.

Optionally, the sub-component is divided into the area where the target in the image to be recognized is located, and the area where each sub-component is located is determined, including:

Divide the sub-components in the area where the target is located according to the preset average proportion of the sub-components;

The foreground and background of each sub-component area are divided by the region segmentation algorithm, and the foreground segmentation result of the sub-component is obtained.

Optionally, before the sub-components are divided into the region where the target is located according to a preset average ratio of sub-components, the method further includes:

Label the sub-components of the images containing the target in the sample data set;

Determine the proportion of the image occupied by the sub-component according to the marked area of the sub-component;

Counting the sum value of the scale values of the images where the same sub-component is located in the sample data set, and determining the average scale value of the sub-component according to the sum value, wherein the average scale value of the sub-component is the sum value of different sub-components ratio.

Optionally, the region segmentation algorithm includes at least one of the following: GrabCut algorithm, GraphCut algorithm and RandomWalker algorithm.

Optionally, the feature of each subcomponent is extracted from the area where each subcomponent is located, and the behavior category to which the target belongs is determined according to the feature of each subcomponent, including:

Perform feature extraction on the area where the sub-component is located and the area where the target is located;

The features extracted by the subcomponents are cascaded with the features extracted from the region where the target is located, and the cascaded features are used as the target features;

The behavior category to which the target belongs is determined from a preset classification model according to the target feature.

Optionally, determining the behavior category to which the target belongs from a preset classification model according to the target feature, including:

Determine the probability of belonging to each behavior category from the preset classification model according to the target feature;

The behavior category with the highest probability is selected as the behavior category to which the target belongs.

Optionally, before determining the behavior category to which the target belongs from the preset classification model according to the target feature, the method further includes:

Get the pre-trained classification model;

Establish a sample data set containing multiple types of behaviors, and label the target area, behavior category, and sub-components of the sample data set; train the pre-training classification model based on the labeled sample data set, and obtain the Describe the preset classification model.

Optionally, after obtaining the preset classification model, the method further includes:

cropping the images in the sample data set to augment the sample data set;

The energy loss function is optimized according to the expanded sample data set, and an optimized preset classification model is obtained.

According to one aspect of the present invention, there is provided an image behavior recognition device, comprising: a memory and a processor; wherein, the memory stores computer instructions, and when the computer instructions are executed by the processor, to realize the above image All and some of the steps in the behavior recognition method.

According to one aspect of the present invention, a computer-readable storage medium is provided, and the computer-readable storage medium stores one or more programs, when the one or more programs are executed by the processor, to achieve the above All steps and some steps in the image behavior recognition method.

The beneficial effects of the present invention are as follows:

The image behavior recognition method, device, and computer-readable storage medium provided by the embodiments of the present invention improve the pooling process by using a local area full convolution network. The resulting characteristics of the component are used to determine the final behavior category. Therefore, the present invention can effectively improve the recognition accuracy by extracting local features and only increasing the computational overhead.

The above description is only an overview of the technical solutions of the present invention, in order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand , the following specific embodiments of the present invention are given.

Description of drawings

In order to explain the embodiments of the present invention or the existing solutions more clearly, the following will briefly introduce the accompanying drawings that need to be used in the embodiments or existing descriptions. Obviously, the accompanying drawings in the following description are only the For some embodiments of the invention, for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a flowchart of an image behavior recognition method provided in an embodiment of the present invention;

2 is a network structure diagram of an image behavior recognition method provided in an embodiment of the present invention;

3 is a schematic diagram of feature cascade in an embodiment of the present invention;

FIG. 4 is a principle block diagram of an image behavior recognition device provided in an embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to illustrate the present invention, but not to limit the present invention.

In the field of computer vision, there are many methods for behavior recognition, but in many cases, the real-time performance and accuracy of background modeling, foreground object detection and tracking are difficult to meet the requirements. As a new branch of machine learning, deep learning has made good improvements in real-time and accuracy. In the field of target detection, there are some typical deep learning model solutions, which are mainly divided into two categories. One is regression-based methods such as YOLO (You Only Look Once), SSD (SingleShot Multibox Detector), etc. These methods are relatively efficient. High but limited accuracy. One is the method based on candidate regions, such as faster RCNN (region-based convolutional neural networks), RFCN (Region-based Fully Convolutional Networks), etc. These methods have higher accuracy but lower efficiency.

Considering that the behavior recognition problem and the target detection problem have a certain similarity, but the difficulty is greater, the present invention chooses to improve on the basis of the RFCN with the highest recognition accuracy at present, and proposes a local area based fully convolutional network ( The behavior recognition method of Local Region-based Fully Convolutional Networks (LRFCN for short) is used for video behavior recognition.

Method embodiment

The image behavior recognition method provided by the embodiment of the present invention, as shown in FIG. 1 and FIG. 2 , specifically includes the following steps:

Step 101 , sub-components are divided into the region where the target in the image to be recognized is located, and the region where each sub-component is located is determined.

Step 102: Extract the feature of each subcomponent from the area where each subcomponent is located, and determine the behavior category to which the target belongs according to the feature of each subcomponent.

In the embodiment of the present invention, the sub-components are divided into the region where the target is located, and the final behavior category to which the target belongs is determined according to the features extracted by the sub-components. Based on this, the present invention uses local features for identification, and only increases the calculation overhead. can effectively improve the recognition accuracy.

Wherein, in an optional embodiment of the present invention, when identifying the region where the target of the image to be identified (also referred to as the region of interest RoI) is identified, the region recommendation network RPN may be used to identify the region of interest RoI. The "regional recommendation network RPN" is a technology well known to those skilled in the art, and will not be described here. Of course, other identification technologies can also be used to identify the region of interest, which is not limited here. Wherein, before the target area of the to-be-recognized image is identified, the to-be-recognized image is subjected to a normalization process, so that a standard image in a unified form can be obtained after the image is normalized.

Wherein, in an optional embodiment of the present invention, when the target area in the image to be recognized is divided into sub-components, the sub-components are divided into the target area according to a preset average ratio value of the sub-components; using a region segmentation algorithm Perform foreground and background segmentation on each sub-component area divided to obtain the foreground segmentation result of the sub-component.

Here, the area where the sub-component is located is preliminarily divided by the average scale value of the sub-component. Wherein, the average proportion value of the sub-components is determined according to the sample data set during the training of the classification model (LRFCN model) to ensure the accuracy of the value.

Specifically, before the sub-component is divided into the region where the target is located according to the preset average proportion value of the sub-component, the average proportion value of the sub-component needs to be acquired. Here is how to determine the average proportion of sub-components, including the following:

Label the sub-components of the images containing the target in the sample data set;

Determine the proportion of the image occupied by the sub-component according to the marked area of the sub-component;

Counting the sum of the scale values of the images where the same sub-component is located in the sample data set, and determining the average scale value of the sub-component according to the sum value, wherein the average scale value of the sub-component is the ratio of the sum values of different sub-components .

Specifically, the calculation method of the average scale value of the subcomponent is as follows:

Among them, (part1) _i +(part2) _i +...(partk) _i =1; partk _i is the proportion of the k sub-components in the ith target in the area where the target is located; k is the number of sub-components; n is the training The number of targets contained in the sample dataset in the library.

That is to say, in the present invention, the proportions of subcomponents 1, subcomponents 2, ... subcomponents K in the ROI of all targets in the sample data set are summed up respectively, and then the average proportion between the subcomponents is determined according to the sum of the proportions. . For example, a specific embodiment divides the human body into three parts: the head, the body, and the lower limbs. Average the proportions of heads, bodies, and lower limbs of all people in the sample data set, assuming that the normalized proportion of the ith person is Head _i :BodyUp _i :BodyDown _i , where Head _i +BodyUp _i +BodyDown _i =1 , if there are n people in the sample data set, the average proportion is

Here, in order to ensure the accuracy of the sub-component area division, it is necessary to further accurately divide the preliminarily divided area. Specifically, during segmentation, a region segmentation algorithm is used to eliminate background interference, so as to distinguish the background and the foreground in each sub-component region, and obtain the foreground segmentation result of the sub-component.

Wherein, preferably, the region segmentation algorithm adopts any one of GrabCut algorithm, GraphCut algorithm or RandomWalker algorithm. Of course, other algorithms can also be used for implementation, which will not be introduced here, without departing from the core idea of the present invention, which are all within the protection scope of the present invention. Here, the GrabCut algorithm is taken as an example to describe the specific implementation process of segmentation.

First, an energy function E is defined to describe the optimization objective of segmentation, and its formula is as follows:

E(α,k,θ,z)=U(α,k,θ,z)+V(α,z)

Among them, the U function represents the area data item of the energy function, the V function represents the smooth item (boundary item) of the energy function; α is the image initialization label (the background label is 0, the foreground label is 1), and k is the GMM (Gaussian Mixture Model). ) number of Gaussian components, θ is the statistical parameters of GMM (weight of Gaussian components, mean vector, covariance matrix), z is the picture data of the sub-component.

Then, by solving the min-cut minimum cut of the energy function, the segmentation pixel set of the foreground and background can be obtained.

Wherein, in an optional embodiment of the present invention, the feature of each subcomponent is extracted from the area where each subcomponent is located, and the behavior category to which the target belongs is determined according to the feature of each subcomponent, including:

Feature extraction is performed on the area where the sub-component is located and the area where the target is located;

Concatenate the features extracted from the subcomponents and the features extracted from the target area, and the cascaded features are used as the target features;

According to the characteristics of the target, the behavior category to which the target belongs is determined from the preset classification model.

Specifically, when extracting the features of each subcomponent, the pixels in the region where the subcomponent is located are convolved with the convolution kernel, and the value after convolution is the feature of the subcomponent. But because LRFCN networks usually have many layers, the convolution operation can be iterated many times. Therefore, the range actually corresponding to the original original image is already larger than the area of the segmentation result.

Among them, in order to ensure the accuracy of image recognition, when extracting the characteristics of each sub-component, the overall characteristics of the region where the target is located are simultaneously extracted. For example, as shown in Figure 3, the local features corresponding to the three partial regions of the head, the body, and the lower limbs are mainly combined in series with the global features corresponding to the entire human body region to form the features that are finally used to describe the entire region.

Based on this, it can be seen that by cascading the features of each sub-component (local pooling) and the features extracted from the entire RoI region (overall pooling), only the introduction of computational overhead can be added. The number of features for image recognition can be The increase can effectively improve the recognition accuracy of the image. Of course, in an optional embodiment of the present invention, the cascading feature of each sub-component feature can also be used as the target feature for identification. Compared with the feature extracted based on the entire RoI area, the identification accuracy of the image can also be effectively improved.

Wherein, in an optional embodiment of the present invention, determining the behavior category to which the target belongs from a preset classification model according to the target feature includes: determining the probability of each behavior category to which the target feature belongs; selecting the behavior category with the highest probability as the target’s behavior category Belonging to the behavior category.

Further, in an embodiment of the present invention, before determining the behavior category to which the target belongs from the preset classification model according to the target feature, a preset classification model (LRFCN model) needs to be determined. Here, the method for determining the preset classification model is determined, which specifically includes the following:

Get a pretrained classification model;

A sample data set containing multiple types of behaviors is established, and the target area, behavior category, and sub-component area of the sample data set are marked, and the pre-trained classification model is trained based on the marked sample data set to obtain a preset classification model.

Here, when obtaining the pre-trained classification model, it is obtained by training on a large database, such as ImageNet, a large database. Specifically, when establishing a multi-class behavior sample dataset, all images in the dataset must have certain differences in background, shooting angle, illumination, and image scale methods. Then manually annotate the target area, behavior type, and the area of each sub-component in the image. Then, the parameters in the LRFCN model are adjusted by training the pre-training model through the labeled sample data set.

Further, optionally, after the pre-trained classification model is trained based on the labeled sample data set to obtain a preset classification model, the method further includes:

The sample data set is expanded by randomly cropping the images in the sample data set; the energy loss function is optimized according to the expanded sample data set, and an optimized preset classification model is obtained.

Specifically, when training the LRFCN model, the energy loss function is the sum of the cross-entropy loss and the bounding box regression loss, as shown in the following formula:

Among them, s is the softmax response of various types, t ^* represents the offset of the prediction result relative to the ground truth, and t is the offset of the prediction result relative to the preset frame. c ^* =0 indicates that the label of RoI is the background, when c ^* >0 [c ^* >0]=1, otherwise it is 0. L _reg represents the loss of the bounding box, and rc represents the fractional average pooling of the spatial location of the RoI class _c . The specific calculation method is shown in the following formula:

_Lreg (t,t ^* )=R(tt ^* )

t _x =(xx _a )/w _a , t _y =(yy _a )/h _a , t _w =log(w/w _a ), t _h =log(h/h _a )

t _x ^* =(x ^* -x _a )/w _a , t _y ^* =(y ^* -y _a )/h _a , t _w ^* =log(w ^* /w _a ), t _h ^* =log(h ^* /h _a )

Among them, R is the Smooth L1 loss function, x, y, w, h are the center point coordinates and width and height of the predicted bounding box, respectively, the subscript a is the center point coordinate and width and height of the preset box, and the superscript * is The center point coordinates and width and height of the ground truth.

Based on the above, it can be seen that the energy loss function can be used to estimate the inconsistency between the predicted value of the model and the real value. The smaller the energy loss function, the better the model accuracy. Therefore, the accuracy of the LRFCN model can be guaranteed by training the energy loss function to improve the recognition accuracy.

The training process of the LFRCN model in the present invention is described by taking five types of behavior recognition, such as eating, watching TV, playing with electronic equipment, falling, and child abuse in the home surveillance video as examples:

In step 201, a sample data set containing multiple types of behaviors is established.

Here, in response to the questions raised, a database is established that includes five categories of behaviors, including eating, watching TV, playing with electronic devices, falling, and child abuse. Each category contains about 2,000 images. The samples of these images are all taken from home surveillance. video.

Secondly, two-thirds of them are randomly selected as training samples and put into the training library, and the remaining one-third are used as test samples. All images contain content derived from real home surveillance video.

Step 202: Manually mark the target area, behavior category, and local areas such as the target head, body, and lower limbs in the image. Specifically, it includes the following:

Step 2021, manually annotate the ground truth of the target in the image, and simultaneously mark the target area and the behavior category label through the frame, for example, the category label is 0, 1, 2, 3, 4;

Step 2022, calibrating the head, body, and lower limbs of the human target in the sample image, and calculating the average proportion of the head, body, and lower limbs in each region according to the calibration result;

Step 2023, demarcate the specific pixel positions covered by the head, body, and lower limbs in the human target in the sample image, and record the specific pixel positions through the image template.

Step 203, obtaining a pre-trained LFRCN network model.

Because the neural network in the LFRCN network model contains a large number of parameters, and the number of samples in the sample data set established by itself is relatively small, it is prone to overfitting when directly training with the sample data set, so it is selected on the larger database ImageNet. First obtain the LFRCN network model, and then train the pre-trained LFRCN network model based on the training library.

Step 204: Train the pre-trained LFRCN network model based on the training library, and fine-tune the parameters of the network model. The process can be broken down into the following small steps:

Step 2041, normalize the size of the images in the training library, so that the largest side of the image is less than 600;

Step 2042, randomly crop each image in the training library to expand the database;

Since there are more network parameters and fewer samples, in order to avoid overfitting, the training library is expanded by randomly cropping images from the images for network training to increase the number of samples.

Step 2043: Optimize the above energy loss function to obtain the final LFRCN network model. Among them, in the training process, the initial learning rate is set to 0.000001 and 50% of the parameters are randomly discarded according to the loss rate of 0.5. The optimization process will not be introduced here, such as the least squares method and the gradient descent method.

Based on the above, it can be seen that the behavior recognition method based on the local area fully convolutional neural network (LRFCN) proposed by the present invention can very accurately detect the target behavior in the video, which can fill the technical gap of the current intelligent security to a certain extent.

Device Embodiment

According to an embodiment of the present invention, an image behavior recognition device is provided, which is used for realizing the above-mentioned image behavior recognition method. As shown in Figure 4. The device includes a processor 42 and a memory 41 storing instructions executable by the processor 42 . Specifically, in the image behavior recognition device provided in the embodiment of the present invention, when the executable instructions in the memory 41 are executed by the processor 42, the image behavior recognition method provided in the method embodiment is implemented. It should be noted that, in the device embodiment, the specific implementation will not be repeated, and reference may be made to the detailed description in the method embodiment, which will not be repeated in this embodiment.

The processor 42 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), or an application specific integrated circuit (ASIC), Or one or more integrated circuits configured to implement embodiments of the invention.

The memory 41 is used to store program codes and transmit the program codes to the CPU. The memory 41 may include a volatile memory (volatile memory) such as random access memory (RAM); the memory 41 may also include a non-volatile memory (non-volatile memory) such as a read-only memory (read-only memory). only memory (ROM), flash memory (flash memory), hard disk drive (HDD) or solid-state drive (SSD); the memory 41 may also include a combination of the above-mentioned types of memory.

Storage medium embodiment

Embodiments of the present invention also provide a computer-readable storage medium. The computer-readable storage medium herein stores one or more programs. The computer-readable storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state hard disk; the memory may also include the above-mentioned types combination of memory. When one or more programs in the computer-readable storage medium can be executed by one or more processors, all the steps and part of the steps in the image behavior recognition method provided by the method embodiments are implemented. For the specific implementation of the steps, reference may be made to the detailed description in the method embodiment, which will not be repeated in this embodiment.

Those of ordinary skill in the art can understand that the realization of all or part of the processes in the methods of the above embodiments can be accomplished by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, The processes of the embodiments of the various methods described above may be included.

Although the application has been described by way of examples, those skilled in the art will recognize that many modifications and variations of the application are possible without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (10)

1. an image behavior recognition method, is characterized in that, comprises: Divide the sub-components in the area where the target in the image to be recognized is located, and determine the area where each sub-component is located; The feature of each subcomponent is extracted from the area where each subcomponent is located, and the behavior category to which the target belongs is determined according to the feature of each subcomponent. 2. The image behavior recognition method according to claim 1, wherein the sub-component is divided into the region where the target in the image to be recognized is located, and the region where each sub-component is located is determined, comprising: Divide the sub-components in the area where the target is located according to the preset average proportion of the sub-components; The foreground and background of each sub-component area are divided by the region segmentation algorithm, and the foreground segmentation result of the sub-component is obtained. 3. The image behavior recognition method according to claim 2, characterized in that, before carrying out the division of sub-components to the region where the target is located according to a preset sub-component average ratio value, further comprising: Label the sub-components of the images containing the target in the sample data set; Determine the proportion of the image occupied by the sub-component according to the marked area of the sub-component; Counting the sum value of the scale values of the images where the same sub-component is located in the sample data set, and determining the average scale value of the sub-component according to the sum value, wherein the average scale value of the sub-component is the sum value of different sub-components ratio. 4 . The image behavior recognition method according to claim 2 , wherein the region segmentation algorithm comprises at least one of the following: GrabCut algorithm, GraphCut algorithm and RandomWalker algorithm. 5 . 5 . The image behavior recognition method according to claim 1 , wherein the feature of each subcomponent is extracted from the region where each subcomponent is located, and the behavior to which the target belongs is determined according to the feature of each subcomponent. 6 . categories, including: Perform feature extraction on the area where the sub-component is located and the area where the target is located; The features extracted by the subcomponents are cascaded with the features extracted from the region where the target is located, and the cascaded features are used as the target features; The behavior category to which the target belongs is determined from a preset classification model according to the target feature. 6. The image behavior recognition method according to claim 5, wherein the determining the behavior category to which the target belongs from a preset classification model according to the target feature comprises: Determine the probability of belonging to each behavior category from the preset classification model according to the target feature; The behavior category with the highest probability is selected as the behavior category to which the target belongs. 7. The image behavior recognition method according to claim 5, wherein before determining the behavior category to which the target belongs from a preset classification model according to the target feature, further comprising: Get the pre-trained classification model; Establish a sample data set containing multiple types of behaviors, and label the target area, behavior category, and sub-components of the sample data set; train the pre-training classification model based on the labeled sample data set, and obtain the Describe the preset classification model. 8. The image behavior recognition method according to claim 7, wherein after obtaining the preset classification model, the method further comprises: cropping the images in the sample data set to augment the sample data set; The energy loss function is optimized according to the expanded sample data set, and an optimized preset classification model is obtained. 9 . An image behavior recognition device, comprising: a memory and a processor; wherein, the memory stores computer instructions, and when the computer instructions are executed by the processor, claims 1 to 8 are implemented. 10 . The steps in any one of the image behavior recognition methods. 10. A computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, when the one or more programs are executed by the processor, to realize claim 1 Steps in the image behavior recognition method according to any one of ~8.