CN111723667A - Method and device for crowd behavior recognition of smart light poles based on human body joint point coordinates - Google Patents

Abstract

The present invention relates to a method for recognizing crowd behavior of smart light poles based on human body joint point coordinates, which specifically includes the following steps: step S1: constructing an image training set for human behavior recognition and a human body joint point coordinate data set corresponding to human behavior; step S2: The camera of the smart light pole obtains the crowd behavior information and extracts the human skeleton behavior, and interpolates the human skeletal behavior of the missing joint points in the crowd behavior information according to the human joint point coordinate data set; Step S3: human skeleton behavior setting time attention layer , after extracting the characteristics of human skeletal behavior, construct a deep learning network and set corresponding hyperparameters; Step S4: train the deep learning network through the deep learning open source framework to obtain a crowd behavior recognition model, and use the crowd behavior recognition model to identify the newly obtained crowd. Behavioural information for identification. Compared with the prior art, the present invention has the advantages of improving the recognition rate of human behavior, reducing the influence of obstructions on the recognition result, and the like.

Description

Method and device for crowd behavior recognition of smart light poles based on human body joint point coordinates

technical field

The invention relates to the field of computer vision, in particular to a method and a device for recognizing the behavior of smart light pole crowd based on human body joint point coordinates.

Background technique

With the rapid development and application of computer vision and artificial intelligence, video analysis technology has flourished and is widely used in many fields of human life. Such as smart security, human-computer interaction, smart home and smart medical care. With the continuous improvement of video surveillance systems in urban communities in China, smart communities have also become a new model of social management innovation. As the edge intelligent hardware for future "AI+5G" smart community cloud control services, smart light poles play a key role in promoting the construction and development of smart communities. Based on the surveillance video of smart light poles, computer vision technology is applied to automatically detect crowd behaviors and automatically trigger alarms for abnormal behaviors, which will play an important role in assisting the smart community security system.

At present, the research on human behavior recognition technology mainly extracts the feature information related to human motion changes in the video sample data, then integrates the static and dynamic features of the human body, and uses the deep learning method to construct a deep spatiotemporal neural network. To realize human behavior recognition of video information. However, since the deep learning network is very dependent on the amount of sample data, and the training time and period are long, the required hardware cost is high. In addition, directly extracting features from the original video will inevitably bring in redundant information, resulting in an insufficiently detailed expression of the description behavior.

The prior art discloses a video-based pedestrian and crowd behavior recognition method, which learns point-level features of each joint, and then treats the features of each joint as a channel of a convolutional layer to learn hierarchical co-occurrence features. In the joint network structure of behavior recognition, the multi-part limb network features are fused into the motion features of a single pedestrian to enhance the behavior recognition of a single pedestrian, but the timeliness of each skeleton action and the speed of the running time are not considered, resulting in a certain error in the recognition results. , the accuracy is not high.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a smart light pole crowd behavior based on the coordinates of human body joint points in order to overcome the defects of the above-mentioned prior art, such as direct feature extraction of the original video, too much redundant information, resulting in insufficiently precise expression of the description behavior. Identification method and apparatus.

The object of the present invention can be realized through the following technical solutions:

A method for identifying crowd behaviors of smart light poles based on the coordinates of human joint points, which specifically includes the following steps:

Step S1: constructing an image training set for human behavior recognition and a human body joint point coordinate dataset corresponding to human behavior;

Step S2: the camera of the smart light pole obtains crowd behavior information and extracts human skeleton behavior, and performs interpolation on the human skeleton behavior with missing joint points in the crowd behavior information according to the human body joint point coordinate data set;

Step S3: setting a time attention layer for the human skeletal behavior, extracting features of the human skeletal behavior to construct a deep learning network and setting hyperparameters of the deep learning network;

Step S4: Train the deep learning network through the deep learning open source framework to obtain a crowd behavior recognition model, and identify the newly acquired crowd behavior information through the crowd behavior recognition model.

The image training set is an open-source behavior recognition data set or a data set recorded by frames of video data captured by the camera of the smart light pole.

The image training set extracts 18 joint point coordinate information of each human skeleton behavior through the pose estimation framework, and after noise processing is performed by setting the probability value of randomly discarding joint points, the human joint point coordinate data set is formed.

Further, the process of interpolating the human skeleton behavior with missing joint points in the crowd behavior information in the step S2 specifically includes:

Step S201: extracting the coordinate information of 18 joint points of the complete human skeleton behavior of the joint points, and calculating the Euclidean distance and angle of the remaining 17 joint points relative to the neck joint point and the height of the complete human body;

Step S202: Calculate the human body height of the missing joint point corresponding to the human skeleton behavior of the missing joint point, and perform the same proportional scaling according to the scale factor between the complete human body height and the human body height of the missing joint point, and calculate the human skeletal behavior of the missing joint point. The joint point coordinates are interpolated.

The human body joint point coordinate data set includes a recognition training set and a recognition test set, and the division method includes a leave-out method and a k-fold cross-validation method.

The set aside method directly divides the human body joint point coordinate data set D into two sets of uniformly distributed and mutually exclusive data, one of which is used as the recognition training set S, and the other is used as the recognition test set T, that is, D=S∪T ,

然后，每次用k-1个子集的并集作为识别训练集，余下的那个子集作为识别测试集。The k-fold cross-validation method first divides the human body joint point coordinate data set D into k mutually exclusive subsets with similar sizes and uniform data distribution, namely D=D ₁ ∪D ₂ ∪...∪D _k ,

Then, each time the union of k-1 subsets is used as the recognition training set, and the remaining subset is used as the recognition test set.

The temporal attention layer includes multiple time steps, and the calculation formula of the activation value of the temporal attention layer is as follows:

为上一个时间步LSTM单元的输出值，β_t为当前时间步的时间注意力层激活值，ReLU为线性整流函数。Among them, x _t is the input of the current time step, x _t-1 is the input of the previous time step, w _{x ~} and b _~ are the weights and biases of the fully connected layer at the current time step,

is the output value of the LSTM unit at the previous time step, β _t is the activation value of the temporal attention layer at the current time step, and ReLU is a linear rectification function.

Further, the activation value of the time attention layer and the output of the deep learning network are weighted and integrated, and participate in the classification of the classification layer of the crowd behavior recognition model. The specific calculation method is as follows:

Among them, z _t is the output value of the deep learning network, β _t is the activation value of the temporal attention layer, T is the total video sequence length, and o is the weighted synthesis of the output value of the deep learning network and the activation value of the temporal attention layer.

Further, the prediction result probability value of the classification layer of the crowd behavior recognition model is specifically:

Among them, i=1,...C, C is the total number of categories of human skeletal behaviors, o _i is the weighted synthesis of the deep learning network output and the activation value of the temporal attention layer, and X is a specific human behavior.

The features of the human skeleton behavior include joint point set distance features, slow motion global features and fast motion global features.

关节点集合距离特征的计算公式具体如下：The joint point coordinates are two-dimensional coordinates, which are expressed as

The calculation formula of the distance feature of the joint point set is as follows:

为在第k帧时两个关节点

和

的欧式距离，N为人体关节点总数，JCD^k为在第k帧时关节点集合距离特征。in,

is the two joint points at the kth frame

and

The Euclidean distance of , N is the total number of joint points of the human body, and JCD ^k is the distance feature of the joint point set at the kth frame.

The calculation formulas of the slow motion global feature and the fast motion global feature are as follows:

和

分别表示在第k帧时的慢速和快速运动的全局运动特征，S^k为第k帧的人体关节点坐标集合，S^k+1和S^k+2分别表示第k帧之后的后1帧和后2帧的人体关节点坐标集合，K为帧总数。in,

and

Represents the global motion features of slow and fast motion at the kth frame, respectively, ^Sk is the coordinate set of human body joints in the kth frame, and ^Sk+1 and ^Sk+2 represent the next frame after the kth frame, respectively. and the coordinate set of human body joints in the next 2 frames, K is the total number of frames.

In the deep learning open source framework, the distance feature of the joint point set, the slow motion global feature and the fast motion global feature are extended into a one-dimensional vector form, connected and fused through the convolution layer function and the deep learning network, and the fused features are Input into the multi-layer perceptron classifier of the deep learning network, and directly perform softmax classification at the last layer of the network layer.

The recognition result of the crowd behavior recognition model in the step S4 is the classification label of human skeletal behavior.

The hyperparameters include the number of network layers, the number of hidden layer nodes, convolution kernel parameters, training batch size (Batch_size), learning rate (learning_rate), Adam optimizer parameters (β ₁ and β ₂ ) and training batch period (Epochs).

A device for using the method for recognizing the behavior of smart light pole crowd based on the coordinates of human body joints, comprising a memory and a processor, the recognition method is stored in the memory in the form of a computer program, and is executed by the processor, and is realized when executed. The following steps:

Step S1: constructing an image training set for human behavior recognition and a human body joint point coordinate dataset corresponding to human behavior;

Step S2: the camera of the smart light pole obtains crowd behavior information and extracts human skeleton behavior, and performs interpolation on the human skeleton behavior with missing joint points in the crowd behavior information according to the human body joint point coordinate data set;

Step S3: setting a time attention layer for the human skeletal behavior, extracting features of the human skeletal behavior to construct a deep learning network and setting hyperparameters of the deep learning network;

Step S4: Train the deep learning network through the deep learning open source framework to obtain a crowd behavior recognition model, and identify the newly acquired crowd behavior information through the crowd behavior recognition model.

Compared with the prior art, the present invention has the following beneficial effects:

1. The present invention performs human behavior recognition by extracting the joint point set distance feature, slow motion global feature and fast motion global feature of the human body joint point coordinate information, and fully considers the static and dynamic features of human behavior changes.

2. The present invention adds a temporal attention layer, and sets importance parameters for the human skeleton joint points of each frame, so that the image of the key frame is more prominent. According to the principle of the soft attention mechanism, it is different from simply extracting the key frame and ignoring it. For other frame algorithms, the results of the deep learning network and the temporal attention layer are weighted and integrated into the classifier for prediction, so that the deep learning network can perform network reasoning faster and describe the changes of human behavior and movement more accurately.

3. The present invention sets random joint points to discard probability values to enhance data samples, and has better robustness to the problem of partial occlusion of joint points in everyday people.

Description of drawings

Fig. 1 is the structural block diagram of the present invention;

FIG. 2 is a schematic flow chart of the present invention.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and provides a detailed implementation manner and a specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

As shown in Figure 2, a method for recognizing crowd behavior of smart light poles based on the coordinates of human joint points includes the following steps:

Step S1: constructing an image training set for human behavior recognition and a human body joint point coordinate dataset corresponding to human behavior;

Step S2: the camera of the smart light pole obtains the crowd behavior information and extracts the human skeleton behavior, and interpolates the human skeleton behavior of the missing joint points in the crowd behavior information according to the human body joint point coordinate data set;

Step S3: setting a temporal attention layer for human skeletal behavior, extracting features of human skeletal behavior to construct a deep learning network and setting hyperparameters of the deep learning network;

Step S4: train the deep learning network through the deep learning open source framework, obtain a crowd behavior recognition model, and identify the newly acquired crowd behavior information through the crowd behavior recognition model.

The image training set is an open source behavior recognition data set or a data set recorded by frames of video data captured by a camera of a smart light pole.

Behavior recognition datasets include UCF dataset, CUHK dataset, UMN dataset, JHMDB dataset or SHREC dataset.

The image training set extracts the coordinate information of 18 joint points of each human skeleton behavior through the pose estimation framework. After noise processing is performed by setting the probability value of randomly discarding joint points, a human joint point coordinate data set is formed.

The process of interpolating the human skeleton behavior of missing joint points in the crowd behavior information in step S2 specifically includes:

Step S201: extracting the coordinate information of 18 joint points of the complete human skeleton behavior of the joint points, and calculating the Euclidean distance and angle of the remaining 17 joint points relative to the neck joint point and the height of the complete human body;

Step S202: Calculate the human body height of the missing joint point corresponding to the human skeleton behavior of the missing joint point, and perform the same proportional scaling according to the scale factor between the complete human body height and the human body height of the missing joint point, and calculate the human skeletal behavior of the missing joint point. The joint point coordinates are interpolated.

The human joint point coordinate data set includes recognition training set and recognition test set, and the division methods include leave-out method and k-fold cross-validation method.

留出法需要保持数据分布的一致性，比如对数据集做分层采样，将每一个行为类别对应的人体关节点坐标数据集70％的样本数据作为识别训练集，剩余30％的样本数据作为识别测试集。The set aside method directly divides the human body joint point coordinate data set D into two sets of uniformly distributed and mutually exclusive data, one of which is used as the recognition training set S, and the other is used as the recognition test set T, that is, D=S∪T,

The set aside method needs to maintain the consistency of the data distribution, such as stratified sampling of the data set, 70% of the sample data of the human body joint point coordinate data set corresponding to each behavior category is used as the recognition training set, and the remaining 30% of the sample data is used as the recognition training set. Identify the test set.

然后，每次用k-1个子集的并集作为识别训练集，余下的那个子集作为识别测试集。k折交叉验证法可以获得k组识别训练集和识别测试集，从而可进行k次训练和测试，最终返回k个测试结果的均值。The k-fold cross-validation method first divides the human body joint point coordinate data set D into k mutually exclusive subsets with similar sizes and uniform data distribution, namely D=D ₁ ∪D ₂ ∪...∪D _k ,

Then, each time the union of k-1 subsets is used as the recognition training set, and the remaining subset is used as the recognition test set. The k-fold cross-validation method can obtain k groups of identification training sets and identification test sets, so that k times of training and testing can be performed, and finally the mean value of k test results is returned.

The temporal attention layer includes multiple time steps, and the calculation formula of the activation value of the temporal attention layer is as follows:

为上一个时间步LSTM单元的输出值，β_t为当前时间步的时间注意力层激活值，ReLU为线性整流函数。Among them, x _t is the input of the current time step, x _t-1 is the input of the previous time step, w _{x ~} and b _~ are the weights and biases of the fully connected layer at the current time step,

is the output value of the LSTM unit at the previous time step, β _t is the activation value of the temporal attention layer at the current time step, and ReLU is a linear rectification function.

The activation value of the temporal attention layer and the output of the deep learning network are weighted and integrated, and jointly participate in the classification layer of the crowd behavior recognition model for classification. The specific calculation method is as follows:

Among them, z _t is the output value of the deep learning network, β _t is the activation value of the temporal attention layer, T is the total video sequence length, and o is the weighted synthesis of the output value of the deep learning network and the activation value of the temporal attention layer.

The prediction result probability value of the classification layer of the crowd behavior recognition model is specifically:

Among them, i=1,...C, C is the total number of categories of human skeletal behaviors, o _i is the weighted synthesis of the deep learning network output and the activation value of the temporal attention layer, and X is a specific human behavior.

The features of human skeletal behavior include joint point set distance features, slow motion global features and fast motion global features.

关节点集合距离特征的计算公式具体如下：The joint point coordinates are two-dimensional coordinates, which are expressed as

The calculation formula of the distance feature of the joint point set is as follows:

为在第k帧时两个关节点

和

的欧式距离，N为人体关节点总数，JCD^k为在第k帧时关节点集合距离特征。in,

is the two joint points at the kth frame

and

The Euclidean distance of , N is the total number of joint points of the human body, and JCD ^k is the distance feature of the joint point set at the kth frame.

The calculation formulas of the slow motion global feature and the fast motion global feature are as follows:

和

分别表示在第k帧时的慢速和快速运动的全局运动特征，S^k为第k帧的人体关节点坐标集合，S^k+1和S^k+2分别表示第k帧之后的后1帧和后2帧的人体关节点坐标集合，K为帧总数。in,

and

Represents the global motion features of slow and fast motion at the kth frame, respectively, ^Sk is the coordinate set of human body joints in the kth frame, and ^Sk+1 and ^Sk+2 represent the next frame after the kth frame, respectively. and the coordinate set of human body joints in the next 2 frames, K is the total number of frames.

和

维度变换成

和

As shown in Figure 1, the deep learning open source framework is the Tensorflow framework, and the API is Keras. In the Tensorflow deep learning open source framework, the joint point set distance feature, slow motion global feature and fast motion global feature are extended into a one-dimensional vector form. The joint point set distance feature (JCD) is dimensionally matched with the dual-scale global motion features of slow and fast motion. Using the idea of linear interpolation, the dual-scale global motion features of slow and fast motion are combined.

and

transform the dimension into

and

The convolution layer function and the deep learning network are used for connection and fusion. The specific fusion process is as follows:

和

为关节点集合距离特征、慢速运动全局特征和快速运动全局特征的特征值，ε^k为融合后的特征值，

为融合运算操作。in,

and

is the eigenvalue of the distance feature of the joint point set, the global feature of slow motion and the global feature of fast motion, ^εk is the eigenvalue after fusion,

For the fusion operation.

The fused features are input into the multi-layer perceptron classifier of the deep learning network, and the softmax classification is directly performed in the last layer of the network layer.

The training optimizer used by the crowd behavior recognition model is Adam, and the loss function used is the Cross Entropy loss function with a regularization term. The specific calculation formula is as follows:

where y _i represents the target value, y _i _ represents the actual output value, n is the total number of action categories, λ ₁ and λ ₂ are the weights, T is the total number of video frames, β _t is the temporal attention layer activation value, and W is the number of Layer perceptron classifier weights.

Step S4 The process of training the deep learning network by the deep learning open source framework includes obtaining the behavior classification error according to the cross entropy loss function, starting the Adam optimizer training, and back-propagating the behavior classification error to update the weights until the cross entropy loss function. Convergence or complete all training batch cycles, and output the corresponding crowd behavior recognition model at this time.

The recognition result of the crowd behavior recognition model in step S4 is the classification label of human skeleton behavior.

Hyperparameters include the number of network layers, the number of hidden layer nodes, convolution kernel parameters, training batch size (Batch_size), learning rate (learning_rate), Adam optimizer parameters (β ₁ and β ₂ ) and training batch period (Epochs ).

The analysis of the crowd behavior recognition model includes drawing the model training accuracy curve and test accuracy curve to judge the network performance and drawing the model test ROC curve or confusion matrix to evaluate the generalization ability of the model. The network performance includes calculation speed, recognition accuracy and Whether to overfit or underfit.

A device for using a method for recognizing behaviors of smart light pole crowds based on the coordinates of human joint points, comprising a memory and a processor, the recognition method is stored in the memory in the form of a computer program, and is executed by the processor, and the following steps are implemented during execution:

Step S1: constructing an image training set for human behavior recognition and a human body joint point coordinate dataset corresponding to human behavior;

Step S2: the camera of the smart light pole obtains the crowd behavior information and extracts the human skeleton behavior, and interpolates the human skeleton behavior of the missing joint points in the crowd behavior information according to the human body joint point coordinate data set;

Step S3: setting a temporal attention layer for human skeletal behavior, extracting features of human skeletal behavior to construct a deep learning network and setting hyperparameters of the deep learning network;

Step S4: train the deep learning network through the deep learning open source framework, obtain a crowd behavior recognition model, and identify the newly acquired crowd behavior information through the crowd behavior recognition model.

In addition, it should be noted that the names of the specific embodiments described in this specification may be different, and the above content described in this specification is only an example to illustrate the structure of the present invention. All equivalent changes or simple changes made according to the structures, features and principles of the present invention are included in the protection scope of the present invention. Those skilled in the art to which the present invention pertains can make various modifications or additions to the specific examples described or adopt similar methods, as long as they do not deviate from the structure of the present invention or go beyond the scope defined by the claims, all It belongs to the protection scope of the present invention.

Claims (10)

1. a method for identifying crowd behaviors of smart light poles based on human body joint point coordinates, is characterized in that, specifically comprises the following steps: Step S1: constructing an image training set for human behavior recognition and a human body joint point coordinate dataset corresponding to human behavior; Step S2: the camera of the smart light pole obtains crowd behavior information and extracts human skeleton behavior, and performs interpolation on the human skeleton behavior with missing joint points in the crowd behavior information according to the human body joint point coordinate data set; Step S3: setting a time attention layer for the human skeletal behavior, extracting features of the human skeletal behavior to construct a deep learning network and setting hyperparameters of the deep learning network; Step S4: Train the deep learning network through the deep learning open source framework to obtain a crowd behavior recognition model, and identify the newly acquired crowd behavior information through the crowd behavior recognition model. 2. The method for recognizing crowd behavior of smart light poles based on human body joint point coordinates according to claim 1, wherein the image training set is an open-source behavior recognition data set or the camera of the smart light pole. A dataset of video data recorded in frames. 3. a kind of intelligent light pole crowd behavior recognition method based on human body joint point coordinates according to claim 1, is characterized in that, described image training set extracts 18 joint points of each human skeleton behavior through pose estimation frame The coordinate information is processed by noise to form the coordinate data set of the joint points of the human body. 4. The method for recognizing crowd behavior of smart light poles based on human body joint point coordinates according to claim 3, wherein in the step S2, the human skeleton behavior with missing joint points in the crowd behavior information is interpolated. The replenishment process specifically includes: Step S201: extracting the coordinate information of 18 joint points of the complete human skeleton behavior of the joint points, and calculating the Euclidean distance and angle of the remaining 17 joint points relative to the neck joint point and the height of the complete human body; Step S202: Calculate the human body height of the missing joint point corresponding to the human skeleton behavior of the missing joint point, and perform the same proportional scaling according to the scale factor between the complete human body height and the human body height of the missing joint point, and calculate the human skeletal behavior of the missing joint point. The joint point coordinates are interpolated. 5 . The method for recognizing crowd behavior of smart light poles based on human body joint point coordinates according to claim 1 , wherein the human body joint point coordinate data set includes a recognition training set and a recognition test set. 6 . 6. The method for recognizing crowd behavior of smart light poles based on human body joint point coordinates according to claim 1, wherein the time attention layer comprises a plurality of time steps, and the calculation of the activation value of the time attention layer The formula is as follows:

Among them, x _t is the input of the current time step, x _t-1 is the input of the previous time step, w _{x ~} and b _~ are the weights and biases of the fully connected layer at the current time step,

is the output value of the LSTM unit at the previous time step, β _t is the activation value of the temporal attention layer at the current time step, and ReLU is a linear rectification function. 7. A method for recognizing crowd behavior of smart light poles based on human body joint point coordinates according to claim 1, wherein the activation value of the time attention layer and the output of the deep learning network are weighted and integrated to jointly participate in the crowd The classification layer of the behavior recognition model is classified, and the specific calculation method is as follows:

Among them, z _t is the output value of the deep learning network, β _t is the activation value of the temporal attention layer, T is the total video sequence length, and o is the weighted synthesis of the output value of the deep learning network and the activation value of the temporal attention layer. 8. The method for recognizing crowd behavior of smart light poles based on human body joint point coordinates according to claim 1, wherein the predicted result probability value of the classification layer of the crowd behavior recognition model is specifically:

Among them, i=1,...C, C is the total number of categories of human skeletal behaviors, o _i is the weighted synthesis of the deep learning network output and the activation value of the temporal attention layer, and X is a specific human behavior. 9 . The method for recognizing crowd behavior of smart light poles based on human joint point coordinates according to claim 1 , wherein the characteristics of the human skeleton behavior include joint point set distance features, slow motion global features and fast motion features. 10 . Motion global features. 10. A device using any one of claims 1-9 based on the human body joint point coordinates of the smart light pole crowd behavior identification method, characterized in that, comprising a memory and a processor, the identification method is in the form of a computer program Stored in the memory and executed by the processor, the following steps are implemented during execution: Step S1: constructing an image training set for human behavior recognition and a human body joint point coordinate dataset corresponding to human behavior; Step S2: the camera of the smart light pole obtains crowd behavior information and extracts human skeleton behavior, and performs interpolation on the human skeleton behavior with missing joint points in the crowd behavior information according to the human body joint point coordinate data set; Step S3: setting a temporal attention layer for the human skeletal behavior, extracting features of the human skeletal behavior to construct a deep learning network and setting hyperparameters of the deep learning network; Step S4: Train the deep learning network through the deep learning open source framework, obtain a crowd behavior recognition model, and identify the newly acquired crowd behavior information through the crowd behavior recognition model.

Images