CN111310668A - A gait recognition method based on skeleton information - Google Patents

Abstract

The present invention provides a gait recognition method based on skeleton information, including: collecting gait video sequences; using OpenPose to perform posture estimation on the gait video sequences to obtain gait key point sequences; constructing spatiotemporal skeleton sequences; inputting the adjacency matrix and the gait key point sequence into a multi-scale spatiotemporal graph convolutional network for training; after the training is completed, using the trained model for testing, extracting gait features, and performing feature matching. The present invention mainly adopts the form of human key points, introduces a graph convolutional neural network for graph structure, and improves the connection method and partitioning strategy. The network adopts a twin mechanism, combines cross entropy loss and contrast loss, and integrates the shallow features, middle features and deep features of the network, which improves the robustness of gait recognition to a certain extent.

Description

A gait recognition method based on skeleton information

technical field

The present invention relates to the technical field of pattern recognition, in particular, to a gait recognition method based on skeleton information.

Background technique

Gait recognition is an emerging biometric recognition technology, which aims to identify people by walking gestures in video sequences. Compared with other biometric technologies, gait recognition has the advantages of non-contact, long-distance and not easy to camouflage. It has more advantages in the field of security and intelligent monitoring. So far, there are still some problems in the application of gait recognition in the actual complex environment.

In recent years, domestic and foreign academic research institutions have paid more and more attention to gait recognition technology. The existing technologies are mainly divided into the following two types:

1. Model-based approach. It is mainly to divide the body into multiple pieces or obtain body joint points, and fit the joint points or part of the motion trajectory of the body movement. This type of method mainly relies on the static characteristics of each block of the body and the motion trajectories of joint points for modeling, including two-dimensional models and three-dimensional models.

2. Non-model-based methods. It is mainly realized by collecting the shape of human walking and gait features such as characteristic parameters, and does not need to reconstruct the model of walking gait. There are roughly three kinds of research objects: gait energy map, contour map sequence and human body key point sequence.

In recent years, with the great progress of deep learning technology in various fields of computer vision, a large number of gait recognition methods based on deep learning have emerged. For example, gait energy maps are used to describe gait sequences, and deep convolutional neural networks are used to train matching models to match human identities. Repeatedly select any two perspectives from all perspectives to match and train the convolutional neural network. This method can obtain better multi-view gait recognition performance. The disadvantage is that when the human walking perspective changes in a large range, all the The extracted multi-view gait features have insufficient representation ability, and are less robust to clothing and carrying objects. Furthermore, gesture information is used for gait feature representation and matching. The pose estimation algorithm is used to obtain the coordinates of human body key points from the gait video sequence, and the gait key point sequence is trained by using convolutional neural network and long-short-term memory network. At the same time, manual features are introduced for gait recognition. However, the existing gait recognition technology still has the following shortcomings:

1. Gait recognition through gait contour map or gait energy map requires high contour quality and background, and is greatly affected by lighting conditions and complex backgrounds, which often leads to incomplete contour map extraction.

2. Affected by covariates such as clothing and carried objects, the covariates cannot be separated from the human body itself, resulting in reduced recognition accuracy.

SUMMARY OF THE INVENTION

According to the technical problem raised above, a gait recognition method based on skeleton information is provided. The present invention mainly adopts the form of human key points, introduces a graph convolutional neural network for graph structure, and improves the connection method and division strategy. The middle-level features and deep-level features improve the robustness of gait recognition to a certain extent.

The technical means adopted in the present invention are as follows:

A gait recognition method based on skeleton information, comprising the following steps:

S1. Collect gait video sequences;

S2. Use OpenPose to perform pose estimation on the above gait video sequence to obtain a sequence of gait key points;

S3, constructing a spatiotemporal skeleton sequence;

S4. Input the adjacency matrix and the gait keypoint sequence into the multi-scale spatiotemporal graph convolutional network for training;

S5. After the training is completed, use the trained model for testing, extract gait features, and perform feature matching.

Further, the step S3 is specifically:

S31. Perform the natural connection of the gait key point sequence in space; at the same time, introduce symmetry to connect the symmetrical joint points (only connect the symmetrical key points of the legs, because the symmetry between the arms is missing under the condition of carrying objects) ); in time, connect the same key points between frames;

S32. Define the sampling function, and the neighborhood set of a node v _ti is defined as:

B(v _ti )={v _tj |d(v _tj ,v _ti )≤D},

Among them, B is the neighborhood set of the node v _ti ; v represents the node; D represents the distance; d(v _tj , v _ti ) represents the shortest path between two nodes, usually D=1; therefore, the sampling function is defined is: p(v _tj , v _ti )=v _tj ;

S33. Select a division strategy, and divide the neighborhood set into four subsets, that is, the node itself is the first subset, and among the asymmetric nodes, the second subset is closer to the center of gravity than the node itself, and the distance to the center of gravity is the second subset. The third subset is farther than the node itself, and the symmetric node is defined as the fourth subset, namely:

S34. Define a weight function. After the neighborhood set is divided into four subsets, each subset has a digital label, and a mapping function l _ti is used to map each node to its subset label. The mapping function is defined as: B( v _ti )→{0,...,K-1}, K=4; the weight function is defined as: w(v _ti ,v _tj )=w'(l _ti (v _tj ));

S35. Extend the spatial graph convolution to the space-time domain, and define the space-time neighborhood set as: B(v _ti )={v _qj |d(v _tj ,v _ti )≤K, |q–t|≤Γ }}, B is the neighborhood set of the node v _ti ; v represents the node; K represents the distance; Γ controls the range of the graph included in the neighborhood, that is, the temporal convolution kernel.

Further, the process of training the multi-scale spatiotemporal graph convolutional neural network is as follows:

S41, after selecting the sample, randomly select a sample from all samples whose ID is the same as the selected sample as a positive sample, and randomly select a sample from all samples whose ID is different from the selected sample as a negative sample;

S42, adopting the twinning mechanism, in one iteration, the selected sample is input into branch 1, the positive sample and the negative sample are input into branch 2 in turn, and branch 1 and branch 2 share parameters;

S43, using SoftMax and cross entropy loss function to classify the selected sample features in branch 1;

S44, using a contrast loss function to compare the features of the selected samples and the positive samples, as well as the features of the selected samples and the negative samples; samples from the same ID have a label of 1, and samples from different IDs have a label of 1 0.

S45, the sum of the two losses, the total loss is:

Loss=Lid+0.5*[Lc(sample,pos,1)+Lc(sample,neg,0)],

Among them, Lid is the cross entropy loss, Lc is the contrast loss, and then backpropagation is performed to update the network.

Further, in the step S4, the following setting process is also included when training the multi-scale spatiotemporal graph convolutional neural network:

Step 1. Input the gait sequence, the dimension is [3, 100, 18], of which 3 is the input key point feature with 3 channels, namely X, Y coordinates and confidence C, 100 is the time dimension has 100 frames, 18 is each There are 18 key points in the frame;

Step 2. The first three layers output 64 channels, the convolution kernel size is (9, 3), 9 is the time convolution kernel size, 3 is the spatial convolution kernel size, and the output dimension is [64, 100, 18];

Step 3. The middle three layers output 128 channels, the convolution kernel size is (9,3), the output dimension is [128,50,18], and in the fourth layer, the time dimension convolution step size is 2;

Step 4. The last three layers output 256 channels, the convolution kernel size is (9,3), the output dimension is [256,25,18], and in the seventh layer, the time dimension convolution step size is 2;

Step 5. Perform global average pooling. After pooling, the feature dimension becomes 256 dimensions;

Step 6. After the features [64, 100, 18] output by the first layer are exchanged for dimensions, average pooling is performed to turn them into 18-dimensional features;

Step 7. After the output features of the fifth layer [128, 50, 18] are exchanged for dimensions, average pooling is performed to turn them into 18-dimensional features;

Step 8. The gait feature is represented by the fusion of shallow features and deep features, and the 18-dimensional features of the first layer, the 18-dimensional features of the fifth layer and the 256-dimensional features of the last layer are spliced into 292-dimensional features. feature;

Step 9. Use the SoftMax classifier to classify the 292-dimensional features.

The present invention also provides a test method for the gait recognition method based on skeleton information, comprising the following steps:

Step 1: Input the sequence of gait key points to be tested;

Step II: Use the trained network to extract gait features, and perform two-norm normalization on the features;

Step III: Perform the operations of Step I and Step II on the samples in the sample database, and the feature vector can be used to represent the pedestrian gait sequence to be retrieved and the pedestrian gait sequence in the retrieval database;

Step IV: Calculate the distance between the pedestrian gait sequence to be retrieved and the pedestrian gait sequence in the retrieval database, that is, for a pedestrian gait sequence to be retrieved, calculate the distance between its features and all the pedestrian gait sequences in the retrieval database;

Step V: Sort the similarity of the samples in the retrieval database from small to large according to the distance calculated above. The higher the distance, the more likely it is consistent with the ID of the pedestrian to be retrieved.

Compared with the prior art, the present invention has the following advantages:

1. The gait recognition method provided by the present invention, aiming at the influence of covariates such as backpacks and clothing, adopts the method of characterizing the gait by the sequence of key points, and solves the problem that the gait characteristics represented by the contour map and the energy map are ineffective under the condition of covariates. The problem of low stickiness improves the accuracy of identification under the influence of covariates.

2. The gait recognition method provided by the present invention, aiming at the unique symmetry characteristics of the gait, introduces symmetry when constructing the space-time skeleton sequence, and adds the symmetrical joint point information of the human body to the adjacency matrix, which enhances the correlation degree of the relevant nodes. , while reducing the noise caused by inaccurate joint estimation and improving the recognition accuracy.

3. Since the deep convolutional neural network extracts high-level features, it can only represent high-level semantic information and cannot describe static features. Therefore, a multi-scale method that combines shallow features, mid-level features and deep features is adopted to enrich the expression of gait features. form, which improves the accuracy of recognition.

Based on the above reasons, the present invention can be widely promoted in the fields of pattern recognition and the like.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

Fig. 1 is the flow chart of the method of the present invention.

FIG. 2 is a schematic diagram of a specific setting of a multi-scale spatiotemporal graph convolutional neural network according to the present invention.

Figure 3 is a schematic diagram of the testing method of the present invention.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only Embodiments are part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

As shown in Figure 1, the present invention provides a gait recognition method based on skeleton information, comprising the following steps:

S1. Collect gait video sequences;

S2. Use OpenPose to perform pose estimation on the above gait video sequence to obtain a sequence of gait key points;

S3, constructing a spatiotemporal skeleton sequence;

Further, as a preferred embodiment of the present invention, the step S3 is specifically:

S31. Perform the natural connection of the gait key point sequence on the human body in space; because the walking posture of a person has the characteristics of symmetry, symmetry is introduced to connect the symmetrical joint points at the same time (only the symmetrical key points of the legs are connected, because Loss of symmetry between the arms in the carrying condition); temporally, connect the same key points from frame to frame;

S32. Define the sampling function, and the neighborhood set of a node v _ti is defined as:

B(v _ti )={v _tj |d(v _tj ,v _ti )≤D},

Among them, B is the neighborhood set of the node v _ti ; v represents the node; D represents the distance; d(v _tj , v _ti ) represents the shortest path between two nodes, usually D=1; therefore, the sampling function is defined is: p(v _tj , v _ti )=v _tj ;

S33. Select a division strategy, and divide the neighborhood set into four subsets, that is, the node itself is the first subset, and among the asymmetric nodes, the second subset is closer to the center of gravity than the node itself, and the distance to the center of gravity is the second subset. The third subset is farther than the node itself, and the symmetric node is defined as the fourth subset, namely:

S34. Define a weight function. After the neighborhood set is divided into four subsets, each subset has a digital label, and a mapping function l _ti is used to map each node to its subset label. The mapping function is defined as: B( v _ti )→{0,...,K-1}, K=4; the weight function is defined as: w(v _ti ,v _tj )=w'(l _ti (v _tj ));

S35. Extend the spatial graph convolution to the space-time domain, and define the space-time neighborhood set as: B(v _ti )={v _qj |d(v _tj ,v _ti )≤K, |q–t|≤Γ }}, B is the neighborhood set of the node v _ti ; v represents the node; K represents the distance; Γ controls the range of the graph included in the neighborhood, that is, the temporal convolution kernel.

S4. Input the adjacency matrix and the gait keypoint sequence into the multi-scale spatiotemporal graph convolutional network for training;

Further, as a preferred embodiment of the present invention, as shown in FIG. 2 , the step S4 further includes the following setting process before training the multi-scale spatiotemporal graph convolutional neural network:

Step 1. Input the gait sequence, the dimension is [3, 100, 18], of which 3 is the input key point feature with 3 channels, namely X, Y coordinates and confidence C, 100 is the time dimension has 100 frames, 18 is each There are 18 key points in the frame;

Step 2. The first three layers output 64 channels, the convolution kernel size is (9, 3), 9 is the time convolution kernel size, 3 is the spatial convolution kernel size, and the output dimension is [64, 100, 18];

Step 3. The middle three layers output 128 channels, the convolution kernel size is (9,3), the output dimension is [128,50,18], and in the fourth layer, the time dimension convolution step size is 2;

Step 4. The last three layers output 256 channels, the convolution kernel size is (9,3), the output dimension is [256,25,18], and in the seventh layer, the time dimension convolution step size is 2;

Step 5. Perform global average pooling. After pooling, the feature dimension becomes 256 dimensions;

Step 6. After the features [64, 100, 18] output by the first layer are exchanged for dimensions, average pooling is performed to turn them into 18-dimensional features;

Step 7. After the output features of the fifth layer [128, 50, 18] are exchanged for dimensions, average pooling is performed to turn them into 18-dimensional features;

Step 8. Since the deep convolutional neural network extracts high-level features, it represents high-level semantic information alone and cannot describe static features. Therefore, the gait features are represented by fusing shallow features and deep features. The features, the 18-dimensional features of the fifth layer and the 256-dimensional features of the last layer are spliced into 292-dimensional features;

Step 9. Use the SoftMax classifier to classify the 292-dimensional features.

In this embodiment, the CASIA-B dataset is used, NM: normal walking condition, BG: carrying condition, CL: wearing coat condition. As shown in the table below:

The specific process of training a multi-scale spatiotemporal graph convolutional neural network is as follows:

In the training phase, the purpose is to train the network so that it can extract features that can represent pedestrians. Therefore, the network is trained in the form of classification. The specific steps are:

S41, after selecting the sample, randomly select a sample from all samples whose ID is the same as the selected sample as a positive sample, and randomly select a sample from all samples whose ID is different from the selected sample as a negative sample;

S42, adopting the twinning mechanism, in one iteration, the selected sample is input into branch 1, the positive sample and the negative sample are input into branch 2 in turn, and branch 1 and branch 2 share parameters;

S43, using SoftMax and cross entropy loss function to classify the selected sample features in branch 1;

S44, using a contrast loss function to compare the features of the selected samples and the positive samples, and the features of the selected samples and the negative samples; the samples from the same ID have a label of 1, and the samples from different IDs have a label of 1. is 0.

S45, the sum of the two losses, the total loss is:

Loss=Lid+0.5*[Lc(sample,pos,1)+Lc(sample,neg,0)],

Among them, Lid is the cross entropy loss, Lc is the contrast loss, and then backpropagation is performed to update the network.

S5. After the training is completed, use the trained model for testing, extract gait features, and perform feature matching.

As shown in Figure 3, the present invention also provides a test method for a gait recognition method based on skeleton information, comprising the following steps:

Step 1: Input the sequence of gait key points to be tested;

Step II: Use the trained network to extract gait features, and perform two-norm normalization on the features;

Step III: Perform the operations of Step I and Step II on the samples in the sample database, and the feature vector can be used to represent the pedestrian gait sequence to be retrieved and the pedestrian gait sequence in the retrieval database;

Step IV: Calculate the distance between the pedestrian gait sequence to be retrieved and the pedestrian gait sequence in the retrieval database, that is, for a pedestrian gait sequence to be retrieved, calculate the distance between its features and all the pedestrian gait sequences in the retrieval database;

Step V: Sort the similarity of the samples in the retrieval database from small to large according to the distance calculated above. The higher the distance, the more likely it is consistent with the ID of the pedestrian to be retrieved.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some or all of the technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (5)

1. A gait recognition method based on skeleton information is characterized by comprising the following steps:

s1, acquiring a gait video sequence;

s2, carrying out attitude estimation on the gait video sequence by adopting OpenPose to obtain a gait key point sequence;

s3, constructing a spatio-temporal skeleton sequence;

s4, inputting the adjacency matrix and the gait key point sequence into a multi-scale space-time graph convolution network for training;

and S5, after the training is finished, testing by using the trained model, extracting gait characteristics and carrying out characteristic matching.

2. The method for recognizing gait based on skeleton information according to claim 1, characterized in that the step S3 is specifically:

s31, carrying out human body natural connection on the gait key point sequence in space; meanwhile, symmetry is introduced to connect symmetrical joint points (only connecting symmetrical key points of the legs because of lack of symmetry between arms under the condition of carrying objects); connecting the same key points from frame to frame in time;

s32, defining a sampling function, a node v_tiIs defined as:

B(v_ti)＝{v_tj|d(v_tj,v_ti)≤D},

wherein B is node v_tiA neighborhood set of (c); v represents a node; d represents a distance; d (v)_tj,v_ti) Representing the shortest path between two nodes, usually taking D ═ 1; thus, the sampling function is defined as: p (v)_tj,v_ti)＝v_tj；

S33, selecting a partitioning strategy, partitioning the neighborhood set into four subsets, that is, the node itself is the first subset, in the asymmetric node, a second subset closer to the gravity center than the node itself, and a third subset farther from the gravity center than the node itself, where the symmetric node is defined as the fourth subset, that is:

s34, defining a weight function, dividing the neighborhood set into four subsets, each subset having a digital label, and adopting a mapping function l_tiMapping each node to its subset label, the mapping function being defined as: b (v)_ti) → {0, … …, K-1}, K ═ 4; the weight function is defined as: w (v)_ti,v_tj)＝w’(l_ti(v_tj))；

S35, expanding the space graph convolution to a space-time domain, and defining a space-time neighborhood set as: b (v)_ti)＝{v_qj|d(v_tj,v_ti) K is less than or equal to, q-t is less than or equal to gamma, and B is a node v_tiA neighborhood set of (c); v represents a node; k represents a distance; Γ controls the extent of the graph, i.e., the temporal convolution kernel, that is included in the neighborhood.

3. The method for gait recognition based on the skeleton information as claimed in claim 1, wherein the process of training the multi-scale space-time graph convolutional neural network is as follows:

s41, after the samples are selected, randomly selecting one sample from all samples with the same ID as the selected samples as a positive sample, and randomly selecting one sample from all samples with different IDs from the selected samples as a negative sample;

s42, inputting the selected sample into a branch 1, sequentially inputting a positive sample and a negative sample into a branch 2 in one iteration by adopting a twin mechanism, wherein the branch 1 and the branch 2 share parameters;

s43, classifying the selected sample characteristics in the branch 1 by adopting SoftMax and a cross entropy loss function;

s44, comparing the characteristics of the selected sample and the positive sample and the characteristics of the selected sample and the negative sample by using a contrast loss function; samples from the same ID are labeled 1, and samples from different IDs are labeled 0.

S45, adding the two part losses, wherein the total loss is as follows:

Loss＝Lid+0.5*[Lc(sample,pos,1)+Lc(sample,neg,0)]，

and performing back propagation to update the network, wherein Lid is cross entropy loss, and Lc is contrast loss.

4. A gait recognition method based on skeleton information according to claim 3, characterized in that said step S4 further includes the following setting process when training the multi-scale space-time graph convolutional neural network:

step 1, inputting a gait sequence, wherein the dimensionality is [3,100,18], 3 is that input key point characteristics have 3 channels which are respectively X, Y coordinates and confidence C, 100 is that the time dimensionality has 100 frames, and 18 is that each frame has 18 key points;

step 2, outputting 64 channels from the first three layers, wherein the convolution kernel size is (9,3), 9 is the time convolution kernel size, 3 is the space convolution kernel size, and the output dimension is [64,100,18 ];

step 3, outputting 128 channels in the middle three layers, wherein the convolution kernel size is (9,3), the output dimensionality is [128,50,18], and in the fourth layer, the time dimension convolution step size is 2;

step 4, outputting 256 channels in the last three layers, wherein the convolution kernel size is (9,3), the output dimensionality is [256,25,18], and in the seventh layer, the time dimension convolution step size is 2;

step 5, performing global average pooling, wherein after the pooling is performed, the characteristic dimension is changed into 256 dimensions;

step 6, carrying out dimension exchange on the features [64,100 and 18] output by the first layer, and then carrying out average pooling to obtain 18-dimensional features;

step 7, carrying out dimension exchange on the output features [128,50 and 18] of the fifth layer, and then carrying out average pooling to obtain 18-dimensional features;

step 8, representing the gait characteristics by fusing the shallow layer characteristics, the middle layer characteristics and the deep layer characteristics, and splicing the 18-dimensional characteristics of the first layer, the 18-dimensional characteristics of the fifth layer and the 256-dimensional characteristics of the last layer to obtain 292-dimensional characteristics;

and 9, classifying the 292-dimensional features by adopting a SoftMax classifier.

5. A gait recognition method testing method based on skeleton information is characterized by comprising the following steps:

step I: inputting a gait key point sequence to be tested;

step II: extracting gait features by using the trained network, and carrying out two-norm normalization on the features;

step III: carrying out the operations of the step I and the step II on the samples in the sample library, and representing the gait sequence of the pedestrian to be searched and the gait sequence of the pedestrian in the search library by using the characteristic vector;

step IV: calculating the distance between the pedestrian gait sequence to be retrieved and the pedestrian gait sequence in the retrieval library, namely calculating the distance between the characteristics of the pedestrian gait sequence to be retrieved and all the pedestrian gait sequences in the retrieval library aiming at one pedestrian gait sequence to be retrieved;

step V: and performing similarity sorting on the samples in the search library according to the calculated distance from small to large, wherein the more front the samples are, the more likely the samples are consistent with the ID of the pedestrian to be searched.

Images