CN111476058A - A Gesture Recognition Method Based on Millimeter Wave Radar - Google Patents

Abstract

The invention discloses a gesture recognition method based on millimeter wave radar, the method includes: constructing a convolutional neural network model; The network model is trained to obtain an optimized recognition model; the gesture trajectory diagram is the movement trajectory of the moving target corresponding to the maximum peak in the distance-Doppler coordinate system; the trajectory diagram of the recognized gesture is input into the optimized recognition model to identify The gesture type of the recognized gesture. In the gesture recognition method provided by the embodiment of the present invention, the convolutional neural network model is used to train the trajectory graphs of various gestures to obtain an optimized recognition model, and the trajectory graphs of the recognized gestures are input into the optimized recognition model, which can quickly and accurately obtain the gesture type. The gesture recognition method is relatively simple, the amount of data processing is small, and the calculation is simple.

Description

A Gesture Recognition Method Based on Millimeter Wave Radar

technical field

The invention relates to the technical field of gesture recognition, in particular to a gesture recognition method based on a millimeter wave radar.

Background technique

The recognition of gesture actions is generally processed based on the information collected by the camera, so as to realize the classification and recognition of different gestures. Gesture recognition has a wide range of applications, such as remotely turning on switches, manipulating tiny electronic devices, automatic sign language translation, etc., which can greatly improve the convenience of people's lives. However, using the camera to recognize gestures has the following disadvantages:

(1) The camera is easily affected by light, resulting in poor gesture recognition. Generally, when the light intensity is reduced by half, the accuracy of gesture recognition will be reduced by one-third.

(2) The camera is affected by the detection distance. When the distance is long, the recognition effect is not good. Moreover, when the person is farther away from the sensor, higher camera resolution is often required, which will cause problems of excessive data volume and high cost.

(3) The camera-based gesture recognition algorithm is relatively complex, with a large amount of data processing, resulting in high power consumption, high requirements on computing resources, and inconvenient integration into small devices.

(4) When a camera is connected to the Internet, it is easily attacked by criminals, resulting in privacy leakage.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a gesture recognition method based on millimeter wave radar. In the gesture gesture recognition method, the convolutional neural network model is used to train the trajectory graphs of various gestures, and an optimized recognition model is obtained, and the trajectory graphs of the recognized gestures are input to optimize the input. The recognition model can quickly and accurately obtain the gesture type. The gesture recognition method is relatively simple, the amount of data processing is small, and the calculation is simple.

In order to solve the above problems, the first aspect of the present invention provides a gesture recognition method based on millimeter wave radar, the method includes: constructing a convolutional neural network model; The training set F trains the convolutional neural network model to obtain an optimized recognition model; input the trajectory map of the recognized gesture into the optimized recognition model to recognize the gesture type of the recognized gesture.

Further, the method for acquiring the trajectory map of gestures includes: acquiring echo data generated by a millimeter-wave radar scanning a gesture; obtaining multiple RD images based on the echo data; searching and obtaining the spectral peaks of each RD image respectively. The maximum peak value of the image is obtained, and then multiple maximum peak points are obtained; in the RD coordinate system, the multiple maximum peak points are sequentially connected to obtain the trajectory map of the gesture.

Further, the structure of the convolutional neural network model sequentially includes: an input layer, a first convolutional layer, a first excitation layer, a first pooling layer, a second convolutional layer, a second excitation layer, a second pooling layer, The first fully connected layer, the second fully connected layer and the output layer.

Further, the step of recognizing the gesture type of the recognized gesture includes: based on the optimized recognition model, respectively obtaining the trajectory map of the recognized gesture and the probabilities of all types of gestures in the training set F, and the sum of the probabilities of all types of gestures is 1; the determination probability is higher than 95% of gestures are of the type that recognizes the gesture.

Further, the different gestures include one or more of forward and backward movement, left and right movement, button and flip.

The second aspect of the present invention further provides a gesture recognition system based on millimeter-wave radar, the system includes: a modeling module for constructing a convolutional neural network model; a training module for acquiring various gesture trajectory graphs as a training set F, The convolutional neural network model is trained based on the training set F to obtain an optimized recognition model; the gesture recognition module inputs the trajectory map of the recognized gesture into the optimized recognition model to recognize the gesture type of the recognized gesture.

Further, the method for acquiring the trajectory map of the gesture by the training module includes: acquiring echo data generated by a millimeter-wave radar scanning a gesture; obtaining multiple RD images based on the echo data; Peak search is performed to obtain the maximum peak value of the image, and then multiple maximum peak points are obtained; in the RD coordinate system, multiple maximum peak points are connected in sequence to obtain the trajectory map of the gesture.

Further, the structure of the convolutional neural network model sequentially includes: an input layer, a first convolutional layer, a first excitation layer, a first pooling layer, a second convolutional layer, a second excitation layer, a second pooling layer, The first fully connected layer, the second fully connected layer and the output layer.

Further, the step that the gesture recognition module recognizes the gesture type of the recognition gesture includes: based on the optimized recognition model, obtaining the probability that the recognition gesture is all types of gestures in the training set F respectively, wherein, the sum of the gesture probabilities of all types is 1; Gestures with a probability higher than 95% are the type of the identified gesture.

Further, the gesture includes one or more of forward and backward movement, left and right movement, button and flip.

The above-mentioned technical scheme of the present invention has the following beneficial technical effects:

(1) A method for obtaining a trajectory graph of a gesture provided by an embodiment of the present invention processes the echo data generated by the millimeter wave radar scanning the gesture to obtain a trajectory graph of the gesture, and each point on the graph refers to the maximum peak value The motion information of the corresponding moving target within a period of time, the motion information includes the distance between the moving target and the radar and the radial velocity of the moving target relative to the millimeter-wave radar within this period of time. Taking the trajectory map of the gesture as a feature of gesture recognition, compared with the prior art, the amount of data processing is greatly reduced, and the calculation is relatively simple.

(2) A millimeter-wave radar-based gesture recognition method and system provided by the embodiments of the present invention. In the gesture recognition method, a convolutional neural network model is used to train the trajectory graphs of various gestures to obtain an optimized recognition model. The trajectory map input optimizes the recognition model, which can quickly and accurately obtain the gesture type. The gesture recognition method is relatively simple, the amount of data processing is small, and the calculation is simple.

Description of drawings

1 is a schematic flowchart of a method for acquiring a gesture trajectory map according to a first embodiment of the present invention;

2 is an RD diagram of a gesture of moving up and down according to the first embodiment of the present invention;

3 is a schematic flowchart of a gesture recognition method based on a millimeter wave radar according to a second embodiment of the present invention;

Fig. 4a is the schematic diagram of finger button gesture action;

Fig. 4b is the trajectory diagram of finger making button;

Fig. 5a is the schematic diagram of palm turning movement;

Fig. 5b is a trajectory diagram of palm flipping motion;

6a is a schematic diagram of a left and right movement gesture;

Figure 6b is a trajectory diagram of a left and right movement gesture;

Fig. 7a is the schematic diagram of up and down movement gesture;

Fig. 7b is the trajectory diagram of the up and down movement gesture;

FIG. 8 is a schematic structural diagram of a gesture recognition system according to a third embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the specific embodiments and the accompanying drawings. It should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concepts of the present invention.

FIG. 1 is a schematic flowchart of a method for acquiring a trajectory map of a gesture according to a first embodiment of the present invention.

As shown in FIG. 1 , the method includes steps S102 to S108.

In an optional embodiment, the method includes steps S102-S108:

Step S102, acquiring echo data generated by scanning a gesture by the millimeter-wave radar. The echo data is echo baseband data.

Specifically, during the time when the hand movement exists, the millimeter-wave radar transmits a chirp signal to the target every few milliseconds, and collects the echo baseband data of the transmitted signal. For example, the duration of the hand movement is 5 seconds, and the millimeter-wave radar collects 100 frames of data in these 5 seconds, and the data of each frame is collected by the millimeter-wave radar in multiple times. For example, the millimeter-wave radar collects data every 5 milliseconds, and the millimeter-wave radar needs to collect 10 times to collect a frame of data.

Optionally, the millimeter-wave radar uses a linear frequency modulated continuous wave (LFMCW) radar, and further optionally, a 77GHz LFMCW millimeter-wave radar is used. The 77GHz LFMCW millimeter-wave radar includes two transmitting antennas and four receiving antennas. The transmitting signal is a linear frequency modulated continuous wave (LFMCW), the maximum bandwidth of the transmitting signal is 4GHz, and the theoretical maximum distance resolution is 3.75cm. It can detect relatively fine fingers sports.

It should be noted that LFMCW is a waveform whose frequency changes periodically with time. The frequency of each cycle generally increases linearly with time. This linear increase is called chirp. The parameters of chirp (such as time and slope) affect system performance. The design of the present invention The relevant parameters of the chirp signal are as follows: the bandwidth B is 3.4404GHz, the chirp period is 778us, the number of chirps per frame is 32, the number of frames per second is 50, and each chirp sample is 176 points. The transmission mode adopts the dual-antenna time-division multiplexing mode to transmit, a total of 20 frames of chirp signals are transmitted per second, and a total of 64 chirps are sent in each frame, which are sent alternately by antennas A and B.

Step S104, obtaining a plurality of RD images based on the echo data.

Specifically, fast Fourier transform (FFT transform) is performed on the echo data collected each time, and a one-dimensional matrix of distance dimension is obtained each time.

The one-dimensional matrix obtained by collecting echo data for the first time is taken as the first row of a two-dimensional matrix N, and the one-dimensional matrix obtained by collecting echo data for the second time is taken as the second row of the two-dimensional matrix N, and so on. , the two-dimensional matrix N is obtained, and the two-dimensional matrix N corresponds to one frame of gesture data collected by the radar.

Fourier transform is performed on the column vector of the two-dimensional matrix N, and a new two-dimensional matrix M is obtained. The two-dimensional matrix M is represented as an RD image (Range-Doppler, two-dimensional range-Doppler) corresponding to one frame of data collected by the radar.

Similarly, using the above steps, multiple RD images of the same gesture are collected. For example, if a gesture has 100 frames of data, 100 RD images will be obtained here.

In step S106, the spectral peaks of each of the RD images are searched respectively to obtain a plurality of maximum peak points.

Specifically, the spectral peaks of each RD image are searched to obtain the row coordinate i and column coordinate j corresponding to the hand motion and the value M _ij , and the M _ij is a peak point of each RD image.

The i, j, and M _ij in the multiple peak points M _ij in each RD graph are taken as row vectors, and multiple sets of row vectors form a new matrix O. Calculate the maximum value of the third column of the above matrix O, and record the row vector where the maximum value is located as V _max , where V _max is the maximum peak point in the RD graph. And so on, to get the maximum peak point of each picture in multiple RD pictures.

Step S108, in the RD coordinate system, connect a plurality of maximum peak points in sequence to obtain a trajectory diagram of the hand movement. The trajectory diagram of the gesture is the movement trajectory of the moving target corresponding to the maximum peak value in the range-Doppler coordinate system.

Specifically, the first column of each maximum row vector V _max is used as the abscissa, and the second column as the ordinate is marked with asterisks in a two-dimensional coordinate system, and the asterisks of two adjacent row vectors are connected at the same time. , and finally get an image, which is the trajectory map of the gesture.

The trajectory graph of the gesture is a two-dimensional distance-Doppler trajectory graph. Each point on the graph refers to the motion information of the moving target corresponding to the maximum peak value within a period of time, and the motion information includes the distance between the moving target and the radar. and the radial velocity of the moving target relative to the millimeter-wave radar during this time. Taking the trajectory map of the gesture as a feature of gesture recognition, compared with the prior art, the amount of data processing is greatly reduced, and the calculation is relatively simple.

2 is an RD diagram (distance-Doppler diagram) of a gesture of moving up and down according to the first embodiment of the present invention. In this diagram, the black and white grid in the middle is represented as a moving target. In this application, the moving target is Refers to the hand or a part of the hand, the depth of the color represents the echo intensity corresponding to the entire moving target. The darker the color, the stronger the echo intensity corresponding to the position of the moving target. In the RD plot, the black grid in the middle represents the largest peak in the RD plot. There is only one peak in this graph. This peak corresponds to the position where the echo of the moving target is the strongest, the abscissa indicates the radial velocity, and the ordinate indicates the distance from the radar.

FIG. 3 is a schematic flowchart of a gesture recognition method according to a second embodiment of the present invention.

As shown in FIG. 3, the method includes steps S201-S203:

Step S201, constructing a convolutional neural network model.

Specifically, the structure of the convolutional neural network model sequentially includes: an input layer, a first convolutional layer, a first excitation layer, a first pooling layer, a second convolutional layer, a second excitation layer, a second pooling layer, The first fully connected layer, the second fully connected layer and the output layer.

More specifically, before building a neural network, it is necessary to configure the learning rate of the convolutional neural network and initialize the weights of each layer of the convolutional neural network to adjust the training speed and recognition effect of the convolutional neural network. For example, the parameters of the initial convolutional neural network are: learning rate and other parameter settings: learning_rate (learning rate) is 0.001, beta1 (exponential decay rate of first-order moment estimation) is 0.9, beta2 (exponential decay of second-order moment estimation) ratio) is 0.999 and epsilon (this parameter is a very small number to prevent division by zero in implementations) is 10E-8. Weights and biases are initialized randomly.

Step S202, obtaining trajectory graphs of various gestures as a training set F, and training a convolutional neural network model based on the training set F to obtain an optimized recognition model. Among them, the trajectories of the gestures in the training set F are all set with corresponding gesture labels, that is, the gestures in the training set F are all known.

Optionally, training the convolutional neural network model includes the following steps:

Step 1, since the input layer only uses grayscale information, each gesture data in the training dataset L is set as a 90x90x1 matrix.

Step 2: The input layer data is filtered through the first convolution layer, the first excitation layer and the first pooling layer in sequence. Among them, the first convolution layer uses 32 filters, the number of channels is 1, the convolution kernel is 3x3, the convolution step size is 1, the padding method adopts the Same method, that is, the input and output sizes are the same, and the first excitation layer uses the ReLU function , after the first convolution and excitation layer, the output is a 90x90x32 matrix. The first pooling layer adopts the maxpooling method, the filter size is 2x2, the step size is 2, and the output after the first pooling layer is 45x45x32.

Step 3: Filter the output data of the first pooling layer through the second convolution layer and the second pooling layer in sequence. Among them, the second convolutional layer is similar to the first convolutional layer, except that the number of channels is 32, and the excitation function also uses the ReLU function. The parameters are similar to those of the first pooling layer, and the output is 23x23x32.

Step 4: Pull the output of the second pooling layer into a one-dimensional vector, and then pass through the first fully-connected layer and the second fully-connected layer in turn to obtain a matrix with an output of 4x1.

Step 5: Calculate the error between the output value and the target value obtained according to each layer in steps 1-4. When the error is greater than the preset expected value, the error is transmitted back to the network, that is, according to the obtained output value and the target value. The weights are updated.

The trajectory maps of various types of gestures in all training sets F are input into the convolutional neural network model and trained, and finally an optimized recognition model with precise parameters is obtained.

Step S203, input the trajectory map of the recognized gesture into the optimized recognition model to recognize the gesture type of the recognized gesture.

Specifically, based on the optimized recognition model, the probabilities that the recognized gestures are all gesture types in the training set F are respectively obtained, wherein the sum of the probabilities of all types of gestures is 1; the gestures with a determined probability higher than 95% are recognized gestures type. For example, there are 4 kinds of gestures in the training set F, and the optimized recognition model calculates that the probability that the recognized gesture is a gesture is a%, the probability that it is a gesture B is b%, the probability that it is a gesture C is c%, and the probability that it is a gesture D is d%. Wherein, the sum of a%, b%, c% and d% is 1. Finally, the optimized recognition model outputs gesture types with probability higher than 95%.

Optionally, the trajectory graphs of the various types of gestures in the above training set F include four gesture actions: up and down movement, left and right movement, palm turning movement, and finger button action.

The action diagrams of the four gestures and the trajectory diagram obtained according to the first embodiment are given below as a schematic diagram. FIG. 4a is a schematic diagram of a finger making a button gesture, and FIG. 4b is a trajectory diagram of the button. FIG. 5a is a schematic diagram of the palm turning movement, and FIG. 5b is a trajectory diagram of the palm turning movement. FIG. 6a is a schematic diagram of a left-right movement gesture, and FIG. 6b is a trajectory diagram of the left-right movement gesture. FIG. 7a is a schematic diagram of the up and down movement gesture, and FIG. 7b is a trajectory diagram of the up and down movement gesture.

In the second embodiment of the present invention, a total of the above four gestures are used, 132 groups of gesture trajectory maps (33 for each gesture) are used for training, and 28 groups of gesture trajectory diagrams (7 for each gesture) are used for testing , the gesture types corresponding to the RD images of the tested gestures are all recognized correctly. It can be seen that the method provided by the second embodiment of the present invention has a higher recognition rate.

Embodiments of the present invention provide a method and system for gesture recognition based on millimeter-wave radar. In the gesture recognition method, a convolutional neural network model is used to train the trajectory graphs of various gestures to obtain an optimized recognition model, and the trajectory graphs of the recognized gestures are input into the Optimize the recognition model to get the gesture type quickly and accurately. The gesture recognition method is relatively simple, the amount of data processing is small, and the calculation is simple.

FIG. 8 is a schematic structural diagram of a gesture recognition system according to a third embodiment of the present invention.

As shown in Figure 8, the system includes: a modeling module, a training module and a gesture recognition module.

The modeling module is used to build convolutional neural network models. Among them, the structure of the convolutional neural network model sequentially includes: an input layer, a first convolutional layer, a first excitation layer, a first pooling layer, a second convolutional layer, an excitation layer, a second pooling layer, and a first full connection layer, second fully connected layer and output layer.

The training module obtains various gesture trajectory graphs as the training set F, and trains the convolutional neural network model based on the training set F to obtain an optimized recognition model. Optionally, in the training process of the convolutional neural network model, the weight and bias of each layer can be estimated by using the training set with known classification results according to the Adam algorithm. According to the continuously input training set, the weight and bias will become more and more accurate.

Optionally, training the convolutional neural network model includes the following steps:

Step 1, since the input layer only uses grayscale information, each gesture data in the training dataset L is set as a 90x90x1 matrix.

Step 2: The input layer data is filtered through the first convolution layer, the first excitation layer and the first pooling layer in sequence. Among them, the first convolution layer uses 32 filters, the number of channels is 1, the convolution kernel is 3x3, the convolution step size is 1, the padding method adopts the Same method, that is, the input and output sizes are the same, and the first excitation layer uses the ReLU function , after the first convolution and excitation layer, the output is a 90x90x32 matrix. The first pooling layer adopts the maxpooling method, the filter size is 2x2, the step size is 2, and the output after the first pooling layer is 45x45x32.

Step 3: Filter the output data of the first pooling layer through the second convolution layer and the second pooling layer in sequence. Among them, the second convolutional layer is similar to the first convolutional layer, except that the number of channels is 32, and the excitation function also uses the ReLU function. The parameters are similar to those of the first pooling layer, and the output is 23x23x32.

Step 4: Pull the output of the second pooling layer into a one-dimensional vector, and then pass through the first fully-connected layer and the second fully-connected layer in turn to obtain a matrix with an output of 4x1.

Step 5: Calculate the error between the output value and the target value obtained according to each layer in steps 1-4. When the error is greater than the preset expected value, the error is transmitted back to the network, that is, according to the obtained output value and the target value. The weights are updated.

The trajectory maps of various types of gestures in all training sets F are input into the convolutional neural network model and trained, and finally an optimized recognition model with precise parameters is obtained.

The gesture recognition module inputs the trajectory map of the recognized gesture into the optimized recognition model to recognize the gesture type of the recognized gesture.

Specifically, the step of recognizing the gesture type of the recognized gesture includes: based on the optimized recognition model, respectively obtaining the probability that the recognized gesture is all types of gestures in the training set F, wherein the sum of the probabilities of all types of gestures The sum is 1; gestures with a probability higher than 95% are determined to be the type of the recognized gesture.

Optionally, when the convolutional neural network model calculates the probability of all types of gestures, and the probability of no gesture is higher than 95%, the model outputs "the gesture cannot be recognized".

Optionally, the trajectory graphs of the various types of gestures in the above training set F include four gesture actions: up and down movement, left and right movement, palm turning movement, and finger button action.

Embodiments of the present invention provide a method and system for gesture recognition based on millimeter-wave radar. In the gesture recognition method, a convolutional neural network model is used to train the trajectory graphs of various gestures to obtain an optimized recognition model, and the trajectory graphs of the recognized gestures are input into the Optimize the recognition model to get the gesture type quickly and accurately. The gesture recognition method is relatively simple, the amount of data processing is small, and the calculation is simple.

It should be understood that the above-mentioned specific embodiments of the present invention are only used to illustrate or explain the principle of the present invention, but not to limit the present invention. Therefore, any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and scope of the present invention should be included within the protection scope of the present invention. Furthermore, the appended claims of this invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalents of such scope and boundaries.

Claims (5)

1. A gesture recognition method based on millimeter wave radar is characterized by comprising the following steps:

constructing a convolutional neural network model;

acquiring a trajectory graph of various gestures as a training set F, and training the convolutional neural network model based on the training set F to obtain an optimized recognition model; the gesture track graph is a moving track of a moving target corresponding to the maximum peak value in a range-Doppler coordinate system;

inputting a trajectory diagram of a recognized gesture into the optimized recognition model to recognize a gesture type of the recognized gesture.

2. The millimeter wave radar-based gesture recognition method according to claim 1, wherein the gesture trajectory graph acquisition method comprises:

acquiring echo data generated by a millimeter wave radar for scanning a gesture;

obtaining a plurality of RD images based on the echo data;

respectively searching the spectral peak of each RD image and solving the maximum peak value of the image so as to obtain a plurality of maximum peak value points;

and sequentially connecting the maximum peak points in an RD coordinate system to obtain a track graph of the gesture.

3. The millimeter wave radar-based gesture recognition method according to claim 1, wherein the structure of the convolutional neural network model sequentially comprises: the device comprises an input layer, a first convolution layer, a first excitation layer, a first pooling layer, a second convolution layer, a second excitation layer, a second pooling layer, a first full-connection layer, a second full-connection layer and an output layer.

4. The millimeter wave radar-based gesture recognition method according to claim 1, wherein the step of recognizing the gesture type of the recognized gesture comprises:

based on the optimized recognition model, respectively obtaining probabilities that the recognized gesture is all gesture types in the training set F, wherein the sum of the gesture probabilities of all types is 1;

determining a gesture with a probability higher than 95% as the type of the recognized gesture.

5. The millimeter wave radar-based gesture recognition method according to any one of claims 1 to 4, wherein the gesture includes one or more of a back-and-forth movement, a left-and-right movement, a button, and a flip.

Images