CN111184512B - Method for recognizing rehabilitation training actions of upper limbs and hands of stroke patient - Google Patents

Abstract

The invention discloses an action recognition method for upper limb and hand rehabilitation training of stroke patients, which adopts a non-negative matrix decomposition model to separate electromyographic signal data blindly, removes non-stationary muscle activation information, and obtains a stable time-varying blind source. Separation results; apply the decomposed time-varying blind source separation result data for further pattern recognition to improve the stability and accuracy of recognition; the learned features maintain both temporal and spatial characteristics through the CNN-RNN model. The CNN-RNN model does not need to perform manual data feature extraction and screening, directly processes the data, automatically extracts features and completes classification and identification, which can realize end-to-end rehabilitation training action recognition and analysis, and combine the attention layer to analyze the first two-layer bidirectional GRU layer. The hidden state of the second layer is weighted by attention, giving more weight to the data with large contribution, making it play a greater role, thereby further improving the accuracy of classification and recognition.

Description

An action recognition method for upper limb and hand rehabilitation training in stroke patients

technical field

The invention belongs to the field of machine learning, and in particular relates to a method for recognizing an upper limb and hand rehabilitation training action of a stroke patient.

Background technique

Rehabilitation training, as an important treatment method in rehabilitation medicine, mainly improves the functional movement disorder of the corresponding limbs of the patient through different exercise training, and restores the patient's motor function as much as possible to achieve the therapeutic effect. Among the patients with limb dysfunction caused by stroke, 80% suffer from upper limb dysfunction. Among patients with upper limb dysfunction, only 30% of patients can finally achieve upper limb function recovery, and 12% of patients have better recovery of hand function. . Rehabilitation of upper limb and hand function has a profound impact on the quality of life and social participation of stroke patients. The recognition of stroke patients' rehabilitation training movements is widely used, and the recognition results can be used as control signals for auxiliary training equipment in clinical practice, such as mechanical exoskeletons, prostheses, etc.; they can also be used as input signals for serious game rehabilitation training, such as virtual reality rehabilitation training. ; Realize interactive rehabilitation training in community or home rehabilitation or assist physicians to monitor training conditions remotely, etc. Surface electromyography (sEMG) is non-invasive, easy to record, and contains rich motor control information, so surface electromyography is usually used to identify rehabilitation training movements.

At present, the machine learning method for rehabilitation training action recognition using surface EMG signals mainly includes three steps, signal preprocessing, feature extraction and classification and recognition. First, the collected original EMG signal is preprocessed, and the noise in the signal is removed through notch filtering, bandpass filtering and full-wave rectification, etc., and then the EMG time series is segmented by threshold method or artificial method, and the training actions are segmented. corresponding signal segment. In the feature extraction step, it is necessary to manually select and set features, and then extract these features from the preprocessed EMG signals. Features mainly include time domain features and frequency domain features. Time domain features include peak value, mean value, root mean square value, kurtosis, and autoregressive coefficients. Frequency domain features include power spectrum, intermediate frequency, center of gravity frequency, and frequency root mean square. Wait. Finally, using the feature set extracted in the previous step and the corresponding category labels as input, the machine learning algorithm is applied to train the classification and recognition model. At present, the classification methods commonly used in rehabilitation training action recognition mainly include decision classification tree, support vector machine (SVM), linear discriminant classifier (LDC), naive Bayes classifier (NB) and Gaussian mixture model (GMM). After the model training is completed, the newly collected EMG signals of training actions can be identified by preprocessing and feature extraction for rehabilitation training actions.

At present, the main process of recognition methods for rehabilitation training actions is to manually set and extract features, and then apply machine learning methods for classification and recognition.

This type of method model has its limitations. The original EMG signal has a certain degree of non-stationarity. Due to differences in physical signs of different patients, differences in stroke injuries, and differences in the degree of standardization of movement completion, etc., the EMG data will be greatly different, which will affect the recognition. Use basic filtering and rectification, etc. Preprocessing is difficult to remove this non-stationarity. And this kind of feature engineering usually requires some domain-specific expertise, which further increases the cost of preprocessing. The recognition performance of the existing model is highly dependent on feature selection, and different features have different effects on the classification and recognition performance; manually setting and extracting features may cause information redundancy due to the possible correlation between features; and for physiological time series data , whose time-varying information is also important, which is lost by artificial feature extraction. The currently used machine learning classifiers (such as SVM, LDC, GMM, etc.) have poor performance in identifying and distinguishing the damaged upper limb and hand movements of stroke patients, as well as some similar movements, such as different finger movements.

SUMMARY OF THE INVENTION

In view of the above deficiencies in the prior art, the present invention provides an action recognition method for stroke patients' upper limb and hand rehabilitation training, which solves the problem of information redundancy and loss of time-varying information caused by manual extraction and setting features, as well as poor machine learning recognition and discrimination performance. The problem.

In order to achieve the above purpose of the invention, the technical solution adopted in the present invention is: a method for recognizing the upper limb and hand rehabilitation training action of stroke patients, comprising the following steps:

S1, collect the electromyographic signal data of the rehabilitation training action;

S2. Preprocess the EMG signal data;

S3, using a non-negative matrix decomposition model to decompose the preprocessed EMG signal data to obtain multiple blind source separation result matrices;

S4. Use multiple blind source separation result matrices to iteratively train the CNN-RNN model to obtain a trained CNN-RNN model;

S5. Repeat steps S1 to S3 for the newly collected training action EMG data to obtain multiple blind source separation result matrices, and input the multiple blind source separation result matrices into the trained CNN-RNN model to obtain rehabilitation training action recognition category.

Further: Step S3 includes the following steps:

S31. Perform manual segmentation on the time dimension of the preprocessed EMG data to obtain an EMG data matrix composed of each piece of data in the corresponding time series;

S32, using a non-negative matrix decomposition model to decompose the electromyographic signal data matrix to obtain multiple blind source separation result matrices.

Further: step S4 includes the following steps:

S41, establish a CNN network model and an RNN network model, and initialize the number of iterations m=0;

S42, input multiple blind source separation result matrices into the CNN network model, perform feature extraction and pooling dimension reduction operations, and obtain feature vectors;

S43, input the feature vector into the RNN network model for processing, and obtain the probability value of the predicted action category;

S44, calculate the distance Loss _m between the predicted action category and the real action category probability value by cross entropy;

S45. Determine whether the difference between the m-th Loss _m value and the m-1 time Loss _m-1 value is less than the threshold, if so, obtain the trained CNN-RNN model, if not, adopt the batch stochastic gradient descent method Update the weight parameters and bias parameters in the CNN network model and the weight parameters of the RNN network model, increase the number of iterations m by 1, and jump to step S42.

Further, the CNN network model in step S41 includes: three layers of convolution layers, three layers of pooling layers and three layers of activation layers.

Further: the calculation formula of the input and output of the convolutional layer is:

为第l-1层卷积层的第i个输入通道的数据，

为第l层卷积层的第j个输出通道的数据，M_l-1为第l-1层卷积层的输入通道数，

为第l层卷积核权重，

为第l层卷积层偏置，1≤l≤3；第1层卷积层的第i个输入通道的数据为盲源分离结果矩阵H_r×n第i行数据。in,

is the data of the ith input channel of the l-1th convolutional layer,

is the data of the jth output channel of the lth convolutional layer, M _l-1 is the number of input channels of the l-1th convolutional layer,

is the weight of the convolution kernel of the lth layer,

is the bias of the convolutional layer of the first layer, 1≤l≤3; the data of the i-th input channel of the first-layer convolutional layer is the data of the i-th row of the blind source separation result matrix H _r×n .

Further: the RNN network model in step S41 includes: two bidirectional GRU layers, an attention layer and a fully connected layer, and each bidirectional GRU layer includes T ^′ GRU units;

The GRU unit includes an update gate and a reset gate;

The input in the first layer of the two-layer bidirectional GRU layer is a feature vector, and the output of the second layer is the input of the attention layer;

The output of the attention layer is the input of the fully connected layer.

Further: The state update equation of the GRU unit is as follows:

为候选集

的权重矩阵，x_t为特征向量，h_t为t时刻的隐含状态，h_t-1为t-1时刻的隐含状态。Among them, [] represents the connection of two vectors, · represents the vector inner product, σ is the activation function, tanh is the hyperbolic tangent activation function, W _r is the weight matrix of the reset gate, W _z is the weight matrix of the update gate,

candidate set

The weight matrix of , x _t is the eigenvector, h _t is the hidden state at time t, and h _t-1 is the hidden state at time t-1.

Further: inputting the hidden state _ht of the second layer in the two-layer bidirectional GRU layer into the attention layer for processing includes the following steps:

A1. Input the hidden state h _t of the second layer in the two-layer bidirectional GRU layer into the attention layer;

A2. Initialize the weight W _w and bias b _w of the attention layer;

A3. According to the weight W _w and bias b _w of the attention layer, obtain the hidden layer representation u _t of the hidden state h _t through the tanh hyperbolic tangent activation function;

A4. Randomly initialize a weight vector u _w , perform softmax normalization on the hidden layer representation u _t to obtain the attention weight α _t ;

A5. Weight the hidden state h _t by the attention weight α _t to obtain the attention weighted representation q _{t of the hidden state h t .}

Further: the attention weighting indicates that the processing process of q _t input to the fully connected layer includes:

C为全连接层的神经元个数；B1. Input the attention weighted representation q _t into the fully connected layer, and perform discrete processing to obtain the attention weight o _k ,

C is the number of neurons in the fully connected layer;

B2. Perform random inactivation operation on the attention weight ok and use _softmax to perform the classification operation to obtain the probability value of the predicted action category.

Further: the calculation formula of the probability value of the predicted action category in step B2 is:

Among them, _sk is the probability value of the predicted k-th action.

The beneficial effects of the present invention are as follows: a method for recognizing the upper limb and hand rehabilitation training action of stroke patients, adopts a non-negative matrix decomposition model to perform blind source separation of electromyographic signal data, removes non-stable muscle activation information, and obtains stable time Change the blind source separation result; apply the decomposed time-varying blind source separation result data for further pattern recognition, improve the stability and accuracy of the recognition; adopt the CNN network model to retain the spatial characteristics of the blind source separation result data, and the RNN network model The feature data is fused to provide time dimension information to facilitate the discriminating operation of the current data. The learned features maintain both temporal and spatial characteristics through the CNN-RNN model. The CNN-RNN model does not need to perform manual data feature extraction and screening. It directly processes the data, automatically extracts features, and completes classification and recognition. It can realize end-to-end rehabilitation training action recognition and analysis, and combine the attention layer for the first two-layer two-way GRU layer. The hidden state of the second layer is weighted by attention, giving more weight to the data with large contribution, making it play a greater role, thereby further improving the accuracy of classification and recognition.

Description of drawings

Fig. 1 is a flow chart of an action recognition method for upper limb and hand rehabilitation training of stroke patients;

Figure 2 is a schematic diagram of the structure of the CNN network model;

Figure 3 is a structural diagram of a GRU unit;

Figure 4 is a partial network model structure diagram of the RNN network model.

Detailed ways

The specific embodiments of the present invention are described below to facilitate those skilled in the art to understand the present invention, but it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes Such changes are obvious within the spirit and scope of the present invention as defined and determined by the appended claims, and all inventions and creations utilizing the inventive concept are within the scope of protection.

As shown in Figure 1, a method for recognizing the movements of upper limb and hand rehabilitation training for stroke patients includes the following steps:

S1, collect the electromyographic signal data of the rehabilitation training action;

Electrodes were placed on the subjects' finger extensors, finger flexors, biceps, triceps, deltoid, thenar muscle, and hypothenar muscle, and a total of 8 channels of surface electromyography (sEMG) were collected. ) signal, the sampling frequency is 2KHz, and the placement of EMG is shown in Table 1.

Table 1 EMG electrode position design

In this embodiment, a total of 25 functional actions are designed in the rehabilitation work training and sports training of the upper arm, forearm, and hand, as shown in Table 2. The subject was sitting relaxed in an armchair. For hand, wrist and elbow movements, place the arms in a fixed position on the table, follow the video or voice instructions, and take a total of 5 seconds from rest to induce contraction and maintain the posture, each movement is repeated 6 times, rest between movements The time is 5 seconds; for the shoulder movement, the subject sits on a chair with no obstructions in front, according to the video or voice instructions, from stillness to causing muscle contraction to complete the movement and maintain the posture for 5 seconds, each movement is repeated 6 times , 5 seconds rest between movements. During the experiment, videos of each movement performed by healthy individuals were used as demonstrations to guide subjects to perform (intend to perform) each movement.

Table 2 Experimental Action Design

S2. Preprocess the EMG signal data, that is, use a 50Hz notch filter to remove power line interference noise, and 20Hz-450Hz bandpass filter to remove motion artifacts (<20) and high-frequency noise (>450), and perform full wave rectification;

S3, using a non-negative matrix decomposition model to decompose the preprocessed EMG signal data to obtain multiple blind source separation result matrices;

Step S3 includes the following steps:

S31. Perform manual segmentation on the time dimension of the preprocessed EMG data to obtain an EMG data matrix composed of each piece of data in the corresponding time series;

S32, using a non-negative matrix decomposition model to decompose the electromyographic signal data matrix to obtain multiple blind source separation result matrices.

The non-negative matrix decomposition model in step S32 is:

X _m×n =W _m×r ×H _r×n

Among them, X _m×n is the EMG data matrix of m×n dimension, m is the number of electrodes, n is the number of measurement values, W _m×r is the muscle activity matrix of m×r dimension, H _r×n is r× N-dimensional blind source separation result matrix.

S4. Use multiple blind source separation result matrices to iteratively train the CNN-RNN model to obtain a trained CNN-RNN model;

Step S4 includes the following steps:

S41, establish a CNN network model and an RNN network model, and initialize the number of iterations m=0;

The CNN network model in step S41 includes: three layers of convolution layers, three layers of pooling layers, and three layers of activation layers, as shown in FIG. 2 .

The calculation formula for the input and output of the convolutional layer is:

为第l-1层卷积层的第i个输入通道的数据，

为第l层卷积层的第j个输出通道的数据，M_l-1为第l-1层卷积层的输入通道数，

为第l层卷积核权重，

为第l层卷积层偏置，1≤l≤3；第1层卷积层的第i个输入通道的数据为盲源分离结果矩阵H_r×n第i行数据。in,

is the data of the ith input channel of the l-1th convolutional layer,

is the data of the jth output channel of the lth convolutional layer, M _l-1 is the number of input channels of the l-1th convolutional layer,

is the weight of the convolution kernel of the lth layer,

is the bias of the convolutional layer of the first layer, 1≤l≤3; the data of the i-th input channel of the first-layer convolutional layer is the data of the i-th row of the blind source separation result matrix H _r×n .

S42, input multiple blind source separation result matrices into the CNN network model, perform feature extraction and pooling dimension reduction operations, and obtain feature vectors;

The activation layer introduces nonlinearity into the CNN network model to improve the fitting ability of the neural network. In this embodiment, the activation layer uses the ReLU nonlinear function, so that the network can be trained faster, without compromising accuracy, and alleviating the problem of vanishing gradients .

The pooling layer is used to reduce the latitude of the input data, and at the same time reduce the number of parameters or weights to be trained, thereby reducing the computational cost and controlling overfitting.

S43, input the feature vector into the RNN network model for processing, and obtain the probability value of the predicted action category;

The RNN network model includes: two-layer bidirectional GRU layer, attention layer and fully connected layer, each bidirectional GRU layer includes T' GRU units; the GRU unit includes an update gate and a reset gate;

The input in the first layer of the two-layer bidirectional GRU layer is a feature vector, and the output of the second layer is the input of the attention layer;

The output of the attention layer is the input of the fully connected layer, as shown in Figure 3.

The state update equation of the GRU unit is as follows:

为候选集

的权重矩阵，x_t为特征向量，h_t为t时刻的隐含状态，h_t-1为t-1时刻的隐含状态。Among them, [] represents the connection of two vectors, · represents the vector inner product, σ is the activation function, tanh is the hyperbolic tangent activation function, W _r is the weight matrix of the reset gate _rt , and W _z is the weight of the update gate z _t matrix,

candidate set

The weight matrix of , x _t is the eigenvector, h _t is the hidden state at time t, and h _t-1 is the hidden state at time t-1.

As shown in Figure 4, inputting the hidden state _ht of the second layer in the two-layer bidirectional GRU layer into the attention layer for processing includes the following steps:

A1. Input the hidden state h _t of the second layer in the two-layer bidirectional GRU layer into the attention layer;

A2. Initialize the weight W _w and bias b _w of the attention layer;

A3. According to the weight W _w and bias b _w of the attention layer, obtain the hidden layer representation u _t of the hidden state h _t through the tanh hyperbolic tangent activation function;

A4. Randomly initialize a weight vector u _w , perform softmax normalization on the hidden layer representation u _t to obtain the attention weight α _t ;

A5. Weight the hidden state h _t by the attention weight α _t to obtain the attention weighted representation q _{t of the hidden state h t .}

The calculation formula of the attention weighted representation q _t in step A5 is:

u _t =tanh(W _w h _t +b _w )

q _t =α _t h _t

The attention weighting indicates that the processing process of the _qt input fully connected layer includes:

C为全连接层的神经元个数；B1. Input the attention weighted representation q _t into the fully connected layer, and perform discrete processing to obtain the attention weight o _k ,

C is the number of neurons in the fully connected layer;

B2. Perform random inactivation operation on the attention weight ok and use _softmax to perform the classification operation to obtain the probability value of the predicted action category.

The calculation formula of the probability value of the predicted action category in step B2 is:

Among them, _sk is the probability value of the predicted k-th action.

S44, calculate the distance Loss _m between the predicted action category and the real action category probability value by cross entropy;

The calculation formula for calculating the distance Loss _m between the prediction and the real action category probability value in step S44 is:

Among them, Batch is the number of batch samples, _n is the nth piece of data, On is the predicted action category probability value distribution of the nth piece of data: {s ₁ ,s ₂ …,s _C }, y _n The true value of the nth piece of data Action class probability value.

S45. Determine whether the difference between the m-th Loss _m value and the m-1 time Loss _m-1 value is less than the threshold, if so, obtain the trained CNN-RNN model, if not, adopt the batch stochastic gradient descent method Update the weight parameters and bias parameters in the CNN network model and the weight parameters of the RNN network model, increase the number of iterations m by 1, and jump to step S42.

S5. Repeat steps S1 to S3 for the newly collected training action EMG data to obtain multiple blind source separation result matrices, and input the multiple blind source separation result matrices into the trained CNN-RNN model to obtain rehabilitation training action recognition category.

The beneficial effects of the present invention are as follows: a method for recognizing the upper limb and hand rehabilitation training action of stroke patients, adopts a non-negative matrix decomposition model to perform blind source separation of electromyographic signal data, removes non-stable muscle activation information, and obtains stable time The result of blind source separation is changed; the decomposed time-varying blind source separation result data is used for further pattern recognition to improve the stability and accuracy of the recognition; the CNN network model is used to retain the spatial characteristics of the blind source separation result data, and the RNN network model The feature data is fused to provide time dimension information to facilitate the discriminating operation of the current data. The learned features maintain both temporal and spatial characteristics through the CNN-RNN model. The CNN-RNN model does not need to perform manual data feature extraction and screening, directly processes the data, automatically extracts features and completes classification and recognition, which can realize end-to-end rehabilitation training action recognition and analysis, and combine the attention layer for the first two-layer two-way GRU layer. The hidden state of the second layer is weighted by attention, giving more weight to the data with large contribution, making it play a greater role, thereby further improving the accuracy of classification and recognition.

Claims (5)

1. A stroke patient upper limb and hand rehabilitation training action recognition method is characterized by comprising the following steps:

s1, collecting myoelectric signal data of rehabilitation training actions;

s2, preprocessing electromyographic signal data;

s3, decomposing the preprocessed electromyographic signal data by adopting a non-negative matrix decomposition model to obtain a plurality of blind source separation result matrixes;

s4, performing iterative training on the CNN-RNN model by adopting a plurality of blind source separation result matrixes to obtain a trained CNN-RNN model;

s5, repeating the steps S1-S3 on newly collected training action electromyographic data to obtain a plurality of blind source separation result matrixes, and inputting the plurality of blind source separation result matrixes into a trained CNN-RNN model to obtain rehabilitation training action recognition categories;

step S3 includes the following steps:

s31, manually segmenting the preprocessed electromyographic signal data in a time dimension to obtain an electromyographic signal data matrix formed by each piece of data of the corresponding time sequence;

s32, decomposing the electromyographic signal data matrix by adopting a non-negative matrix decomposition model to obtain a plurality of blind source separation result matrixes;

the step S4 includes the following steps:

s41, establishing a CNN network model and an RNN network model, and initializing the iteration number m to be 0;

the RNN network model in step S41 includes: two bidirectional GRU layers, an attention layer and a full connection layer, wherein each bidirectional GRU layer comprises T' GRU units;

the GRU unit comprises an updating gate and a resetting gate;

the input in the first of the two bidirectional GRU layers is a feature vector, and the output of the second layer is the input of the attention layer;

the output of the attention layer is the input of the full connection layer;

the state update equation for the GRU unit is as follows:

wherein, the [ alpha ], [ beta ]]Representing the concatenation of two vectors, representing the inner product of the vectors, σIs an activation function, tanh is a hyperbolic tangent activation function, W_rTo reset the weight matrix of the gate, W_zIn order to update the weight matrix for the gate,

as a candidate set

Weight matrix of x_tIs a feature vector, h_tIs an implicit state at time t, h_t-1Is an implicit state at the moment t-1;

the CNN network model in step S41 includes: three layers of convolution layer, three layers of pooling layer and three layers of activation layer; the method comprises the following steps that a convolutional layer is connected with an active layer and then connected with a pooling layer to form a group of network units, and the three groups of network units are sequentially connected to form a CNN network model;

s42, inputting the blind source separation result matrixes into a CNN network model, and performing feature extraction and pooling dimension reduction operation to obtain feature vectors;

s43, inputting the feature vector into an RNN network model for processing to obtain a probability value of the predicted action category;

s44, calculating the distance Loss of the probability values of the predicted action category and the real action category through cross entropy_m；

S45, judging the Loss of the mth time_mValue sum Loss of m-1 times_m-1And if the difference value of the values is smaller than the threshold value, obtaining the trained CNN-RNN model, otherwise, updating the weight parameter and the offset parameter in the CNN network model and the weight parameter of the RNN network model by adopting a batch random gradient descent method, adding 1 to the iteration number m, and jumping to the step S42.

2. The method for recognizing rehabilitation training actions of upper limbs and hands of stroke patients as claimed in claim 1, wherein the calculation formula of the input and output of the convolutional layer is as follows:

wherein,

is the data of the ith input channel of the l-1 th convolutional layer,

data of the jth output channel of the first convolutional layer, M_l-1The number of input channels of the l-1 th convolutional layer,

for the l-th layer of the convolution kernel weights,

the first layer of convolution layer is offset, l is more than or equal to 1 and less than or equal to 3; the data of the ith input channel of the 1 st convolutional layer is a blind source separation result matrix H_r×nThe ith row of data.

3. The method of claim 1, wherein the hidden state h of the second layer of the two-layer bidirectional GRU layer is used for recognizing the rehabilitation training actions of the upper limbs and the hands of the patient with stroke_tThe input into the attention layer for processing comprises the following steps:

a1 hidden state h of the second layer of two-layer bidirectional GRU layer_tInput into the attention layer;

a2 weight W of initial attention layer_wAnd bias b_w；

A3, weight W according to attention level_wAnd bias b_wObtaining the hidden state h by tanh hyperbolic tangent activation function_tHidden layer representation of u_t；

A4, randomly initializing a weight vector u_wFor the hidden layer to represent u_tPerforming softmax standardization to obtain the attention weight alpha_t；

A5, implicit State h_tBy paying attention toForce weight alpha_tWeighting to obtain hidden state h_tIs expressed as a weighted attention representation q_t。

4. The method of claim 1, wherein the attention-weighted representation q represents a weight of the patient's upper limbs and hands in the rehabilitation training action recognition_tThe process of inputting the full connection layer for processing comprises the following steps:

b1, representing attention by weighting q_tInputting the full connection layer, and performing discrete processing to obtain attention weighting o_k，

C is the number of the neurons of the full connection layer;

b2 weighting attention o_kAnd carrying out random inactivation operation and classification operation by using softmax to obtain the probability value of the predicted action category.

5. The method for recognizing rehabilitation training actions on upper limbs and hands of patients with stroke according to claim 4, wherein the probability value of the predicted action category in the step B2 is calculated by the formula:

wherein s is_kIs the probability value of the predicted kth action.

Images