CN111860130A - Audio-based gesture recognition method, device, terminal device and storage medium - Google Patents

Abstract

The present application is applicable to the technical field of human-computer interaction, and provides an audio-based gesture recognition method, device, terminal device and storage medium. The audio-based gesture recognition method includes: acquiring a target audio signal, the target audio signal is an audio signal received after a preset original audio signal is modulated and propagated through a target gesture made by a user; Channel estimation is performed on the audio signal and the target audio signal to obtain characteristic data of the channel estimation; the characteristic data of the channel estimation is identified to obtain the recognition result of the target gesture. The present application extracts the feature data of the target gesture on the original audio signal and the target audio signal through channel estimation, so as to obtain accurate channel estimation feature data, so as to improve the accuracy of the gesture recognition result.

Description

Audio-based gesture recognition method, device, terminal device and storage medium

technical field

The present application belongs to the technical field of human-computer interaction, and in particular, relates to an audio-based gesture recognition method, device, terminal device and storage medium.

Background technique

With the popularity of smart devices, more and more sensors are equipped with smart devices, which makes it more and more convenient to use sensors embedded in existing commercial devices to perform gesture recognition.

The existing audio-based gesture recognition methods are usually based on the continuous wave Doppler effect for gesture recognition. Although this method can overcome the limited use scene caused by gesture recognition based on inertial sensors of vision and wearable devices, the accuracy of gesture recognition is low due to the low resolution of the continuous wave signal used. .

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present application provide an audio-based gesture recognition method, device, terminal device, and storage medium, which extract feature data of target gestures on the original audio signal and the target audio signal through channel estimation to obtain accurate channel Estimated feature data to improve the accuracy of gesture recognition results.

In a first aspect, an embodiment of the present application provides an audio-based gesture recognition method, including:

Acquiring a target audio signal, the target audio signal is the audio signal received after the preset original audio signal is modulated and propagated through the target gesture made by the user;

Perform channel estimation based on the original audio signal and the target audio signal to obtain characteristic data of the channel estimation;

Identify the feature data of the channel estimation to obtain the recognition result of the target gesture.

In this embodiment of the present application, the modulated original audio is played, the user makes a gesture to obtain a target audio signal containing gesture features, and channel estimation is performed by using the original audio signal and the target audio signal to obtain accurate channel estimation feature data. The feature data of the channel estimation is identified, and an accurate target gesture recognition result is output.

Further, the original audio signal is a periodic signal, and the channel estimation is performed based on the original audio signal and the target audio signal to obtain characteristic data of the channel estimation including:

Perform demodulation processing on the target audio signal to obtain a target fundamental frequency signal;

The target fundamental frequency signal is segmented to obtain a plurality of target signal segments, and the length of each target signal segment is the same as the period of the original audio signal;

For each of the target signal segments, channel estimation is performed with a period of signal segments in the original audio signal, respectively, to obtain respective channel characteristic data;

The channel characteristic data of each of the target signal segments are integrated to obtain the characteristic data of the channel estimation.

The target signal segment is obtained by demodulating the target audio signal and segmenting it according to the period, and then using a period signal segment in the original audio signal to perform channel estimation with each target signal segment to obtain the respective channel characteristic data. By combining the channel feature data, more accurate channel estimation feature data can be obtained, thereby improving the accuracy of gesture recognition.

Further, performing demodulation processing on the target audio signal to obtain the target fundamental frequency signal includes:

Carrying out carrier reduction and IQ decomposition to the target audio signal to obtain a real part signal and an imaginary part signal of the reduced carrier signal;

Using a low-pass filter to denoise the real part signal and the imaginary part signal of the down-carrier signal to obtain the target fundamental frequency signal.

Since the audio signal received by the microphone belongs to the audio signal modulated by the original audio signal, channel estimation is practically performed on the signal before modulation. Therefore, it is necessary to demodulate the target audio signal, that is, carry out carrier reduction and IQ decomposition to obtain the real part signal and imaginary part signal of the reduced carrier signal; Obtain the target audio signal with noise interference filtered out, and improve the resolution of the target audio signal.

Further, each of the target signal segments has a time stamp, and the channel feature data obtained by integrating the channel feature data of each of the target signal segments to obtain the channel estimation feature data includes:

Representing the channel feature data of each of the target signal segments in the form of vectors, respectively, to obtain a feature vector of each of the target signal segments;

Arranging the feature vectors of each of the target signal segments according to their respective timestamp sizes to form a feature matrix;

For each column in the feature matrix, the respective element values contained in each are subtracted from the corresponding element values contained in the previous column to obtain the feature matrix after static elimination;

The eigenmatrix after static elimination is determined as eigendata of the channel estimation.

By converting the channel feature data into feature vectors, and arranging and combining the feature vectors according to the time sequence, a feature matrix is generated, and the feature matrix after static elimination is obtained by subtracting the element values of the front and rear columns of the feature matrix. characteristic data. The channel estimation feature data after static elimination can reduce the influence of the static reflection signal on the gesture signal, and realize accurate recognition of fine-grained gestures.

Further, the identifying the feature data of the channel estimation to obtain the recognition result of the target gesture includes:

Using a pre-built domain adaptive neural network model to identify the feature data of the channel estimation to obtain a recognition result of the target gesture;

Wherein, the domain adaptive neural network model is constructed by the following steps:

Obtain a first training data set and a second training data set, the first training data set includes multiple sets of channel estimation sample data with assigned domain labels and gesture labels, and the second training data set includes multiple sets of assigned domain labels Channel estimation sample data without assigning gesture labels;

Using the first training data set and the second training data set as training sets, the domain adaptive neural network model is obtained by training.

The pre-built domain adaptive neural network model trains the neural network model through a large number of channel estimation samples assigned domain labels and gesture labels and channel estimation samples that are only assigned domain labels but not assigned gesture labels. Gesture extraction and recognition of irrelevant target features to optimize the accuracy of recognition.

Further, using the first training data set and the second training data set as training sets, and obtaining the domain adaptive neural network model through training includes:

Inputting the first training data set into an initial domain adaptive neural network model to obtain the gesture prediction result and the domain prediction result of the first training data set;

According to the assigned domain label and gesture label of the first training data set, and the gesture prediction result and the domain prediction result of the first training data set, calculate the first cross entropy E _label ;

Inputting the second training data set into the initial domain adaptive neural network model to obtain the gesture prediction result and the domain prediction result of the second training data set;

Calculate the second cross entropy E _unlab according to the assigned domain label of the second training data set, and the gesture prediction result and the domain prediction result of the second training data set;

Based on the assigned domain labels of the first training dataset, the assigned domain labels of the second training dataset, the domain prediction results of the first training dataset and the domain of the second training dataset The prediction result is calculated to obtain the domain cross entropy E _s ;

Calculate the cross-entropy of the initial domain adaptive neural network model according to the first cross-entropy, the second cross-entropy and the domain cross-entropy E=E _label +αE _unlab -βE _s , where α and β are domains Model parameters of the adaptive neural network model;

Taking the minimum value of the cross entropy E as the goal, the model parameters of the initial domain adaptive neural network model are optimized to obtain the domain adaptive neural network model.

By calculating the first cross-entropy, the second cross-entropy and the domain cross-entropy of gesture recognition in the two training datasets respectively, the cross-entropy of the domain-adaptive neural network model is constructed, and the minimum value of the cross-entropy is taken as the target, and the domain is adjusted The model parameters of the adaptive network model are optimized to obtain the optimized domain adaptive neural network model. In the optimization process, the weight of signal features that are unique to the domain and have nothing to do with gestures is reduced, the accuracy of recognition of new domain datasets is improved, and the generalization of the system is improved.

Further, the obtaining the target audio signal includes:

Obtaining N copies of the target audio signal collected by N different microphone devices, where N is an integer greater than 1;

The channel estimation is performed based on the original audio signal and the target audio signal, and the characteristic data obtained for the channel estimation includes:

For each piece of the target audio signal, perform channel estimation with the original audio signal respectively to obtain N pieces of channel feature data;

The N pieces of channel characteristic data are integrated to obtain the characteristic data of the channel estimation.

Using multiple audio acquisition devices to perform multi-dimensional acquisition and processing of target audio signals can obtain more feature data for channel estimation and improve gesture recognition results. For example, when the gesture is rotated clockwise and counterclockwise, the two gestures are in a mirror image relationship. If a single microphone and a single speaker are used, only 1-dimensional channel estimation information can be calculated, and it is impossible to identify whether the specific gesture is rotated clockwise or counterclockwise. However, if multiple microphone devices are used, more gesture information can be obtained, so as to determine whether to rotate clockwise or counterclockwise.

In a second aspect, an embodiment of the present application provides an audio-based gesture recognition device, including:

a signal acquisition module, configured to acquire a target audio signal, where the target audio signal is an audio signal received after a preset original audio signal is modulated and propagated through a target gesture made by a user;

a channel estimation module, configured to perform channel estimation based on the original audio signal and the target audio signal to obtain characteristic data of the channel estimation;

The gesture recognition module is used for recognizing the feature data of the channel estimation to obtain the recognition result of the target gesture.

In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the computer program The steps of implementing the gesture recognition method described in the first aspect above.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the gesture recognition according to the first aspect above is implemented steps of the method.

Compared with the prior art, the embodiment of the present application has the beneficial effect of extracting the feature data of the target gesture on the original audio signal and the target audio signal through channel estimation, so as to obtain accurate channel estimation feature data, so as to improve the accuracy of the gesture recognition result. Accuracy.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only for the present application. In some embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a flowchart of an audio-based gesture recognition method provided by an embodiment of the present application;

FIG. 2 is a flowchart of a channel estimation method provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of the principle of channel estimation provided by an embodiment of the present application;

4 is a heat map corresponding to characteristic data of channel estimation obtained after a maximum likelihood estimation algorithm provided by an embodiment of the present application;

5 is a heat map corresponding to feature data of channel estimation after static elimination and a heat map corresponding to channel estimation feature data of an enlarged single gesture provided by an embodiment of the present application;

6 is a schematic structural diagram of a domain adaptive neural network model provided by an embodiment of the present application;

7 is a reference diagram for threshold determination of a calling neural network model provided by an embodiment of the present application;

8 is a schematic structural diagram of an audio-based gesture recognition device provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of a terminal device provided by an embodiment of the present application.

Detailed ways

In the following description, for the purpose of illustration rather than limitation, specific details such as specific device structures and technologies are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

The terms used in the following embodiments are for the purpose of describing particular embodiments only, and are not intended to be limitations of the present application. As used in the specification of this application and the appended claims, the singular expressions "a," "an," "the," "above," "the," and "the" are intended to also Expressions such as "one or more" are included unless the context clearly dictates otherwise. It should also be understood that, in this embodiment of the present application, "one or more" refers to one, two or more; "and/or", which describes the association relationship of associated objects, indicates that there may be three kinds of relationships; for example, A and/or B can mean that A exists alone, A and B exist simultaneously, and B exists independently, wherein A and B can be singular or plural. The character "/" generally indicates that the associated objects are an "or" relationship.

The audio-based gesture recognition method provided by the embodiments of the present application may be based on mobile phones, tablet computers, medical devices, wearable devices, in-vehicle devices, augmented reality (AR)/virtual reality (VR) devices, laptops, The implementation of terminal devices or servers with speakers and microphones such as ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. No restrictions apply.

Barker code is a kind of binary code group with special regularity proposed by R.H. Barker in the early 1950s. It has good autocorrelation, so it is also better for channel feature data extraction when applied to channel estimation. After the Barker code is stretched, smoothed, supplemented, and amplified, a fundamental frequency signal with a single cycle length greater than 300 bits is generated as the original audio signal to obtain a good resolution. The estimated feature data is also more accurate. Although the longer the cycle length, the higher the resolution, but too long the cycle will cause system delay. After repeated experiments and verification, the better choice is the fundamental frequency signal with a single cycle length of 480 bits, which can not only take into account the resolution of the signal, but also avoid the phenomenon of system delay. For the convenience of understanding, the following examples will directly use the original audio signal with a single cycle length of 480 bits as an illustration.

FIG. 1 shows a flowchart of an audio-based gesture recognition method provided by the present application. In one embodiment, the gesture recognition method includes:

101. Obtain a target audio signal, which is an audio signal received after a preset original audio signal is modulated and propagated through a target gesture made by a user;

First, the target audio signal is obtained. The target audio signal is obtained by using a preset original audio signal, which is modulated and played through a speaker. During the propagation process, the modulated original audio signal will change after the target gesture made by the user. Channel characteristic data of the propagated audio signal, at this time, the microphone device can receive the target audio signal containing the target gesture characteristic data. The original audio signal is played after modulation because the original audio signal is the fundamental frequency signal, which is very harsh to play directly; and the signal and system theory points out that the channel calculated by the signal at the fundamental frequency is consistent with the channel that the signal passes through at the carrier frequency, so it can be used Channel estimation is performed on the demodulated target fundamental frequency signal and the original audio signal, and the obtained channel estimation result is consistent with the channel estimation result passed after uploading the frequency, and the accuracy rate is high. Therefore, the original audio signal is modulated during playback. Specifically, in this embodiment, the original audio signal is subjected to carrier wave to reach a frequency band that cannot be recognized by human ears, and then the playback is performed, and then the target audio signal is demodulated into The baseband signal is used for channel estimation.

102. Perform channel estimation based on the original audio signal and the target audio signal to obtain characteristic data of the channel estimation;

The key point of audio-based gesture recognition lies in the need to accurately extract feature data that can be recognized. In this embodiment, the original audio signal with high resolution is used for channel estimation, which can improve the extraction accuracy of the feature data of the channel estimation. However, the target audio signal is the modulated audio signal of the original audio signal. Therefore, before performing channel estimation, the target audio signal needs to be demodulated to obtain the fundamental frequency signal, and further participate in the channel estimation to obtain the characteristic data of the channel estimation. .

FIG. 2 shows a flowchart of a channel estimation method provided by the present application, and FIG. 3 is a schematic diagram of a channel estimation principle provided by the present application.

2 and 3, in one embodiment, the step of performing channel estimation based on the original audio signal and the target audio signal includes:

201. Perform demodulation processing on the target audio signal to obtain a target fundamental frequency signal;

The channel estimation is performed based on the fundamental frequency signal, so in this step, the obtained target audio signal needs to be demodulated to obtain the target fundamental frequency signal. The specific demodulation steps include:

First, the target audio signal is subjected to down-carrier and IQ decomposition to obtain the real part signal and imaginary part of the down-carrier signal; specifically, the received target audio signal is sub-carrier to obtain a down-carrier signal, and then IQ decomposition is performed, that is, the target audio signal is decomposed by IQ. The audio signal is multiplied by Acos(2πf ct) and _-Asin (2πf ct ₎ , respectively, where A is the amplitude. After the decomposition, the real part signal and imaginary part of the down-carrier signal can be obtained. It should be noted that there is no specific sequence for down-carrier and IQ decomposition. The carrier can be reduced first and then IQ decomposition is performed, or IQ decomposition can be performed first and then IQ decomposition. The carrier is dropped, and the final result will not change due to the processing order. Since the obtained target audio signal may have noise interference, after obtaining the real part signal and imaginary part signal of the reduced carrier signal, the low-pass filter is used to achieve denoising and improve the resolution of the target fundamental frequency signal. Then, the denoised real part signal and the imaginary part signal are combined into a complex number form, and the fundamental frequency signal in the complex number form is the target fundamental frequency signal. At this step, the obtained target audio signal has been demodulated and de-noised to generate the target fundamental frequency signal, and the preliminary conditions for participating in channel estimation have been met. However, the target fundamental frequency signal at this time is a very long signal, and the characteristic data of channel estimation obtained directly for channel estimation is not accurate, so segmentation processing is required.

202. Segment the target fundamental frequency signal to obtain a plurality of target signal segments, and the length of each target signal segment is the same as the period of the original audio signal;

The original audio is a periodic audio signal, so the obtained target fundamental frequency signal also has periodic characteristics. During segmentation, the characteristics of the cycle can be used to divide and cut the target fundamental frequency signal with one cycle as a segment to obtain a plurality of target signal segments. The cycle length of the obtained target signal segment and the cycle length of the original audio are The same; of course, one cycle of the original audio signal can also be extracted, and the single cycle of the original audio signal and the target fundamental frequency signal can be used to align and then cut into segments, and multiple targets with the same cycle length as the original audio can also be obtained. signal fragment. After obtaining a plurality of target signal segments, these signal segments can be used to perform channel estimation and output channel estimation feature data.

203. For each of the target signal segments, perform channel estimation with a period of signal segments in the original audio signal, respectively, to obtain respective channel characteristic data;

The purpose of segmenting the baseband signal in step 202 is to align with the original audio signal in the channel estimation process, so as to obtain more accurate channel estimation feature data. Therefore, in this step, channel estimation is performed on the original audio signal of a single cycle and each target signal segment respectively, and the cycle lengths are equal. Channel estimation is performed directly without other processing, and an accurate channel estimation result is obtained. However, at this time, the channel feature data corresponding to each signal segment is obtained, and the corresponding recognition may be only a part of a gesture in the target gesture. Therefore, it is necessary to integrate multiple channel feature data together to obtain a coherent target gesture correspondence. After the characteristic data of the channel estimation is obtained, and then the recognition is performed, the accurate recognition result of the target gesture can be obtained.

204. Integrate the channel characteristic data of each of the target signal segments to obtain the characteristic data of the channel estimation.

The integrated channel feature data, that is, the feature data of the channel estimation, is used as input for recognition, and a consistent target gesture recognition result can already be output.

The above content has clearly explained the data preprocessing that needs to be done for channel estimation. Next, it will be explained how to perform channel estimation on the target baseband signal.

The basic formula of channel estimation is:

R(n)=S[n]*h[n]

Where * represents convolution, R(n) and S[n] represent the received and demodulated target fundamental frequency signal respectively, the original fundamental frequency signal transmitted, h[n] is the channel response, the purpose of the algorithm is to pass R (n) and S[n] to obtain h[n].

The calculation complexity of the channel estimation algorithm is relatively high, so the calculation complexity is required in the calculation process. Therefore, in the solution process of the channel estimation of the present invention, the maximum likelihood estimation (Least Square) algorithm is used to simplify the computational complexity. In the LS algorithm, given the transmitted original audio signal S and the received target audio signal R, the channel h is calculated. According to the processing of the target audio signal, the signal length of a single cycle is set to d, so that the vector form of the transmitted signal S and the vector form of the received signal R are obtained:

S={s ₁ , s ₂ , ..., s _d-1 , s _d }

R={r ₁ , r ₂ ,..., r _d-1 , r _d }

Further construct the maximum likelihood estimation algorithm matrix:

MR求解出信道向量，其中(M^TM)^-1M的数值可以预先通过原始音频信号S求解得出，大大简化了算法的复杂度。Among them, the small scale l and p satisfy the formula l+p=d, and the algorithm complexity of directly calculating the matrix h is O(l ³ +lp ² ). The maximum likelihood estimation algorithm is used to directly use

MR solves the channel vector, wherein the value of (M ^T M) ^-1 M can be obtained by solving the original audio signal S in advance, which greatly simplifies the complexity of the algorithm.

For example, for the input original audio signal of 480 bits per cycle, the maximum likelihood estimation algorithm is used for calculation, and the channel estimation data with a length of 1=140 bits can be obtained. Usually the time of a single gesture movement is between 0.3 seconds and 1.4 seconds, so the system selects 1.4 seconds as the window length, and at this time, it can be calculated that one window contains 140 sets of channel estimation data. This constitutes the feature data for channel estimation, with a size of 140*140. Obviously, this calculation is much simpler than directly solving the channel estimation formula.

In the actual application process, different gesture amplitudes are large and small. When the palm is making fine-grained gestures, the channel change is small. If it is desired to clearly track channel changes, the static reflections in the reflected signal can be eliminated in the following manner.

In one embodiment, the specific step of static elimination includes: each of the target signal segments has a time stamp, and the channel feature data of each of the target signal segments is represented in the form of a vector, to obtain each of the target signal segments. The feature vector of the segment; the feature vectors of each of the target signal segments are arranged according to their respective timestamps to form a feature matrix; for each column in the feature matrix, the respective element values contained in the respective elements are subtracted. Each corresponding element value contained in the previous column is used to obtain the feature matrix after static elimination; the feature matrix after static elimination is determined as the feature data of the channel estimation. Identifying the feature data of channel estimation after static elimination can give accurate results even for fine-grained gestures.

Taking a set of feature data of gesture channel estimation as an example, the effect of static elimination is shown in Figure 4 and Figure 5, where Figure 4 shows the channel estimation feature data corresponding to the obtained channel estimation algorithm after maximum likelihood estimation. The heat map, it can be seen from the figure that there are many interfering signals doped in it, most of these signals are caused by static reflection; then look at Figure 5, which shows the characteristic data of channel estimation after static elimination. The heat map, obviously, there are very few interference signals at this time, and the enlarged image pointed out by the arrow is a heat map of the feature data of channel estimation generated by a gesture. It can be seen from the comparison of the two figures that even fine-grained gestures can be accurately obtained through static elimination.

So far, the feature data extraction part of the target gesture of audio-based gesture recognition has been completed: that is, the target audio signal is processed by demodulation, noise reduction, etc. to obtain the target fundamental frequency signal; using the periodic characteristics of the original audio, the target fundamental frequency signal Divide into multiple target signal segments with the same length as a single cycle of the original audio signal; use the single cycle of the original audio signal to do signal estimation with each target signal segment respectively to obtain their respective channel feature data; these channel feature data according to the time series After arranging to form a matrix, each column in the matrix is subtracted from each corresponding element value contained in the previous column to obtain a statically eliminated feature matrix, that is, the feature data of channel estimation. Using the fundamental frequency signal as the basis for channel estimation, the accuracy of the obtained channel estimation feature data has been significantly improved; after static elimination, even fine-grained gestures can accurately extract the corresponding channel estimation feature data for subsequent identification. The input, gesture recognition results are also more accurate.

103. Identify the feature data of the channel estimation to obtain a recognition result of the target gesture.

Identify the feature data of the channel estimation to obtain the recognition result of the target gesture. In the prior art, the basic convolutional neural network model can realize the recognition of preset gestures, but in the experimental observation, it is found that different testers have different habits of different preset gestures, such as gestures of waving up and down. The speed and magnitude of the gestures of the tested persons may be different. Therefore, the basic neural network model is used for gesture recognition. Although the neural network can correctly recognize gestures when given enough training data sets, this phenomenon will affect the recognition accuracy when the system is generalized.

In order to solve this practical problem, the present invention applies the domain adaptation network structure to the gesture recognition algorithm. The domain can be defined as the individual related to the data set generated in the recognition algorithm and irrelevant to the recognition purpose. gather. For example, in a gesture recognition system, the features brought by the gesture habits of different testers can be defined as domains, and the feature data introduced by different environments where the testers are located can also be defined as domains. The main function of domain adaptive network recognition is: when using a large number of data marked with the domain and the recognition result and a large number of data sets marked with the domain but not marked with the recognition result, it can realize the extraction of domain-independent eigenvalues and optimize them. recognition accuracy.

In one embodiment, the identifying the feature data of the channel estimation to obtain the identification result of the target gesture includes:

A pre-built domain adaptive neural network model is used to identify the feature data of the channel estimation, and the recognition result of the target gesture is obtained; specifically, the feature data of the channel estimation is firstly input into the pre-trained domain adaptive In the network model, the characteristics of the target gesture are extracted through a neural convolutional network; the characteristics of the target gesture are identified through a fully connected network layer and an activation function, and the recognition result of the target gesture is output; or the target gesture is obtained. For the recognition result, calculate the accuracy rate of the recognition result of the target gesture: when the accuracy rate falls within the range of a preset accuracy rate threshold, output the recognition result of the target gesture. When the accuracy rate falls within the preset accuracy rate threshold range, repeatedly extracting and recognizing the features of the target gesture for recognition, until the accuracy rate falls within the preset accuracy rate threshold value range, output the recognition result of the target gesture that meets the requirements .

In addition to the preset accuracy threshold range as the goal to improve the accuracy of recognition, the most fundamental method is to build a convergent domain adaptive neural network model. FIG. 6 shows the structure of the domain adaptive neural network model provided by this application.

6, in one embodiment, the domain adaptive neural network model is constructed by the following steps:

Obtain a first training data set and a second training data set, the first training data set includes multiple sets of channel estimation sample data with assigned domain labels and gesture labels, and the second training data set includes multiple sets of assigned domain labels Channel estimation sample data without assigning gesture labels;

Using the first training data set and the second training data set as training sets, the domain adaptive neural network model is obtained by training.

In one embodiment, using the first training data set and the second training data set as training sets, the training to obtain the domain adaptive neural network model includes:

Input the first training data set into the initial domain adaptive neural network model to obtain the gesture prediction result and domain prediction result of the first training data set; according to the assigned domain label and gesture label of the first training data set , and the gesture prediction result and domain prediction result of the first training data set, the first cross entropy E _label is obtained by calculation; the second training data set is input into the initial domain adaptive neural network model to obtain the first cross entropy E label ; 2. The gesture prediction result and the domain prediction result of the second training data set; according to the assigned domain label of the second training data set, and the gesture prediction result and the domain prediction result of the second training data set, the second cross entropy is obtained by calculating _Eunlab .

Based on the assigned domain labels of the first training dataset, the assigned domain labels of the second training dataset, the domain prediction results of the first training dataset and the domain of the second training dataset The prediction result is calculated to obtain the domain cross entropy E _s ; according to the first cross entropy, the second cross entropy and the domain cross entropy, the cross entropy of the initial domain adaptive neural network model is calculated E=E _label +αE _unlab- βE _s , wherein α and β are the model parameters of the domain adaptive neural network model; taking the minimum value of the cross entropy E as the target, the model parameters of the initial domain adaptive neural network model are optimized to obtain the obtained The domain adaptive neural network model.

The purpose of the present invention is gesture recognition, and all training data sets are assigned domain labels, one of which is assigned gesture labels, and the other is not assigned gesture labels. The influence of irrelevant features on the recognition results. During training, due to the small number of training samples, both new users and new environments can be regarded as new domains, so the data of each new user can be regarded as assigned domain labels (new domains), but no gestures are marked the second dataset.

Then, feature extraction is performed through the pre-built domain adaptive neural network model, gesture recognition and domain recognition are performed, and the first cross entropy, the second cross entropy and the domain cross entropy are respectively constructed. The cross entropy of the domain adaptive network model is further constructed. By learning the model, the weight of domain-specific feature data is reduced. This model has two advantages. First, it increases the weight of the domain-independent feature dataset and increases the accuracy. Secondly, for the data of new users, it can be directly input into the network for training, and the new model obtained will be better than the current model when used for other new users. In this way, even when the user uses the system for the first time, the system cannot know the user's personal behavior and environment in advance, but with the continuous addition of unlabeled gesture data from new users, the network will learn gesture recognition features that are suitable for all users. , while reducing the influence of the feature data generated by the environment and personal gesture habits on the recognition accuracy.

Specifically, to realize the calculation of the first cross-entropy and the second cross-entropy, the steps can be as follows: First, use a 5-layer convolutional neural network to perform feature extraction, that is, "input→[convolution→ReLu→pooling]*N→full In the structure of "connection layer→output", N is set to 5, and the structure of each convolutional layer is similar, with one convolutional layer and one pooling layer. The number of convolution kernels of the first, second, and third layers are set to 32, 64, 128, 256, and 512, respectively, and the convolution sizes are [10*10], [5*5], [5*5], [ 5*5], [2*2], and finally extract the features from the two input training data sets, define δ as the relevant parameter, X is the input training data set, and the extracted feature F can be defined, and the specific relationship is :

F=CNN(X,δ)

The extracted feature F is obtained through the fully connected layer to obtain the parameter Z, and W _a1 and b _a1 are defined as the parameters of the fully connected network layer, and the formula is solved through the learning process of the neural network:

Z=softplus(W _a1 F+b _a1 )

定义W_a2与b_a2为预测层中的参数，通过神经网络学习过程求解，得到下述公式：The fully connected result Z obtains the prediction result through the softmax layer

Define W _a2 and b _a2 as the parameters in the prediction layer, and solve them through the neural network learning process, and obtain the following formula:

表示标注了手势的第一训练数据集和未标注手势的第二训练数据集提取的预测结果集合，依据训练数据集拆分

得到

和

分别表示第一训练数据集提取得到的手势预测结果，和第二训练数据集提取得到的手势预测结果，同理，划分输入数据为X^label和X^unlab。in,

Represents the set of prediction results extracted from the first training dataset with gestures and the second training dataset without gestures, split according to the training dataset

get

and

respectively represent the gesture prediction result extracted from the first training data set and the gesture prediction result extracted from the second training data set. Similarly, the input data is divided into X ^label and X ^unlab .

可以计算得到交叉熵(cross entropy)E_label，Knowing the true label y ^label and the predicted label

The cross entropy (cross entropy) E _label can be calculated,

Where |X ^label | represents the number of the first training data set for marking gestures, and N represents the number of gesture actions to be recognized.

预测的手势结果计算交叉熵E_unlab，For the second training dataset, which is not labeled with real gestures, only use

The predicted gesture result calculates the cross entropy E _unlab ,

|X ^unlab | represents the number of unlabeled gestures in the second training dataset.

After completing the calculation of the gesture-related cross-entropy, the domain-related cross-entropy is calculated next. The specific steps include that the input S of the domain discrimination network part is defined as the concatenation of the feature extraction part and the gesture recognition result,

是识别部分的结果，⊙表示串联。参数F表示训练数据集中标注了手势的数据和未标注手势的数据的集合，域辨别网络由两层全连接层构成，定义第一层全连接层输出为T:where F is the result of the feature proposal part, which contains a first training dataset of labeled gestures and a second training dataset of unlabeled gestures;

is the result of the recognition part, and ⊙ means concatenation. The parameter F represents the set of data marked with gestures and data not marked with gestures in the training data set. The domain discrimination network consists of two fully connected layers, and the output of the first fully connected layer is defined as T:

T=Softplus(W _s1 S+B _s1 )

其中W_s1，W_s2和B_s1，B_s2为全连接网络需要学习的参数，W是权值矩阵，B是偏置，下标代表层数。在域识别网络结构中，定义交叉熵E_s作为损失函数，其中S_ij是one-hot形式的域标签向量，|X|表示所有属于S的样本个数。The output of the second fully connected network layer

Among them, W _s1 , W _s2 and B _s1 , B _s2 are the parameters to be learned by the fully connected network, W is the weight matrix, B is the bias, and the subscript represents the number of layers. In the domain recognition network structure, the cross-entropy E _s is defined as the loss function, where S _ij is the domain label vector in one-hot form, and |X| represents the number of all samples belonging to S.

For ease of understanding, examples are given. During training, it is assumed that there are 10,000 groups of 140*140 training data assigned domain labels and gesture labels to form the first training data set, and 1000 groups of 140*140 training data that are assigned domain labels but not assigned gesture labels constitute the second training data set. First, the two training data sets are passed through the convolutional neural network for feature extraction, and the fully connected layer and the activation function are added after the convolutional layer, and the predicted result will be obtained.

It should be noted that for each set of input data, the result of annotation prediction can be obtained through the model. Then, for the first training data set, 10,000 gesture prediction results can be obtained, and the first cross entropy E _label can be calculated from the 10,000 gesture prediction results and the original 10,000 actual gestures. For the second training data set, only 1000 gesture prediction results can be obtained. Since there are no real gestures, assuming 1000 gesture prediction results are the original 1000 actual gestures, 1000 gesture prediction results and 1000 gesture prediction results can be calculated The second cross entropy function E _label is obtained.

The features extracted by the convolutional neural network are the data after flatten, that is, one-dimensional data. The flatten here can be understood as dimensionality reduction. Assuming that there is a 2*2*500 structure left after pooling, the length after flatten is 2000 one-dimensional data. Therefore, after feature extraction is performed on the first training data set or the second training data set, one-dimensional data is obtained.

Assuming that a set of one-dimensional data is 512 bits, the first training data set corresponds to 8 sets of gestures, and the data result of the gesture recognition part is an 8-bit one-dimensional array (1, 0, 0, 0) in the form of onehot (one-hot encoding). , 0, 0, 0, 0), the first 1 in the one-dimensional array represents the first gesture. In this way, data with a group length of 512 bits can be easily concatenated with these 8 pieces of data to obtain data with a length of 520. At this time, only a simple full connection and activation function are needed to identify the domain. For all training datasets, whether labeled or unlabeled, the domain has been given, so the domain recognition cross-entropy Es can be directly calculated. Combined with the E _label and E _label calculated above, the cross entropy of the domain adaptive neural network model is combined.

E=E _label +αE _unlab -βE _s

代表为计算这两组交叉熵时需要优化的参数集合，同时定义

为神经网络模型域识别部分计算域交叉熵E_s时需要优化的参数集合，以及神经网络识别结构最终实现域自适应识别需要优化的参数集合

系统构建新的交叉熵E实现对参数集合

的学习，其中α和β域自适应神经网络的模型参数，Knowing the loss function E of the entire network, the network parameters can be trained directly using the gradient descent algorithm. Using the system to obtain the cross entropy E _label of the first training data set and the cross entropy E _unlab of the second training data set, define

Represents the set of parameters that need to be optimized when calculating the two sets of cross entropy, and defines

The parameter set that needs to be optimized when calculating the domain cross entropy E _s for the domain identification part of the neural network model, and the parameter set that needs to be optimized for the final realization of domain adaptive identification by the neural network identification structure

The system constructs a new cross-entropy E to realize the set of parameters

learning, where the α and β domains adapt the model parameters of the neural network,

E=E _label +αE _unlab -βE _s

和

这三组参数的集合，即当给定合适的α和β两个模型参数时，令E的值最小。The basic idea of the training part of the neural network is to find the final

and

The set of these three sets of parameters, that is, when the appropriate two model parameters, α and β, are given, the value of E is minimized.

Specifically, during training, the theoretical formula calculated by the gradient descent method used is as follows:

where μ is the learning rate.

Due to the very high energy consumption of real-time system design, it is impossible to always call the neural network model for recognition. Therefore, motion detection can also be designed in real-time systems.

It can be seen from the foregoing that when channel estimation is performed in real time, channel data with a size of 140 bits is estimated for every 480 bits of data input. It has been proved by experiments that when there is palm movement, the variance of the 140-bit channel characteristic data h is very large, where h has the same meaning as above, and here refers to the target signal frequency band with a period length of 480 bits and the original audio frequency of 480 bits. The channel characteristic data obtained after channel estimation is performed on the signal. Referring to FIG. 7 , the figure shows the reference graph for calling the neural network model threshold value determination, so the variance of h is used as the threshold value determination condition. When the h-variance at a certain moment is greater than 1/10 of the average variance data, the system judges that there is a suspected gesture, gives the number of features of the channel estimation in the first 1.4s, and calls the neural network model for identification, which can avoid calling the neural network model all the time. The problem of high energy consumption of equipment. Of course, the model may be a common neural network model, a domain adaptive neural network model or other applicable network models, and the type of the network model is not limited here.

The gesture recognition method disclosed in the present invention realizes the feature data extraction of the channel estimation of the target gesture based on the channel estimation of the fundamental frequency signal, and recognizes the feature data of the channel estimation through the pre-built domain adaptive neural network model. The feature data extraction part can obtain accurate data extraction results, and the use of the domain adaptive neural network model for identification can reduce the domain feature weight irrelevant to the identification results, thereby improving the accuracy of identification and the generalization of identification. The provided scheme has broader application prospects.

However, although the aforementioned solutions have been able to significantly improve the accuracy of gesture recognition and the generalization of system recognition, there is still a problem, that is, when the system only uses a single microphone and a single speaker, the obtained target audio signal can only be calculated in one dimension. Characteristic data for channel estimation. If the two gesture actions are mirrored, the system will not recognize it, such as gesture clockwise rotation and gesture counterclockwise rotation.

In order to solve the problem that a single microphone device cannot recognize the mirror gesture when collecting the target audio signal, in one embodiment, when acquiring the target audio signal, N parts of the target audio signal collected by N different microphone devices are selected, and N is an integer greater than 1; the performing channel estimation based on the original audio signal and the target audio signal, and obtaining the characteristic data of the channel estimation includes: for each copy of the target audio signal, performing the channel estimation with the original audio signal respectively Channel estimation is performed to obtain N pieces of channel characteristic data; the N pieces of channel characteristic data are integrated to obtain the characteristic data of the channel estimation. Using multiple audio acquisition devices to perform multi-dimensional acquisition and processing of target audio signals can obtain more feature data for channel estimation, recognize mirror gestures, and improve gesture recognition results.

Obviously, when channel estimation is performed using the target audio signals received by multiple sets of microphones, more feature information can be obtained. For example, assuming that the number of microphones is M, when the user makes a single gesture, the system will obtain M independent channel estimation feature data. At this time, the size of the channel estimation feature data corresponding to a single gesture is 140*140*M; further Gesture recognition can be performed from multiple dimensions, distinguish mirror gestures, and obtain more accurate gesture recognition results.

FIG. 8 shows a structural block diagram of an audio-based gesture recognition apparatus provided by an embodiment of the present application. For convenience of description, only parts related to the embodiment of the present application are shown.

8, in one embodiment, the audio-based gesture recognition device includes:

A signal acquisition module 301, configured to acquire a target audio signal, the target audio signal is an audio signal received after a preset original audio signal is modulated and propagated through a target gesture made by a user;

a channel estimation module 302, configured to perform channel estimation based on the original audio signal and the target audio signal to obtain characteristic data of the channel estimation;

The gesture recognition module 303 is configured to recognize the feature data of the channel estimation to obtain the recognition result of the target gesture.

In one embodiment, the channel estimation module 302 may include:

a signal modulation unit for demodulating the target audio signal to obtain a target fundamental frequency signal;

a signal segmenting unit, configured to segment the target fundamental frequency signal to obtain a plurality of target signal segments, and the length of each target signal segment is the same as the period of the original audio signal;

a channel estimation unit, configured to perform channel estimation on each of the target signal segments with a period of signal segments in the original audio signal to obtain respective channel characteristic data;

The data integration unit is configured to integrate the channel characteristic data of each of the target signal segments to obtain the characteristic data of the channel estimation.

In one embodiment, the signal conditioning unit may include:

a signal decomposition sub-unit, used to perform down-carrier and IQ decomposition on the target audio signal

, obtain the real part signal and imaginary part signal of the down-carrier signal;

A signal denoising subunit, configured to use a low-pass filter to denoise the real part signal and the imaginary part signal of the down-carrier signal to obtain the target fundamental frequency signal.

In one embodiment, each of the target signal segments is time stamped, and the data integration unit may include:

a data conversion subunit, used to represent the channel feature data of each of the target signal segments as a vector, respectively, to obtain a feature vector of each of the target signal segments;

a data arrangement subunit, used for arranging the feature vectors of each of the target signal segments according to their respective time stamp sizes to form a feature matrix;

The static elimination subunit is used for each column in the feature matrix to deduct the respective element values contained in the respective element values from the corresponding element values contained in the previous column to obtain the statically eliminated feature matrix; the static elimination The latter feature matrix is determined as feature data of the channel estimation.

In one embodiment, gesture recognition module 303 may include:

a gesture recognition unit, configured to use a pre-built domain adaptive neural network model to recognize the feature data of the channel estimation, and obtain the recognition result of the target gesture;

Wherein, the domain adaptive neural network model in the gesture recognition unit is constructed by the following units:

A model building unit, configured to obtain a first training data set and a second training data set, the first training data set includes multiple groups of channel estimation sample data that have been assigned domain labels and gesture labels, and the second training data set includes Multiple sets of channel estimation sample data with domain labels assigned but no gesture labels assigned;

A model training unit, configured to use the first training data set and the second training data set as training sets to train to obtain the domain adaptive neural network model.

In one embodiment, the model training unit may include:

a first training subunit, configured to input the first training data set into an initial domain adaptive neural network model to obtain the gesture prediction result and the domain prediction result of the first training data set;

The first calculation subunit is used to calculate the first cross entropy E _label according to the assigned domain label and gesture label of the first training data set, as well as the gesture prediction result and the domain prediction result of the first training data set ;

a second training subunit, configured to input the second training data set into the initial domain adaptive neural network model to obtain the gesture prediction result and the domain prediction result of the second training data set;

The second calculation subunit is used to calculate the second cross entropy E _unlab according to the assigned domain label of the second training data set, and the gesture prediction result and the domain prediction result of the second training data set;

The third calculation subunit is configured to be configured according to the assigned domain label of the first training data set, the assigned domain label of the second training data set, the domain prediction result of the first training data set, and the The domain prediction result of the second training data set is described, and the domain cross entropy E _s is obtained by calculation;

The fourth calculation subunit is used to calculate the cross entropy of the initial domain adaptive neural network model according to the first cross entropy, the second cross entropy and the domain cross entropy E=E _label +αE _unlab -βE _s , where α and β are the model parameters of the domain adaptive neural network model;

The model optimization subunit is configured to optimize the model parameters of the initial domain adaptive neural network model with the goal of taking the minimum value of the cross entropy E to obtain the domain adaptive neural network model.

In one embodiment, the signal acquisition module 301 may include:

A multi-dimensional signal acquisition unit, configured to acquire N parts of the target audio signal collected by N different microphone devices, where N is an integer greater than 1;

The channel estimation module may include:

A multi-dimensional channel estimation unit, configured to perform channel estimation with the original audio signal for each of the target audio signals to obtain N pieces of channel feature data;

A multi-dimensional data integration unit, configured to integrate the N pieces of channel characteristic data to obtain the characteristic data of the channel estimation.

Embodiments of the present application further provide a terminal device, including a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor implements the computer program as described herein when executing the computer program. The steps of each audio-based gesture recognition method proposed by the application.

Embodiments of the present application also provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, implements the audio-based gesture recognition methods proposed in the present application. step.

Embodiments of the present application also provide a computer program product, which, when the computer program product runs on a terminal device, enables the terminal device to execute the steps of each audio-based gesture recognition method proposed in the present application.

FIG. 9 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As shown in FIG. 9 , the terminal device 4 in this embodiment includes: at least one processor 40 (only one is shown in the figure), a processor, a memory 41, and a processor stored in the memory 41 and available in the at least one processor A computer program 42 running on the processor 40, when the processor 40 executes the computer program 42, implements the steps in any of the above-mentioned embodiments of the audio-based gesture recognition method.

The terminal device 4 may be a computing device such as a desktop computer, a notebook, a palmtop computer and a cloud server, and a wearable device such as a smart watch and a smart bracelet. The terminal device may include, but is not limited to, the processor 40 and the memory 41 . Those skilled in the art can understand that FIG. 9 is only an example of the terminal device 4, and does not constitute a limitation on the terminal device 4. It may include more or less components than the one shown, or combine some components, or different components , for example, may also include input and output devices, network access devices, and the like.

The so-called processor 40 may be a central processing unit (Central Processing Unit, CPU), and the processor 40 may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (Application Specific Integrated Circuits) , ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 41 may be an internal storage unit of the terminal device 4 in some embodiments, such as a hard disk or a memory of the terminal device 4 . In other embodiments, the memory 41 may also be an external storage device of the terminal device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card) and so on. Further, the memory 41 may also include both an internal storage unit of the terminal device 4 and an external storage device. The memory 41 is used to store an operating device, an application program, a boot loader (Boot Loader), data, and other programs, such as program codes of the computer program, and the like. The memory 41 can also be used to temporarily store data that has been output or will be output.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. Module completion, that is, dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit, and the above-mentioned integrated units may adopt hardware. It can also be realized in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application. For the specific working process of the units and modules in the above apparatus, reference may be made to the corresponding process in the foregoing method embodiments, and details are not described herein again.

In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be Incorporation may either be integrated into another device, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the present application realizes all or part of the processes in the methods of the above embodiments, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When executed by a processor, the steps of each of the above method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to the terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media. For example, U disk, mobile hard disk, disk or CD, etc. In some jurisdictions, under legislation and patent practice, computer readable media may not be electrical carrier signals and telecommunications signals.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the above-mentioned embodiments, those of ordinary skill in the art should understand that: it can still be used for the above-mentioned implementations. The technical solutions described in the examples are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the application, and should be included in the within the scope of protection of this application.

Claims (10)

1. An audio-based gesture recognition method, comprising:

acquiring a target audio signal, wherein the target audio signal is an audio signal received after a preset original audio signal is modulated and is transmitted through a target gesture made by a user;

performing channel estimation based on the original audio signal and the target audio signal to obtain characteristic data of channel estimation;

and identifying the characteristic data of the channel estimation to obtain an identification result of the target gesture.

2. The gesture recognition method according to claim 1, wherein the original audio signal is a periodic signal, and the performing channel estimation based on the original audio signal and the target audio signal to obtain the characteristic data of the channel estimation comprises:

Demodulating the target audio signal to obtain a target base frequency signal;

segmenting the target base frequency signal to obtain a plurality of target signal segments, wherein the length of each target signal segment is the same as the period of the original audio signal;

for each target signal segment, respectively performing channel estimation with a signal segment of one period in the original audio signal to obtain respective channel characteristic data;

and integrating the channel characteristic data of each target signal segment to obtain the characteristic data of the channel estimation.

3. The gesture recognition method of claim 2, wherein the demodulating the target audio signal to obtain a target baseband signal comprises:

carrying out carrier reduction and IQ decomposition on the target audio signal to obtain a real part signal and an imaginary part signal of the carrier reduction signal;

and denoising the real part signal and the imaginary part signal of the carrier-reduced signal by using a low-pass filter to obtain the target fundamental frequency signal.

4. The gesture recognition method according to claim 2, wherein each of the target signal segments has a time stamp, and the integrating the channel characteristic data of the respective target signal segments to obtain the characteristic data of the channel estimation comprises:

Respectively representing the channel characteristic data of each target signal segment into a vector form to obtain a characteristic vector of each target signal segment;

arranging the eigenvectors of the target signal segments according to the sizes of the corresponding timestamps to form an eigenvector matrix;

for each column in the feature matrix, subtracting each corresponding element value contained in the previous column from each element value contained in the feature matrix to obtain the feature matrix after static elimination;

and determining the feature matrix after static elimination as feature data of the channel estimation.

5. The gesture recognition method according to claim 1, wherein the recognizing the feature data of the channel estimation to obtain the recognition result of the target gesture comprises:

recognizing the characteristic data of the channel estimation by adopting a pre-constructed domain adaptive neural network model to obtain a recognition result of the target gesture;

wherein the domain adaptive neural network model is constructed by the following steps:

acquiring a first training data set and a second training data set, wherein the first training data set comprises a plurality of groups of channel estimation sample data of distributed domain labels and gesture labels, and the second training data set comprises a plurality of groups of channel estimation sample data of distributed domain labels but not distributed gesture labels;

And training to obtain the domain adaptive neural network model by taking the first training data set and the second training data set as training sets.

6. The gesture recognition method of claim 5, wherein training the domain-adaptive neural network model using the first training data set and the second training data set as training sets comprises:

inputting the first training data set into an initial domain adaptive neural network model to obtain a gesture prediction result and a domain prediction result of the first training data set;

calculating to obtain a first cross entropy E according to the assigned domain label and the assigned gesture label of the first training data set and the gesture prediction result and the domain prediction result of the first training data set_label；

Inputting the second training data set into the initial domain adaptive neural network model to obtain a gesture prediction result and a domain prediction result of the second training data set;

calculating to obtain a second cross entropy E according to the assigned domain label of the second training data set and the gesture prediction result and the domain prediction result of the second training data set_unlab；

Calculating to obtain a domain cross entropy E according to the assigned domain label of the first training data set, the assigned domain label of the second training data set, the domain prediction result of the first training data set and the domain prediction result of the second training data set _s；

Calculating the cross entropy E-E of the initial domain adaptive neural network model according to the first cross entropy, the second cross entropy and the domain cross entropy_label+αE_unlab-βE_sWherein alpha and beta are model parameters of the domain adaptive neural network model;

and optimizing the model parameters of the initial domain self-adaptive neural network model by taking the minimum value of the cross entropy E as a target to obtain the domain self-adaptive neural network model.

7. The gesture recognition method according to any one of claims 1 to 6, wherein the acquiring a target audio signal includes:

acquiring N parts of target audio signals respectively acquired by N different microphone devices, wherein N is an integer greater than 1;

the performing channel estimation based on the original audio signal and the target audio signal to obtain feature data of channel estimation includes:

for each target audio signal, respectively carrying out channel estimation on the target audio signal and the original audio signal to obtain N parts of channel characteristic data;

and integrating the N parts of channel characteristic data to obtain the characteristic data of the channel estimation.

8. An audio-based gesture recognition apparatus, comprising:

the signal acquisition module is used for acquiring a target audio signal, wherein the target audio signal is an audio signal which is received after a preset original audio signal is modulated and is transmitted through a target gesture made by a user;

The channel estimation module is used for carrying out channel estimation based on the original audio signal and the target audio signal to obtain characteristic data of channel estimation;

and the gesture recognition module is used for recognizing the characteristic data of the channel estimation to obtain a recognition result of the target gesture.

9. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the gesture recognition method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the gesture recognition method according to any one of claims 1 to 7.

Images