CN113362840B - General voice information recovery device and method based on undersampled data of built-in sensor - Google Patents

Abstract

The invention discloses a general voice information recovery device and method based on built-in sensing undersampling data. The device includes a signal preprocessing module, a fundamental frequency estimation module, a spectrum reconstruction module and a spectrum voice conversion module. The frequency estimation module, the spectral reconstruction module and the spectral speech conversion module are connected in sequence, which can not only recover the speech information in the sensor data with extremely narrow bandwidth and serious aliasing, but also solve the problem of poor transferability of the learning-based model. The data collected by the built-in sensor of the mobile phone has different characteristics in different scenarios. The present invention directly constructs a voice information recovery system without using the data set for model training based on the inherent characteristics of the sensor data and the characteristics of the voice signal. It can adapt to changes in users, environments and devices, and effectively recover hidden voice signals from the built-in sensors of mobile phones.

Description

Universal voice information recovery device and method based on built-in sensor undersampled data

technical field

The invention discloses a general voice information restoration device and method based on built-in sensing undersampling data.

Background technique

With the development of smartphone assistants, voice has become more and more popular in human-computer interaction, and has even gradually become the first choice for the blind, the elderly, children and other special groups. As a result, more and more IoT devices and mobile devices are deploying voice assistants. For example, on mobile devices, there's Apple's Siri, Google's Assistant, and Samsung's Bixby; on smart speakers, there's Amazon's Alexa and Google's Google Home; on traditional PC devices, there's Apple's Siri and Microsoft's Microsoft Cortana. Research shows that by 2023, the global market value of voice assistants will reach about 7.8 billion US dollars. However, due to the personalized service nature of voice interaction, in the process of human-computer interaction, some sensitive information is usually embedded in the voice, such as the verbal password used for authentication and the voice content during chatting. When smart devices use speakers to play these voices, they inevitably leak sensitive information. The invention utilizes the built-in accelerometer, gyroscope and magnetometer of the mobile phone to collect the vibration signal and magnetic field signal leaked when the speaker plays the voice, and recover the implicit voice information.

Disadvantages of the prior art:

When the mobile phone speaker is playing voice, the built-in sensors (accelerometer, gyroscope, magnetometer) of the mobile phone can be used to collect the signal leaked by the speaker. There are two main ways to recover high-resolution voice information from low-resolution sensor data. The first is to use voice super-resolution technology to convert low-bandwidth sensor signals into high-bandwidth voice signals. However, the bandwidth of sensor data is extremely narrow and there is serious aliasing, and speech super-resolution technology cannot effectively recover the speech information in it. The second is to use the speech restoration technology based on learning (machine learning, deep learning). However, this method has two major disadvantages. First, it inevitably introduces the collection and labeling of sensor data, and it takes a lot of time. Perform model training. Second, the performance of the learning-based sensor data speech recovery model will drop significantly when migrated to different subjects, environments, and devices.

The present invention provides a model-driven sensor data voice information restoration device, which saves the model training process, can effectively restore hidden voice information from sensor data, and maintains sufficiently high robustness.

SUMMARY OF THE INVENTION

The present invention has made improvements in view of the deficiencies of the prior art, and provides a general voice information recovery device and method based on built-in sensor undersampling data, and the present invention is realized through the following technical solutions:

The invention discloses a general speech information recovery device based on built-in sensing under-sampling data. The device includes a signal preprocessing module, a fundamental frequency estimation module, a spectrum reconstruction module and a spectrum speech conversion module. The module, the spectrum reconstruction module and the spectrum speech conversion module are connected in sequence.

As a further improvement, the signal preprocessing module of the present invention: used to collect the data of the mobile phone accelerometer, gyroscope and magnetometer, and sent the collected sensor data into the high-pass filter; the fundamental frequency estimation module: used for Estimating the fundamental frequency of speech signals implicit in sensor data; spectrum reconstruction module: used to reconstruct high-frequency harmonics, correct abnormal low-frequency harmonics, and restore low-resolution spectrum to high-resolution spectrum; spectrum Speech conversion module: It is used to convert the recovered high-resolution spectrum into a human-audible speech signal using the Griffin-Lim algorithm.

The invention also discloses a recovery method using a general voice information recovery device based on built-in sensing undersampling data, which is characterized by:

1) Through the signal preprocessing module, collect the data of the z-axis of the mobile phone accelerometer, the y-axis of the gyroscope, and the z-axis of the magnetometer, and then send the collected sensor data into a high-pass filter to filter out meaningless low-frequency noise;

2) Through the fundamental frequency estimation module, the fundamental frequency estimation algorithm based on aliasing is used to estimate the fundamental frequency. The algorithm can simultaneously consider the normal harmonics in the speech signal and the abnormal harmonics generated when the signal is undersampled, and estimate the hidden frequency in the sensor data. The fundamental frequency of the included speech signal;

3) According to the estimated fundamental frequency, through the spectrum reconstruction module, the harmonics of the high frequency band are reconstructed and the abnormal harmonics of the low frequency band are corrected, and the low-resolution spectrum is restored to a high-resolution spectrum;

4) Through the spectral speech conversion module, the recovered high-resolution frequency spectrum is converted into a speech signal audible to the human ear by using the Griffin-Lim algorithm.

As a further improvement, in step 2) of the present invention, the fundamental frequency estimation module uses short-time Fourier transform to convert the filtered sensor time-domain signal into an amplitude spectrum M(t, f), when the original When the frequency of the signal is greater than half of the sampling rate of the sensor, the signal actually collected by the sensor will be aliased due to undersampling. The relationship between the frequency change before and after the signal undersampling is:

f is the original frequency, SR is the sampling rate, and A(f) is the changed frequency;

Use aliasing-based harmonic addition to measure the likelihood H(f) that frequency f is fundamental:

的最高阶谐波的阶数，m代表频率低于1250Hz的最高阶谐波的阶数。Among them, M(t, f) is the amplitude spectrum of the sensor data, t is the frame number, f is the frequency, k is the order of the harmonic, and n is the frequency lower than

The order of the highest-order harmonic of , m represents the order of the highest-order harmonic whose frequency is lower than 1250Hz.

As a further improvement, due to the frequency change in the present invention, there are normal harmonics in the corresponding speech signal and abnormal harmonics generated by undersampling in the amplitude spectrum M(t, f).

根据H(f)越大，f是基频可能性的越大的特点频谱中每一帧的基频f_p为

同时考虑频谱中的正常谐波和为欠采样生成的异常谐波这两个部分能够提高基频估计的准确性，其中85Hz到255Hz代表大部分成年人说话时的基频范围。As a further improvement, in H(f) described in the present invention, the former term accumulates the energy of normal harmonics in the spectrum, the latter term accumulates the energy of abnormal harmonics generated by undersampling, and each frame in the spectrum accumulates the energy of abnormal harmonics. The fundamental frequency f _p of

According to the characteristic that the larger H(f) is, the greater the possibility of f is the fundamental frequency, the fundamental frequency f _p of each frame in the spectrum is

Considering both the normal harmonics in the spectrum and the abnormal harmonics generated for undersampling can improve the accuracy of the fundamental frequency estimate, where 85Hz to 255Hz represents the fundamental frequency range in which most adults speak.

As a further improvement, in step 3), the spectrum of the sensor signal is reconstructed by the spectrum reconstruction module, the bandwidth of the sensor's amplitude spectrum is expanded by the super-resolution algorithm of aliasing correction, and the reconstructed amplitude spectrum is recorded as M _new (t, f ) (initially a zero matrix), the original amplitude spectrum is M _old (t, f), and the specific steps are:

a. The algorithm traverses each frame of the original spectrum M _old (t, f). In each round of traversal, the fundamental frequency estimation module is used to estimate the size of the fundamental frequency f _p , and then according to the integral of the fundamental frequency and the harmonic frequency Multiple relationship, get the frequency kf _p of each harmonic;

之间的频谱，语音谐波的频率为

将M_old(t，kf_p)直接赋值给M_new(t，kf_p)；(以保留低频段的正常语音谐波，并且去除低频段的混叠。)对于频率处于

到F_endHz之间的频谱，语音谐波的频率为

F_end为重建频谱的最高频率；b. The algorithm reconstructs the spectrum, for frequencies from 0Hz to

The frequency spectrum between the speech harmonics is

M _old (t, kf _p ) is directly assigned to M _new (t, kf _p ); (to preserve the normal speech harmonics of the low frequency band and remove the aliasing of the low frequency band.) For frequencies in

to F _end Hz, the frequency of the speech harmonics is

F _end is the highest frequency of the reconstructed spectrum;

到F_endHz)所在的位置以及能量的大小，原始正常语音谐波的频率为kf_pHz，由于欠采样，其频率以公式(1)转变成A(kf_p)Hz；c. Estimate this part of the harmonics (frequency is

To the position of F _end Hz) and the magnitude of the energy, the frequency of the original normal speech harmonic is kf _p Hz, and due to undersampling, its frequency is transformed into A(kf _p ) Hz by formula (1);

d. Replace the unknown normal harmonic spectrum M _new (t, kf _p ) with the known aliased harmonic spectrum M _old (t, A(kf _p )), in completing the pair M _old (t, f) After traversing each frame t, the system generates the reconstructed magnitude spectrum M _new (t, f).

As a further improvement, in step 4) described in this aspect, the spectrum-to-speech conversion module uses the Griffin-Lim algorithm to recover the audible speech signal based on the reconstructed spectrum M _new (t, f), and the algorithm passes N times The speech signal is estimated iteratively from the reconstructed spectrum _Mnew (t,f).

As a further improvement, the concrete steps of step 4) of the present invention are:

e. The Griffin-Lim algorithm randomly generates a phase spectrum P ₀ , and then uses the inverse short-time Fourier transform to convert the phase spectrum P ₀ and the amplitude spectrum M _new (t, f) into a speech signal x ₀ ;

由于幅度频谱

与重建频谱M_new(t，f)存在一定的区别，算法只保留相位频谱P₁，并将P₁送入下一次迭代过程；f. Do a short-time Fourier transform on the speech signal x ₀ to obtain the phase spectrum P ₁ and the amplitude spectrum

Due to the magnitude spectrum

There is a certain difference from the reconstructed spectrum M _new (t, f), the algorithm only retains the phase spectrum P ₁ , and sends P ₁ to the next iteration process;

与重建频谱M_new(t，f)足够相似，Griffin-Lim算法利用给定的重建频谱M_new(t，f)生成了对应的语音信号。The gf and Griffin-Lim algorithms continuously modify the phase spectrum P _i through N iterations until the generated amplitude spectrum

Similar enough to the reconstructed spectrum M _new (t, f), the Griffin-Lim algorithm generates the corresponding speech signal with the given reconstructed spectrum M _new (t, f).

The beneficial effects of the present invention are as follows:

The invention constructs a sensor data hidden information recovery device without training, which can not only recover the speech information in the sensor data with extremely narrow bandwidth and serious aliasing, but also solves the problem of poor transferability of the learning-based model. The data collected by the built-in sensor of the mobile phone has different characteristics in different scenarios. Specifically, the speaker voice signal captured by the built-in sensor of the mobile phone may come from different users and have different text content; the built-in sensor of the mobile phone collects data It may be in different environments, and the noise and arrangement in the environment will affect the quality of the data collected by the sensor; the data collected by the built-in sensors of different brands and models of mobile phones will have different characteristics. These three situations will cause the data collected by the built-in sensors of the mobile phone to not have a unified feature, making it difficult for data-driven machine learning and deep learning methods to train a model with high robustness to recover the speech implicit in the sensor data. information. However, the present invention directly constructs a voice information recovery system based on the inherent characteristics of sensor data and the characteristics of voice signals, without using data sets for model training, and can adapt to changes in users, environments and equipment, and effectively build a voice information system from mobile phones. The hidden voice signal is recovered from the sensor.

(传感器采样率的一半)，本发明不仅在高频段(

到1000Hz)上重建了缺失的语音谐波频谱，而且消除了低频段(0Hz到

)上存在的混叠，使得重建的频谱与原始的语音频谱更加相似，从而提高了语音信息恢复的质量。本发明能够取得比较低的LSD，与传统语音超分辨率方法相比，能够更有效的恢复低分辨率传感器数据中的语音信息。另外，本发明能够在不同的设备上有效的运行，且去除用户交互对于传感器测量数据的干扰，实现有效的语音信息恢复。In the spectral reconstruction module, the present invention expands the bandwidth of the raw sensor data amplitude spectrum using an alias-corrected super-resolution algorithm. The bandwidth of the raw sensor data spectrum is

(half of the sensor sampling rate), the present invention is not only in the high frequency band (

to 1000Hz), the missing speech harmonic spectrum is reconstructed, and the low frequency band (0Hz to 1000Hz) is removed.

), which makes the reconstructed spectrum more similar to the original speech spectrum, thereby improving the quality of speech information recovery. Compared with the traditional voice super-resolution method, the present invention can obtain relatively low LSD, and can more effectively restore the voice information in the low-resolution sensor data. In addition, the present invention can operate effectively on different devices, remove the interference of user interaction on sensor measurement data, and realize effective voice information recovery.

来衡量t时刻f是基频的可能性大小，并用

来确定基频。其中

将正常谐波的能量累加，

将异常谐波的能量累加。这样不仅考虑到了正常的语音谐波，也考虑到了异常的混叠谐波，提高了基频估计的准确性。这一精确的基频估计结果能够有效的提高语音恢复的质量。Due to the serious aliasing of the undersampled sensor data, there are not only normal harmonics, but also abnormal aliasing harmonics (high-frequency speech harmonics are undersampled into low-frequency abnormal harmonics) in the sensor data. Find the correct fundamental frequency of speech. Therefore, in the fundamental frequency estimation module, the present invention uses the aliasing-based fundamental frequency estimation algorithm to estimate the magnitude of the fundamental frequency, that is, using

to measure the possibility that f is the fundamental frequency at time t, and use

to determine the fundamental frequency. in

Adding up the energy of the normal harmonics,

Accumulates the energy of abnormal harmonics. In this way, not only the normal speech harmonics, but also the abnormal aliasing harmonics are considered, and the accuracy of the fundamental frequency estimation is improved. This accurate fundamental frequency estimation result can effectively improve the quality of speech recovery.

Description of drawings

1 is a system block diagram of the present invention;

Figure 2 is a comparison diagram of the response of the accelerometer to the speech signal leaked by the speaker in different scenarios;

Fig. 3 is the amplitude spectrogram of magnetometer signal, real speech signal and reconstructed speech signal;

Fig. 4 is the overall performance comparison diagram of the present invention;

5 is a performance comparison diagram of the present invention for different use subjects;

6 is a performance comparison diagram of the present invention for different text contents;

7 is a performance comparison diagram of the present invention on different devices;

8 is a performance comparison diagram of the present invention under different environments;

FIG. 9 is a performance comparison diagram of the present invention under different interaction modes.

Detailed ways

The present invention discloses a general speech information recovery device and method based on built-in sensing undersampling data. Fig. 1 is a system block diagram of the present invention, which includes four parts, namely, a signal preprocessing module, a fundamental frequency estimation module, and a frequency spectrum. The reconstruction module and the spectral-voice conversion module, the signal preprocessing module, the fundamental frequency estimation module, the spectral reconstruction module and the spectral-voice conversion module are connected in sequence.

In the signal preprocessing, the data of the z-axis of the mobile phone's accelerometer, the y-axis of the gyroscope, and the z-axis of the magnetometer are first collected, and then the collected sensor data is sent to a high-pass filter to filter out meaningless low-frequency noise. In the fundamental frequency estimation, the fundamental frequency estimation algorithm based on aliasing is used to estimate the fundamental frequency. According to the estimated fundamental frequency, in spectral reconstruction, not only the harmonics in the high frequency band can be reconstructed, but also the abnormal harmonics in the low frequency band can be corrected. Finally, the low-resolution spectrum is restored to a high-resolution spectrum. In spectral speech conversion, the system uses the Griffin-Lim algorithm to convert the recovered high-resolution frequency spectrum into a speech signal audible to human ears.

The whole process of signal preprocessing is as follows: when the mobile phone speaker plays the voice signal, the data of the z-axis of the mobile phone accelerometer, the y-axis of the gyroscope, and the z-axis of the magnetometer are collected. Since the user's behavior when using the mobile phone may interfere with the measurement of the sensor, the system sends the collected sensor signal into a high-pass filter with a cutoff frequency of 80Hz to filter out meaningless low-frequency noise and retain the signal leaked when the speaker plays voice. . Figure 2 is a comparison diagram of the response of the accelerometer to the voice signal leaked by the speaker in different scenarios, showing the measured value of the built-in accelerometer of the mobile phone when the mobile phone uses the speaker to play a single tone signal. Figures 2(a) and 2(b) are the measured values of the built-in accelerometer when the mobile phone is on the desktop and in the handheld state, respectively. In the handheld state, the movement of the user's hand seriously interferes with the measured value of the acceleration, thereby covering up the measured value of the acceleration. A signal leaked when the speaker is playing speech. Figure 2(c) shows the accelerometer signal after the high-pass filter, and it can be seen that the signal leaked by the speaker is successfully separated.

Then, a fundamental frequency estimation module is used to estimate the fundamental frequency of the speech signal implicit in the sensor data. Using a short-time Fourier transform, the filtered sensor time-domain signal is converted into an amplitude spectrum M(t,f). Since the sampling rate of the built-in sensor in the mobile phone is much lower than the actual frequency of the voice signal, the signal actually collected by the sensor will be aliased due to under-sampling. The relationship between the frequency change before and after the signal undersampling can be described as:

具体来说，一个原始频率是f的信号，以采样率SR欠采样后，信号的频率会变成A(f)。考虑到这种频率变化，幅度频谱M(t，f)中不仅存在对应语音信号中的正常谐波，还存在欠采样生成的异常谐波。所以本系统使用基于混叠的谐波相加法来衡量频率f是基频的可能性H(f)。H(f)可以表示成:

Specifically, a signal whose original frequency is f will become A(f) after under-sampling with the sampling rate SR. Considering this frequency change, there are not only normal harmonics in the corresponding speech signal, but also abnormal harmonics generated by undersampling in the magnitude spectrum M(t, f). So this system uses the aliasing based harmonic addition method to measure the probability H(f) that the frequency f is the fundamental frequency. H(f) can be expressed as:

其中M(t，f)是传感器数据的幅度频谱，t代表帧号，f代表频率。k代表谐波的阶数。n代表频率低于

的最高阶谐波的阶数，m代表频率低于1250Hz的最高阶谐波的阶数。H(f)的前一项包括了频谱中的正常谐波，后一项包括了因为欠采样而产生的，频率发生非线性变化的异常谐波，同时考虑这两个部分能够提高基频估计的准确性。根据H(f)越大，t时刻f是基频的可能性越大的特点，频谱中每一帧的基频f_p可以表示为

其中85Hz到255Hz代表大部分成年人说话时的基频范围。

where M(t, f) is the magnitude spectrum of the sensor data, t is the frame number, and f is the frequency. k represents the order of the harmonic. n stands for frequency below

The order of the highest-order harmonic of , m represents the order of the highest-order harmonic whose frequency is lower than 1250Hz. The former term of H(f) includes normal harmonics in the spectrum, and the latter term includes abnormal harmonics with nonlinear frequency changes due to undersampling. Considering these two parts at the same time can improve the fundamental frequency estimation accuracy. According to the characteristic that the larger H(f) is, the greater the possibility that f is the fundamental frequency at time t is, the fundamental frequency f _p of each frame in the spectrum can be expressed as

Among them, 85Hz to 255Hz represents the fundamental frequency range when most adults speak.

之间的频谱，SR为传感器采样率，语音谐波的频率为

算法只保留这一频带上语音谐波所在的频段,即M_old(t，kf_p)处的频谱，具体来说，系统将M_old(t，kf_p)直接赋值给M_new(t，kf_p)，以保留低频段的正常语音谐波，并且去除低频段的混叠。对于频率处于

到F_endHz之间的频谱，F_end代表重建频谱的最高频率),语音谐波的频率为

装置根据欠采样时频率变化的关系来估计这一部分谐波所在的位置以及能量的大小。具体来说，原始正常语音谐波的频率为kf_pHz，由于欠采样，其频率以公式(1)转变成A(kf_p)Hz。接下来系统用已知的混叠谐波频谱M_old(t，A(kf_p))来替换未知的正常谐波频谱M_new(t，kf_p)。在完成对M_old(t，f)中每一帧t的遍历后，系统生成了重建后的幅度频谱M_new(t，f)。在本发明中，如果传感器数据的带宽能达到250Hz，重建频谱的带宽F_end就能达到1000Hz。On the basis of the fundamental frequency estimation, the system further reconstructs the spectrum of the sensor signal, and the speech super-resolution technology utilizes the relationship between the fundamental frequency and harmonics to expand the narrowband signal. However, speech super-resolution techniques cannot be directly used to recover audible speech from low-resolution sensor signals due to aliasing caused by undersampling. Therefore, the present system uses an aliasing-corrected super-resolution algorithm capable of extending the bandwidth of the sensor's amplitude spectrum. Denote the reconstructed amplitude spectrum as M _new (t, f) (initially a zero matrix), and the original amplitude spectrum as M _old (t, f). The algorithm includes harmonic reconstruction in the high frequency band and aliasing removal in the low frequency band. First, the algorithm traverses each frame of the original spectrum M _old (t, f). In each round of traversal, the fundamental frequency estimation module is used to estimate the size of the fundamental frequency f _p , and then according to the integral of the fundamental frequency and the harmonic frequency The multiple relationship is obtained to obtain the frequency kf _p of each harmonic. Next, the algorithm reconstructs the spectrum, for frequencies between 0Hz and

The frequency spectrum between, SR is the sensor sampling rate, and the frequency of speech harmonics is

The algorithm only retains the frequency band where the voice harmonics are located in this frequency band, that is, the frequency spectrum at M _old (t, kf _p ). Specifically, the system directly assigns M _old (t, kf _p ) to M _new (t, kf ) _p ) to preserve the normal speech harmonics of the low frequency band and remove the aliasing of the low frequency band. For frequencies at

to F _end Hz, where F _end represents the highest frequency of the reconstructed spectrum), the frequency of the speech harmonics is

The device estimates the position of this part of the harmonics and the magnitude of the energy according to the relationship between the frequency changes during under-sampling. Specifically, the frequency of the original normal speech harmonic is kf _p Hz, and its frequency is transformed into A(kf _p ) Hz by formula (1) due to undersampling. Next the system replaces the unknown normal harmonic spectrum M _new (t, kf _p ) with the known aliased harmonic spectrum M _old (t, A(kf _p )). After completing the traversal of each frame t in M _old (t, f), the system generates a reconstructed magnitude spectrum M _new (t, f). In the present invention, if the bandwidth of the sensor data can reach 250 Hz, the bandwidth F _end of the reconstructed spectrum can reach 1000 Hz.

由于幅度频谱

与重建频谱M_new(t，f)存在一定的区别，算法只保留相位频谱P₁，并将P₁送入下一次迭代过程。以此类推，Griffin-Lim算法通过一次次的迭代不断修正相位频谱P_i，直到生成的幅度频谱

与重建频谱M_new(t，f)足够相似。在经过一定次数的迭代之后，Griffin-Lim算法利用给定的重建频谱M_new(t，f)生成了对应的语音信号。Based on the reconstructed spectrum M _new (t, f), the system uses the Griffin-Lim algorithm to recover the audible speech signal from it. The algorithm estimates the speech signal from the reconstructed spectrum M _new (t, f) through iterations. The Griffin-Lim algorithm first randomly generates a phase spectrum P ₀ , and then uses the inverse short-time Fourier transform to convert the phase spectrum P ₀ and the amplitude spectrum M _new (t, f) into a speech signal x ₀ . After that, do a short-time Fourier transform on the speech signal x ₀ to obtain the phase spectrum P ₁ and the amplitude spectrum

Due to the magnitude spectrum

Different from the reconstructed spectrum M _new (t, f), the algorithm only retains the phase spectrum P ₁ and sends P ₁ to the next iterative process. By analogy, the Griffin-Lim algorithm continuously corrects the phase spectrum P _i through iterations until the generated amplitude spectrum

Similar enough to the reconstructed spectrum M _new (t, f). After a certain number of iterations, the Griffin-Lim algorithm uses the given reconstructed spectrum M _new (t, f) to generate the corresponding speech signal.

The invention constructs a sensor data hidden information recovery device without training, which can not only recover the speech information in the sensor data with extremely narrow bandwidth and serious aliasing, but also solves the problem of poor transferability of the learning-based model. The present invention can adapt to the changes of users, environments and equipment, and effectively recover hidden voice signals from the built-in sensor of the mobile phone.

In order to verify the validity of the present invention, the present invention was deployed on three mobile phones (Huawei P40, Xiaomi Mi 10, OPPO Find X2). When the voice is played on the mobile phone speaker, the measurement data of the three mobile phone motion sensors (including the accelerometer and gyroscope) are directly collected. The sampling rates of the motion sensors of Huawei P40, Xiaomi 10, and OPPO Find X2 are 500Hz, 397Hz, and 418Hz, respectively. . Since the sampling rate of the built-in magnetometer of the mobile phone is too low (100Hz), a magnetometer with a sampling rate of 500Hz and model MMC3416xPJ is attached to the surface of the mobile phone to collect the magnetic field leaked when the speaker plays voice. The collected sensor data will be sent to the hidden information recovery device at the back end to recover the voice signal. The logarithmic spectral distance (LSD) is chosen to measure the difference between the recovered speech spectrum of the recovered system and the corresponding real speech spectrum, and the LSD can be expressed as:

和x分别是重建的语音信号和原始的语音信号，

和X分别是重建的语音信号和原始的语音信号的对数能量频谱。LSD数值越小，说明语音恢复的质量越好。图3为磁力计信号，真实语音信号和重建语音信号的幅度频谱图，图3(a)和图3(b)分别展示了磁力计数据的原始频谱和对应的真实语音频谱(低于1000Hz)。可见，磁力计数据的原始频谱的带宽只有真实语音频谱带宽的

且存在着严重的混叠。图3(c)展示了本发明从磁力计数据中重建的语音频谱，可见重建的语音频谱不仅在高频段上重建了谐波的结构，而且消除了低频段上存在的混叠。in,

and x are the reconstructed speech signal and the original speech signal, respectively,

and X are the logarithmic energy spectra of the reconstructed speech signal and the original speech signal, respectively. The smaller the LSD value, the better the quality of speech recovery. Figure 3 shows the magnitude spectrograms of the magnetometer signal, the real voice signal and the reconstructed voice signal. Figure 3(a) and Figure 3(b) show the original spectrum of the magnetometer data and the corresponding real voice spectrum (below 1000Hz), respectively. . It can be seen that the bandwidth of the original spectrum of the magnetometer data is only the bandwidth of the real voice spectrum.

And there is serious aliasing. Figure 3(c) shows the speech spectrum reconstructed from magnetometer data in the present invention. It can be seen that the reconstructed speech spectrum not only reconstructs the structure of harmonics in the high frequency band, but also eliminates the aliasing existing in the low frequency band.

Firstly, the performance of the present invention and two existing speech super-resolution techniques (spectral folding and spectral shifting) are compared. FIG. 4 is an overall performance comparison diagram of the present invention, showing the performance comparison between the present invention and these two speech super-resolution technologies. It can be seen that the present invention can obtain relatively low LSD. Specifically, for magnetometers, accelerometers and gyroscopes, the present invention can outperform spectral folding and spectral shifting methods by 29.5%, 32.1% and 13.5%. This shows that, compared with the traditional speech super-resolution method, the present invention can more effectively recover speech information in low-resolution sensor data.

In order to verify the mobility of the present invention, the present invention is deployed in different scenarios.

When the voice played by the speaker comes from different subjects, FIG. 5 is a performance comparison diagram of the present invention for different using subjects, showing the performance of the present invention. It can be seen that for the speech from different subjects (20), there is no obvious change in LSD. Specifically, for the magnetometer, accelerometer, and gyroscope data, the standard deviation of the LSD is 0.09, 0.11, and 0.21, respectively. This demonstrates the robustness of the present invention to speech from different subjects. When the voice played by the speaker contains different texts, FIG. 6 is a performance comparison diagram of the present invention for different text contents, showing the performance of the present invention. It can be seen that for voices with different text contents (10 kinds), LSD does not compare. Significant variation, specifically, the standard deviation of LSD is 0.16, 0.12 and 0.01 for the magnetometer, accelerometer and gyroscope data, respectively. This reflects the robustness of the present invention to different speech texts.

When the mobile phone models deployed in the present invention are different, FIG. 7 is a performance comparison diagram of the present invention on different devices, showing the performance of the present invention. It can be seen that for the data collected from the accelerometer and the magnetometer, the LSD is lower than 1.5, which means that the present invention can operate effectively on different devices. Since the sensitivity of the gyroscope is lower than that of the accelerometer and the magnetometer, the LSD of the speech spectrum recovered from the gyroscope data is relatively high.

When the present invention is deployed in different environments, FIG. 8 is a performance comparison diagram of the present invention in different environments, showing the performance of the present invention. It can be seen that in different environments (the noise intensity is 45.3dbSPL in the laboratory, the noise intensity is For the dormitory under 48.9dbSPL and the canteen with noise intensity of 74.6dbSPL), the LSD is still relatively close and relatively low. 0.311. This means that the present invention can operate effectively in different environments. In different user interaction scenarios, FIG. 9 is a performance comparison diagram of the present invention under different interaction modes, showing the performance of the present invention. It can be seen that in the desktop scene (the user puts the mobile phone on the desktop), the hand-held scene (the user holds the mobile phone), and the tapping scene (the user continuously taps the screen), the present invention can still obtain similar LSDs. Specifically, For both magnetometer and accelerometer data, the LSD range is below 0.1. This means that the present invention can remove the interference of user interaction on sensor measurement data, and achieve effective voice information recovery.

It will be apparent to those skilled in the art that the present invention may be modified in various ways. Such changes are not considered to depart from the scope of the present invention. All such modifications obvious to those skilled in the art will be included within the scope of the present claims. within.

Claims (7)

1. The device is characterized by comprising a signal preprocessing module, a fundamental frequency estimation module, a spectrum reconstruction module and a spectrum voice conversion module which are sequentially connected, wherein the recovery method of the device is as follows:

1) the method comprises the following steps of collecting data of a z axis of an accelerometer, a y axis of a gyroscope and a z axis of a magnetometer of the mobile phone through a signal preprocessing module, and sending the collected sensor data to a high-pass filter;

2) estimating the fundamental frequency size by a fundamental frequency estimation module by using a fundamental frequency estimation algorithm based on aliasing, wherein the algorithm can simultaneously consider normal harmonics in the voice signal and abnormal harmonics generated during signal undersampling and estimate the fundamental frequency of the voice signal implicit in the sensor data;

3) according to the estimated fundamental frequency, reconstructing harmonic waves of a high frequency band and correcting abnormal harmonic waves of a low frequency band through a frequency spectrum reconstruction module, and restoring the frequency spectrum of the low resolution into a frequency spectrum of the high resolution;

4) converting the recovered high-resolution frequency spectrum into a voice signal audible to human ears by using a Griffin-Lim algorithm through a frequency spectrum voice conversion module;

in the step 2), the fundamental frequency estimation module converts the filtered sensor time domain signal into an amplitude spectrum M (t, f) by using short-time fourier transform, when the frequency of the original signal is greater than a half of the sampling rate of the sensor, the signal actually acquired by the sensor will produce aliasing due to undersampling, and the relationship of frequency change before and after the signal undersampling is as follows:

f is the original frequency, SR is the sampling rate, A (f) is the varied frequency;

the probability that frequency f is the fundamental frequency h (f) is measured using an aliasing-based harmonic summation:

wherein M (t, f) is the amplitude spectrum of the sensor data, t is the frame number, f is the frequency, k is the order of harmonics, and n is the frequency below

M represents the order of the highest order harmonics with a frequency below 1250 Hz.

2. The apparatus for recovering general speech information based on undersampled data of built-in sensor according to claim 1, wherein said signal preprocessing module: the device is used for acquiring data of an accelerometer, a gyroscope and a magnetometer of the mobile phone and sending the acquired sensor data to a high-pass filter; the fundamental frequency estimation module: for estimating a fundamental frequency of a speech signal implicit in the sensor data; the spectrum reconstruction module: the frequency spectrum recovery device is used for reconstructing harmonic waves of a high frequency band, correcting abnormal harmonic waves of a low frequency band and recovering a frequency spectrum of a low resolution into a frequency spectrum of a high resolution; the spectrum voice conversion module: for converting the recovered high resolution spectrum into a speech signal audible to the human ear using the Griffin-Lim algorithm.

3. The apparatus for recovering general speech information based on under-sampled data of built-in sensor according to claim 1, wherein due to frequency variation, the amplitude spectrum M (t, f) has normal harmonics corresponding to the speech signal and abnormal harmonics generated by under-sampling.

4. The apparatus for recovering general speech information according to claim 1, wherein the former term accumulates the energy of normal harmonics in the spectrum, the latter term accumulates the energy of abnormal harmonics generated by undersampling, and the fundamental frequency f of each frame in the spectrum_pIs composed of

5. The apparatus for recovering general speech information based on undersampled data of built-in sensor according to claim 1, wherein in step 3), the spectrum of the sensor signal is reconstructed by the spectrum reconstruction module, the bandwidth of the sensor amplitude spectrum is expanded by the super-resolution algorithm of aliasing correction, and the reconstructed amplitude spectrum is recorded as M_new(t, f), initially a zero matrix, with an original amplitude spectrum of M_old(t, f), the concrete steps are as follows:

a. algorithm traversal of original spectrum M_oldIn each round of traversal, the fundamental frequency f is estimated by the fundamental frequency estimation module for each frame (t, f)_pAccording to the integral multiple relation of the fundamental frequency and the harmonic frequency, the frequency kf of each harmonic is obtained_p；

b. Algorithm reconstructs the spectrum, for frequencies in the range 0Hz to

Spectrum of (1) and (b)Frequency of harmonic of sound is kf_pHz，

Will M_old(t，kf_p) Direct assignment to M_new(t，kf_p) (ii) a To preserve the normal speech harmonics of the low band and to remove aliasing of the low band, for frequencies at

To F_endFrequency spectrum between Hz, frequency of speech harmonics kf_pHz，

F_endThe highest frequency of the reconstructed spectrum;

c. estimating the part of harmonic according to the relation of frequency change in the undersampling, wherein the frequency of the part of harmonic is

To F_endHz, the location and the amount of energy, the frequency of the original normal speech harmonic is kf_pHz, whose frequency is converted to A (kf) by equation (1) due to undersampling_p)Hz；

d. Using known aliased harmonic spectrum M_old(t，A(kf_p) ) to replace the unknown normal harmonic spectrum M_new(t，kf_p) After completing the pair M_oldAfter traversing each frame t in (t, f), the system generates a reconstructed amplitude spectrum M_new(t，f)。

6. The apparatus for recovering general speech information based on undersampled data of built-in sensor according to claim 1, wherein in said step 4), the spectrum speech conversion module is based on reconstructed spectrum M_new(t, f) recovering therefrom a speech signal audible to the human ear using the Griffin-Lim algorithm which reconstructs the spectrum M from the N iterations_newAnd (t, f) estimating the voice signal.

7. The device for recovering general voice information based on undersampled data of built-in sensor according to claim 6, wherein the specific steps of the step 4) are as follows:

d. Griffin-Lim algorithm randomly generates a phase spectrum P₀Then, the phase spectrum P is transformed by inverse short-time Fourier transform₀And reconstructing the amplitude spectrum M_new(t, f) into a speech signal x₀；

e. For speech signal x₀Performing short-time Fourier transform to obtain a phase frequency spectrum P₁And predicting the amplitude spectrum

Due to prediction of the amplitude spectrum

And reconstructing the amplitude spectrum M_new(t, f) there is a certain difference, and the algorithm only reserves the phase frequency spectrum P₁And is combined with P₁Sending to the next iteration process;

f. the Griffin-Lim algorithm continuously corrects the phase spectrum P through N iterations_iUp to the predicted amplitude spectrum

And reconstructing the amplitude spectrum M_new(t, f) are sufficiently similar that the Griffin-Lim algorithm utilizes a given reconstructed amplitude spectrum M_new(t, f) a corresponding speech signal is generated.

Images