WO2025213833A1 - Neural network-based voice packet loss concealment method and device - Google Patents

Abstract

A neural network-based voice packet loss concealment method and device. The method comprises: acquiring a pre-trained neural network used for voice packet loss concealment; receiving an audio to be processed and frame loss position information corresponding to said audio; and inputting an input feature generated on the basis of said audio and the frame loss position information corresponding to said audio into the neural network to obtain a packet-loss-concealed audio corresponding to said audio.

Description

Voice packet loss compensation method and device based on neural network

This application claims priority to the Chinese invention patent application entitled “Voice packet loss compensation method and device based on neural network” and application number 202410411048.5 filed on April 7, 2024. The entire contents of the application are incorporated by reference into this application.

Technical Field

The embodiments of the present disclosure relate to the field of computer technology, and more particularly to a method and apparatus for voice packet loss compensation based on a neural network.

Background Art

With the development of internet and communications technologies, audio streaming has become a crucial component of network communications. In practice, audio packets can be lost during transmission due to various factors, such as network congestion, bandwidth limitations, and hardware failures. This can severely impact the quality of voice communication and degrade the user experience. Therefore, recovering from audio packet loss is an urgent problem that needs to be addressed.

Summary of the Invention

The embodiments of the present disclosure describe a method and apparatus for voice packet loss compensation based on a neural network. The neural network trained by this method can more accurately compensate for voice packet loss.

According to a first aspect, a method for training a neural network for speech packet loss compensation is provided, wherein the neural network to be trained includes an encoder layer, an intermediate layer and a decoder layer, and the intermediate layer is connected between the encoder layer and the decoder layer. The method includes: obtaining a training sample set, wherein each training sample includes sample packet loss audio, its corresponding sample frame loss position information and sample non-packet loss audio; generating input features based on the sample packet loss audio and its corresponding sample frame loss position information; inputting the input features into the neural network to be trained; inputting the features output by the intermediate layer into a pre-trained fundamental frequency prediction network, and the fundamental frequency prediction network outputs a predicted fundamental frequency; and adjusting the network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequency and the true fundamental frequency calculated based on the sample non-packet loss audio.

In one embodiment, the neural network to be trained is a U-Net neural network, wherein the intermediate layer is a bottleneck layer in the U-Net structure. Thus, voice packet loss compensation can be achieved using the U-Net neural network.

In one embodiment, the pitch prediction network includes a bidirectional long short-term memory network. When processing audio data, the bidirectional long short-term memory network can fully consider the contextual information in the audio data, thereby improving the processing capability of the audio data.

In one embodiment, the features output by the intermediate layer include features of the frame corresponding to the frame loss position corresponding to the sample frame loss position information, and the predicted fundamental frequency output by the fundamental frequency prediction network includes the fundamental frequency of the frame corresponding to the frame loss position.

In one embodiment, generating input features based on the packet-loss audio sample and its corresponding sample frame-loss location information includes: performing sub-band decomposition on the packet-loss audio sample to obtain multiple sub-bands; and generating input features based on the results of converting the multiple sub-bands into the time-frequency domain and the sample frame-loss location information. This allows the packet-loss audio sample to be decomposed into multiple sub-bands for processing, significantly reducing computational complexity.

In one embodiment, the encoder layer includes multiple encoders, each of which includes a gated convolution layer and a time-frequency dilated convolution layer. The time-frequency dilated convolution layer is used to extract features through dilated convolution in the time and frequency dimensions. This effectively increases the receptive field of the convolution layer.

In one embodiment, the decoder layer includes multiple decoders, each decoder includes a first branch and a second branch in parallel, the first branch is used to predict the real part of the audio, and the second branch is used to predict the imaginary part of the audio; the neural network to be trained outputs sample predicted audio based on the real and imaginary parts of the predicted audio output by the encoder layer; and the method further includes: inputting the sample predicted audio into at least one pre-trained discriminator, and each discriminator outputs a discrimination result for the sample predicted audio; calculating the loss based on at least one discrimination result, the sample predicted audio, and the sample un-lost audio, and adjusting the network parameters of the neural network to be trained based on the loss. Therefore, a generative adversarial network (GAN) can be used to train the neural network to be trained.

In one embodiment, the at least one discriminator includes a first discriminator for determining the probability that the sample predicted audio is real audio and a second discriminator for determining the audio quality of the sample predicted audio. Thus, the accuracy of the generator can be improved by using multiple discriminators.

In one embodiment, the method further includes: inputting the sample predicted audio output by the neural network to be trained and its corresponding sample unpacked audio into a pre-trained speech recognition model; obtaining coding layer features of the sample predicted audio and its corresponding sample unpacked audio in the speech recognition model; and adjusting the network parameters of the neural network to be trained based on the difference loss between the two obtained coding layer features. In this way, the network parameters of the neural network to be trained can be adjusted using the pre-trained speech recognition model, thereby improving the accuracy of the neural network to be trained.

According to a second aspect, a method for voice packet loss compensation based on a neural network is provided, comprising: obtaining a neural network for voice packet loss compensation trained according to any one of the methods of the first aspect; receiving audio to be processed and frame loss position information corresponding to the audio to be processed; and inputting input features generated based on the audio to be processed and the frame loss position information corresponding to the audio to be processed into the neural network to obtain audio corresponding to the audio to be processed after packet loss compensation.

According to a third aspect, a device for training a neural network for speech packet loss compensation is provided, wherein the neural network to be trained includes an encoder layer, an intermediate layer and a decoder layer, and the intermediate layer is connected between the encoder layer and the decoder layer. The device includes: an acquisition unit configured to acquire a training sample set, wherein each training sample includes sample packet loss audio, its corresponding sample frame loss position information and sample non-packet loss audio; a generation unit configured to generate input features based on the sample packet loss audio and its corresponding sample frame loss position information; a first input unit configured to input the input features into the neural network to be trained; a second input unit configured to input the features output by the intermediate layer into a pre-trained fundamental frequency prediction network, and the fundamental frequency prediction network outputs a predicted fundamental frequency; and an adjustment unit configured to adjust the network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequency and the true fundamental frequency calculated based on the sample non-packet loss audio.

According to a fourth aspect, a speech packet loss compensation device based on a neural network includes: a model acquisition unit, configured to obtain a neural network for speech packet loss compensation trained according to any one of the methods of the first aspect; a receiving unit, configured to receive audio to be processed and frame loss position information corresponding to the audio to be processed; and a feature input unit, configured to input input features generated based on the audio to be processed and the frame loss position information corresponding to the audio to be processed into the neural network to obtain audio after packet loss compensation corresponding to the audio to be processed.

According to a fifth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above methods in the first aspect.

According to the sixth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute any one of the methods in the first aspect.

According to the seventh aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, any one of the above methods in the first aspect is implemented.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram showing an application scenario in which an embodiment of the present disclosure can be applied;

FIG2 is a schematic flow chart showing a method for training a neural network for voice packet loss compensation according to one embodiment;

FIG3 is a schematic diagram showing an example of an encoder structure;

FIG4 is a schematic diagram showing an example of a fundamental frequency prediction network structure;

FIG5 is a schematic diagram showing an example of a neural network structure to be trained;

FIG6 is a schematic diagram showing an example of a frequency domain multi-resolution discriminator;

FIG7 is a schematic diagram showing an example of a time-domain multi-cycle discriminator;

Figure 8 shows a schematic diagram of an example of MetricGAN discriminator training;

FIG9 shows a method for compensating for speech packet loss based on a neural network according to one embodiment;

FIG10 shows a schematic block diagram of an apparatus for training a neural network for voice packet loss compensation according to one embodiment;

FIG11 shows a schematic block diagram of a speech packet loss compensation apparatus based on a neural network according to an embodiment;

FIG12 shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present application.

DETAILED DESCRIPTION

It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

The technical solutions provided by the present disclosure are further described in detail below in conjunction with the accompanying drawings and embodiments. It will be understood that the specific embodiments described herein are merely for explaining the relevant inventions and are not intended to limit the inventions. It should also be noted that, for ease of description, only the portions relevant to the relevant inventions are shown in the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments of the present disclosure and the features therein may be combined with each other.

As mentioned earlier, audio data packets may be lost during transmission, which will have a serious impact on the quality of voice communication and cause a degradation in user experience. The main goal of voice packet loss compensation (PLC) technology is to recover or conceal lost data packets as much as possible through various means, thereby maintaining the continuity and clarity of voice communication. Traditional packet loss compensation technology can compensate for packet loss based on redundant coding or interpolation of signal processing. For example, the forward error correction technology used in many codecs, when poor network conditions are detected, the sender can transmit redundant information about past frames to recover short packet losses. However, this method introduces additional network overhead and additional delay, and cannot handle longer packet losses.

To this end, embodiments of the present disclosure provide a neural network-based voice packet loss compensation method. First, a neural network can be trained using the method provided in embodiments of the present disclosure for training a neural network for voice packet loss compensation. This allows the trained neural network to more accurately compensate for voice packet loss without introducing additional network overhead or latency.

According to the embodiment of the present disclosure, a method and device for speech packet loss compensation based on a neural network is provided. First, it is necessary to train a neural network. The neural network to be trained may include an encoder layer, an intermediate layer and a decoder layer, and the intermediate layer is connected between the encoder layer and the decoder layer. The training samples used include sample packet loss audio, and its corresponding sample frame loss position information and sample non-packet loss audio. During the training process, input features can be generated based on the sample packet loss audio and its corresponding sample frame loss position information, and the input features can be input into the neural network to be trained. Then, the features output by the intermediate layer can be input into a pre-trained fundamental frequency prediction network, and the fundamental frequency prediction network outputs a predicted fundamental frequency. Then, based on the predicted fundamental frequency and the actual fundamental frequency calculated based on the sample non-packet loss audio, the network parameters of the encoder layer and the intermediate layer are adjusted. In this way, the outputs of the encoder layer and the intermediate layer can be made more accurate, and the trained neural network can then perform speech packet loss compensation more accurately.

Figure 1 illustrates a schematic diagram of an application scenario in which embodiments of the present disclosure can be applied. As shown in Figure 1 , in the application scenario illustrated in Figure 1 , electronic device 10 can first train a neural network 101 for voice packet loss compensation. Electronic device 20 can then retrieve neural network 101 from electronic device 10 and use the trained neural network 101 to perform voice packet loss compensation.

Specifically, the process of training the neural network 101 by the electronic device 10 may include the following steps 1-5: Step 1: Obtain a training sample set. Here, each training sample in the training sample set may include the sample audio with packet loss, its corresponding sample frame loss position information, and the sample audio without packet loss. Step 2: Generate input features based on the sample audio with packet loss and its corresponding sample frame loss position information. Step 3: Input the input features into the neural network to be trained. In this application scenario, the neural network to be trained may include an encoder layer, an intermediate layer, and a decoder layer, with the intermediate layer connected between the encoder layer and the decoder layer. Step 4: Input the features output by the intermediate layer into a pre-trained fundamental frequency prediction network, which outputs a predicted fundamental frequency. Step 5: Update the network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequency and the actual fundamental frequency calculated based on the sample audio without packet loss. In addition, the network parameters of the entire neural network to be trained may also be updated based on the prediction results output by the decoder layer and the sample audio without packet loss. This results in a trained neural network 101.

Afterwards, the electronic device 20 can obtain the trained neural network 101 from the electronic device 10, and input the audio to be processed and the frame loss position information corresponding to the audio to be processed into the neural network 101, thereby obtaining the audio after packet loss compensation corresponding to the audio to be processed.

Continuing with FIG2 , FIG2 illustrates a flow chart of a method for training a neural network for voice packet loss compensation according to one embodiment. The method can be performed by any device, apparatus, platform, or device cluster with computing and processing capabilities. As shown in FIG2 , the method for training a neural network for voice packet loss compensation may include the following steps 201 to 205, specifically:

Step 201: Obtain a training sample set. In this embodiment, each training sample in the training sample set may include sample packet loss audio, sample frame loss position information corresponding to the sample packet loss audio, and sample non-packet loss audio corresponding to the sample packet loss audio. The sample frame loss position information may be used to indicate the position of a frame in the sample packet loss audio where data loss occurred. In this example, the sample packet loss audio may be audio at various sampling rates, for example, at a 48 kHz sampling rate.

In practice, audio data is usually processed and transmitted in frames, and each frame can contain audio of a certain length. During network transmission, due to network congestion, delays, or other reasons, some frames may not successfully reach their destination, resulting in packet loss. In order to detect and handle these packet loss situations, a flag bit is usually set in each frame of audio data. For example, this flag bit can be a binary bit used to indicate whether the frame has arrived intact. If the flag bit is set to indicate a packet loss state, the receiving end can detect the loss of this frame of data. Therefore, the position of the frame in the audio where data loss occurred can be determined based on the flag bit.

Step 202: Generate input features based on the packet-loss audio sample and its corresponding sample-loss frame location information. In this embodiment, the input features can be generated based on the packet-loss audio sample and its corresponding sample-loss frame location information. For example, the packet-loss audio sample can be first converted to the frequency domain and then concatenated with the sample-loss frame location information to generate the input features.

In some implementations, step 202 may include the following steps 1 and 2. Specifically, in step 1, sub-band decomposition is performed on the sample packet loss audio to obtain multiple sub-bands. In step 2, input features are generated based on the conversion results of the multiple sub-bands into the time-frequency domain and the sample frame loss location information.

In this implementation, multiple methods can be used to perform sub-band decomposition on the sample loss audio. For example, a stable and efficient Pseudo-Quadrature Mirror Filter Bank (PQMF) can be used to perform sub-band decomposition, dividing the sample loss audio into multiple sub-bands. In PQMF, sub-band decomposition can include K FIR (Finite Impulse Response) filter banks. The sub-band decomposition process can include FIR analysis, downsampling, and short-time Fourier transform (STFT).

For example, assuming that y represents the original audio to be input to the neural network to be trained, _yk represents the audio after subband analysis and downsampling, k∈[1,K] is the subband number, and _yk can represent the kth subband. The sampling rate of _yk is It can represent the frequency domain corresponding to y _k after short-time Fourier transform (STFT). Stack Y _k along the channel dimension to obtain the frequency domain subband features of the input neural network to be trained After that, the input features can be directly concatenated with the frame loss position information, or compressed and concatenated with the frame loss position information to obtain the input features. For example, the input features can be concatenated with the frame loss position information after amplitude spectrum compression with a scaling factor of 0.5.

It can be understood that the sub-band decoding process corresponding to the above sub-band decomposition process may include iSTFT (Inverse Short-Time Fourier Transform), upsampling and FIR synthesis. Corresponding to the sub-band analysis process, the output of the neural network is For each subband After inverse Fourier transform, the time domain subband audio is obtained After upsampling and synthesis filter, it is restored to audio In this implementation, K can be 4, that is, the audio signal is divided into 4 sub-bands for processing.

This implementation allows audio with sample loss to be decomposed into multiple subbands for processing, significantly reducing computational complexity. Therefore, this implementation is particularly suitable for audio at high sampling rates, such as audio with a sampling rate greater than or equal to 48 kHz.

Step 203: Input the input features to the neural network to be trained. In this embodiment, the obtained input features can be input to the neural network to be trained and processed by the neural network to be trained. Here, the neural network to be trained may include an encoder layer, an intermediate layer, and a decoder layer. The intermediate layer is connected to the encoder layer and the decoder layer. The encoder layer and the decoder layer may or may not be connected.

In some implementations, the neural network to be trained can be a neural network with a U-Net structure. The U-Net structure can primarily include an encoder, a bottleneck layer, and a decoder. The bottleneck layer in the U-Net structure can be an intermediate layer. In the U-Net structure, a skip connection can be used between the encoder and the decoder.

In some implementations, the encoder layer may include several encoders. As needed, each encoder may include various layers. For example, it may include a convolution layer, a pooling layer, and the like. For example, each encoder may include a gated convolution layer and a time-frequency dilated convolution layer, and the time-frequency dilated convolution layer may be used to extract features through dilated convolutions in the time and frequency dimensions. As shown in FIG3 , FIG3 shows a schematic diagram of an example of an encoder structure. In the example shown in FIG3 , the encoder may include a gated convolution layer and a time-frequency dilated convolution layer in sequence, and the time-frequency dilated convolution layer may include dilated convolutions FConv (Frequency Dilated Convolution) and TConv (Time Dilated Convolution) in the frequency and time dimensions. FConv and TConv can be regarded as multi-scale modeling along the frequency axis and the time axis, fully perceiving historical information. In addition, the time-frequency hole convolution layer can also include a BN (Batch Normalization) layer, a PReLU (Parametric Rectified Linear Unit, ReLU with parameters) activation function, PWConv (Pointwise convolution), etc. PWConv can be used to align the input and output dimensions. Here, b convolution layers with expansion degrees from 1 to 2 ^b-1 are spliced together to form a time-frequency hole convolution layer, which can effectively improve the receptive field of the convolution layer. It can be understood that the encoder structure shown in Figure 3 is only schematic and not a limitation of the encoder structure. In practice, the layers included in the encoder can be set according to actual needs.

In step 204, the features output by the intermediate layer are input into a pre-trained fundamental frequency prediction network, which then outputs a predicted fundamental frequency. In this embodiment, the intermediate layer can be used for packet loss compensation and can be implemented using various neural networks. For example, the intermediate layer can be implemented using a bidirectional GRU (Gate Recurrent Unit), which can extract the correlation between frequency and time dimensions to compensate for packet loss.

Therefore, the features output by the intermediate layer can include features of the frame corresponding to the frame loss location corresponding to the sample frame loss location information, and the pre-trained FFT prediction network can be used to predict the FFT based on the input features. Therefore, the features output by the intermediate layer are input into the FFT prediction network, which can output a predicted FFT. This predicted FFT can include the FFT of the frame corresponding to the frame loss location. In this example, the FFT prediction network can be any of various neural networks.

In some implementations, the FFT prediction network may include a bidirectional long short-term memory (Bi-LSTM) network. As shown in FIG4 , FIG4 illustrates a schematic diagram of an example FFT prediction network structure. In the example shown in FIG4 , the FFT prediction network may include, in sequence, a reshape layer, a Maxpool layer, a reshape layer, a Bi-LSTM layer, a Linear layer, a reshape layer, a Linear_C layer, a Linear_F layer, and so on. Here, the output of the first reshape layer forms a skip connection with the output of the Linear layer. The reshape layer can be used to adjust the data dimension, the Maxpool layer can be used for downsampling, and the Linear layer can be used for linear mapping. The Linear_C layer can represent linear mapping in the channel dimension, and the Linear_F layer can represent linear mapping in the frequency dimension. It should be understood that the FFT prediction network structure in FIG4 is merely illustrative and does not limit the FFT prediction network structure. In practice, the layers included in the FFT prediction network can be set according to actual needs.

As an example, the fundamental frequency prediction network can be trained in various ways. For example, it can be trained in a supervised manner. For example, in a supervised manner, a sample set can be first obtained, and each sample in the sample set can include a sample audio feature and a fundamental frequency label corresponding to the sample audio feature. Here, the sample audio feature can be obtained through the network structure of the neural network to be trained. Afterwards, the sample audio features in the sample can be input into the fundamental frequency prediction network to be trained, and the fundamental frequency prediction network to be trained outputs a predicted fundamental frequency. Then, based on a predefined loss function, the difference loss between the predicted fundamental frequency and the fundamental frequency label is calculated, and the network parameters of the fundamental frequency prediction network are adjusted with the goal of minimizing the difference loss.

Step 205: Adjust the network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequency and the actual fundamental frequency calculated based on the sampled unpacked audio. In this embodiment, the actual fundamental frequency of each frame can be calculated using the sampled unpacked audio, and then the network parameters of the encoder layer and the intermediate layer can be adjusted based on the predicted fundamental frequency and the actual fundamental frequency. For example, the following formula can be used as the loss function:

Among them, T can represent the number of frames, can represent the fundamental frequency of the i-th frame predicted by the fundamental frequency prediction network, _fi can represent the true fundamental frequency of the i-th frame, and _Lf0 can represent the difference loss of the fundamental frequency prediction. Afterwards, the network parameters of the encoder layer and the intermediate layer can be adjusted with the goal of minimizing _Lf0 .

In some implementations, corresponding to the encoder layer, the decoder layer may also include multiple decoders, each of which may include a first branch and a second branch in parallel. The first branch may be used to predict the real part of the audio, and the second branch may be used to predict the imaginary part of the audio. The neural network to be trained may output sample predicted audio based on the real and imaginary parts of the predicted audio output by the encoder layer.

As shown in Figure 5, Figure 5 shows a schematic diagram of an example of a neural network structure to be trained. In the example shown in Figure 5, the input may include sample packet loss audio And the sample frame loss position information m(t), t∈T, B can represent batch size, N can represent audio length, and T can represent the number of frames. The output can include sample prediction audio after packet loss compensation

In the example shown in Figure 5, the sample packet loss audio After passing through the pseudo-quadrature mirror filter bank (PQMF), short-time Fourier transform (STFT), and compression, it is stacked with the sample frame loss position information m(t) and input into the neural network to be trained. The number of encoders included in the encoder layer is the same as the number of decoders included in the decoder layer, and each encoder and corresponding decoder can be skipped. In this example, the encoder layer can include 4 encoders, each of which can include a gated convolution layer and a time-frequency hole convolution layer. The middle layer can be implemented by 3 bidirectional GRUs (Gate Recurrent Units), which can extract the correlation between frequency and time dimensions and compensate for packet loss. Correspondingly, the decoder layer also includes 4 decoders, each of which can include a transposed gated convolution layer and a time-frequency hole convolution layer. Each decoder can include a first branch and a second branch in parallel. The first branch can be used to predict the real part of the audio, and the second branch can be used to predict the imaginary part of the audio. After that, the real and imaginary parts of the predicted audio output by the encoder layer can be concatenated, decompressed, inverse short-time Fourier transform (ISTFT), and pseudo-orthogonal mirror filter bank (PQMF) in sequence to obtain the sample predicted audio after packet loss compensation.

It is understood that the various parts of the neural network structure to be trained shown in Figure 5 are merely illustrative and not intended to limit the neural network structure to be trained. In practice, the structure of the neural network to be trained can be designed according to actual needs.

Based on the sample predicted audio output by the neural network to be trained, the above-mentioned method for training a neural network for speech packet loss compensation may further include the following steps S1 and S2. Specifically, in step S1, the sample predicted audio is input into at least one pre-trained discriminator, and each discriminator outputs a discrimination result for the sample predicted audio. In step S2, based on the at least one discrimination result, the sample predicted audio, and the sample non-packet loss audio, a loss is calculated, and network parameters of the neural network to be trained are adjusted based on the loss.

In this implementation, a generative adversarial network (GAN) can be used to train the neural network. The neural network to be trained can serve as a generator, and its output of sample predicted audio can be input into a discriminator, which then outputs a judgment result for the sample predicted audio. The network parameters of the neural network to be trained are then adjusted based on the judgment result output by the discriminator. In this example, one or more discriminators can be used.

In some implementations, the at least one discriminator may include a first discriminator for determining the probability that the sample predicted audio is real audio, and may also include a second discriminator for determining the audio quality of the sample predicted audio.

For example, the first discriminator can include a frequency domain multi-resolution discriminator, a time domain multi-period discriminator, and so on. During training, the real audio without packet loss and the audio generated by the generator can be input into the first discriminator, and the first discriminator determines whether the input audio is the audio generated by the generator. The discriminator loss is calculated, and the network parameters of the first discriminator are updated through backpropagation. In this example, the loss of the first discriminator can be calculated using the following formula:

Where s can represent the real non-packet-loss audio; x can represent the packet-loss audio corresponding to s; G can represent the generator; D can represent the discriminator; G(x) can represent the result after the generator processes x; It can represent a computational expectation, and its subscript indicates the object of the computational expectation.

As mentioned above, the first discriminator may include a frequency domain multi-resolution discriminator. As shown in Figure 6, Figure 6 shows a schematic diagram of an example of a frequency domain multi-resolution discriminator. In the example shown in Figure 6, after the audio waveform (Waveform) is input into the frequency domain multi-resolution discriminator, the frequency domain multi-resolution discriminator can use short-time Fourier transform (STFT) with different window lengths and window shifts to transform the time domain waveform. Features of different spectral resolutions are downsampled using two-dimensional convolution. For example, the window lengths can be [30, 60, 120, 240, 480, 960], respectively, and the input audio can be discriminated from different frequency division ratios. It can be understood that the frequency domain multi-resolution discriminator shown in Figure 6 is a well-known existing discriminator, and the contents of its various parts are shown in the figure and will not be repeated here.

The first discriminator can also include a time-domain multi-period discriminator (MPD). As shown in Figure 7, Figure 7 shows a schematic diagram of an example of a time-domain multi-period discriminator. In the example shown in Figure 7, the time-domain multi-period discriminator can fold the input one-dimensional sample point sequence into a two-dimensional plane with a certain period, and then apply two-dimensional convolution for processing. Specifically, the sub-discriminator of each specific period is first padded to ensure that the number of sample points is an integer multiple of the period to facilitate folding into a two-dimensional plane. Next, it enters multiple convolutional layers, and the output channel number can be, for example, [32, 128, 512, 1024] respectively. After convolution, leaky_relu activation is used. Finally, post-processing is performed using a convolutional layer with an input channel number, for example, 1024, and an output channel number, for example, 1, and flattening is used as the final output of the time-domain multi-period discriminator. It can be understood that the time-domain multi-period discriminator shown in Figure 7 is a well-known existing discriminator. The contents of its various parts are shown in the figure and will not be repeated here.

As an example, the second discriminator can be a MetricGAN discriminator. The packet loss-compensated audio can be fed into the MetricGAN discriminator to estimate the PESQ (Perceptual Evaluation of Speech Quality), and the loss can be calculated by comparing it with the actual PESQ obtained through the Python library. The MetricGAN discriminator is then updated based on the loss. For example, the loss of the MetricGAN discriminator can be calculated using the following loss function:

Where s can represent the real non-packet-loss audio; x can represent the packet-loss audio corresponding to s; G can represent the generator; D can represent the discriminator; G(x) can represent the result after the generator processes x. Q' can represent the pypesq library function. Can represent computational expectations.

As shown in Figure 8, Figure 8 shows a schematic diagram of an example of MetricGAN discriminator training. In the example shown in Figure 8, for each input, the MetricGAN discriminator can output a predicted PESQ score (Predicted PESQ score). As an example, the MetricGAN discriminator can include four layers of downsampled two-dimensional convolutions and three layers of fully connected layers, where the four layers of downsampled two-dimensional convolutions are followed by three layers of fully connected layers. The output of the MetricGAN discriminator is a predicted PESQ score. During the training phase, the predicted PESQ score is first compared with the true PESQ score output by the pypesq library function to calculate the mean squared error (MSE) loss, and the network parameters of the MetricGAN discriminator are updated. Afterwards, the MSE loss is calculated with the maximum PESQ value, and the network parameters of the generator are updated.

In some implementations, the above-mentioned method for training a neural network for voice packet loss compensation may further include the following steps 1 to 3. Specifically: Step 1, the sample prediction audio output by the neural network to be trained and its corresponding sample non-packet loss audio are respectively input into a pre-trained speech recognition model. In this example, the speech recognition model can be any existing speech recognition model, for example, it can be a whisper model. Step 2, obtain the coding layer features of the sample prediction audio and its corresponding sample non-packet loss audio in the speech recognition model. Step 3, based on the difference loss of the two obtained coding layer features, adjust the network parameters of the neural network to be trained. For example, the network parameters of the neural network to be trained can be adjusted with the goal of minimizing the difference loss.

In this implementation, taking the whisper model as an example, the difference loss of the two encoding layer features can be calculated using the following formula:

in, It can represent the coding layer features of the whisper model corresponding to the sample prediction audio, and W(s) can represent the coding layer features of the whisper model corresponding to the sample unlost audio.

It is understood that the network parameters of the neural network to be trained can also be adjusted based on the sample predicted audio and its corresponding sample unpacked audio. For example, the difference loss between the sample predicted audio and its corresponding sample unpacked audio can be calculated, and the network parameters of the neural network to be trained can be adjusted with the goal of minimizing the difference loss. For example, the difference loss can be calculated using the following formula:

Wherein, _Lmae can represent the mae (Mean Absolute Error) loss in the time domain, N can represent the audio length, and _si can represent the audio without packet loss at the i-th sampling point. It can represent the sample prediction audio of the i-th sampling point.

Here, L _plcpa can represent the mean square error of the amplitude spectrum compression in the frequency domain, and S(t,f) can represent the spectrum corresponding to the sample without packet loss audio. It can represent the spectrum corresponding to the sample prediction audio, t and f can represent the subscripts of the time dimension and frequency dimension respectively, p can represent the compression coefficient, j and It is possible to represent some phases related to the Fourier transform. It can be expressed as the mean square error of the amplitude spectrum, It can be expressed as the mean square error of the phase.

For another example, the difference loss can be calculated using the following formula:

Where s represents the true audio without packet loss; x represents the packet loss audio corresponding to s; G represents the generator; and D represents the discriminator. G(x) represents the result of processing x by the generator, and PESQ _max represents the maximum value of the PESQ indicator.

The above describes the training process of a neural network for voice packet loss compensation. The neural network obtained in this way can compensate for packet loss of input audio.

Next, please refer to Figure 9, which illustrates a neural network-based voice packet loss compensation method according to one embodiment. This method can be performed by any device, equipment, platform, or device cluster with computing and processing capabilities. As shown in Figure 9, the neural network-based voice packet loss compensation method may include steps 901 to 903. Specifically, step 901: Obtain a pre-trained neural network for voice packet loss compensation. In this embodiment, a neural network for voice packet loss compensation trained using the method described in Figure 2 can be obtained. This neural network can perform packet loss compensation based on packet loss audio and frame loss location information corresponding to the packet loss audio. Step 902: Receive audio to be processed and frame loss location information corresponding to the audio to be processed. Step 903: Input features generated based on the audio to be processed and the frame loss location information corresponding to the audio to be processed are input into the neural network to obtain packet loss-compensated audio corresponding to the audio to be processed. According to another embodiment, a device for training a neural network for voice packet loss compensation is provided. This device for training a neural network for voice packet loss compensation can be deployed on any device, equipment, platform, device cluster, or the like with computing and processing capabilities.

Figure 10 shows a schematic block diagram of an apparatus for training a neural network for voice packet loss compensation according to one embodiment. The apparatus shown in Figure 10 is used to perform the method shown in Figure 2. The neural network to be trained includes an encoder layer, an intermediate layer, and a decoder layer, with the intermediate layer connected between the encoder layer and the decoder layer. As shown in Figure 10, the device 100 for training a neural network for speech packet loss compensation includes: an acquisition unit 1001, configured to acquire a training sample set, each training sample including sample packet loss audio, its corresponding sample frame loss position information and sample non-packet loss audio; a generation unit 1002, configured to generate input features based on the sample packet loss audio and its corresponding sample frame loss position information; a first input unit 1003, configured to input the above-mentioned input features into the above-mentioned neural network to be trained; a second input unit 1004, configured to input the features output by the above-mentioned intermediate layer into a pre-trained fundamental frequency prediction network, and the above-mentioned fundamental frequency prediction network outputs a predicted fundamental frequency; an adjustment unit 1005, configured to adjust the network parameters of the above-mentioned encoder layer and the above-mentioned intermediate layer based on the above-mentioned predicted fundamental frequency and the actual fundamental frequency calculated based on the sample non-packet loss audio.

In some optional implementations of this embodiment, the neural network to be trained is a neural network with a U-Net structure, and the intermediate layer is a bottleneck layer in the U-Net structure.

In some optional implementations of this embodiment, the fundamental frequency prediction network includes a bidirectional long short-term memory network.

In some optional implementations of this embodiment, the features output by the intermediate layer include features of the frame corresponding to the frame loss position corresponding to the sample frame loss position information, and the predicted fundamental frequency output by the fundamental frequency prediction network includes the fundamental frequency of the frame corresponding to the frame loss position.

In some optional implementations of this embodiment, the generation unit 1002 is further configured to perform sub-band decomposition on the above-mentioned sample packet loss audio to obtain multiple sub-bands; and generate input features based on the conversion results of the above-mentioned multiple sub-bands into the time-frequency domain and the above-mentioned sample frame loss position information.

In some optional implementations of this embodiment, the encoder layer includes multiple encoders, each encoder includes a gated convolution layer and a time-frequency dilated convolution layer, and the time-frequency dilated convolution layer is used to extract features through dilated convolution in the time dimension and the frequency dimension.

In some optional implementations of this embodiment, the decoder layer includes multiple decoders, each decoder includes a first branch and a second branch in parallel, the first branch is used to predict the real part of the audio, and the second branch is used to predict the imaginary part of the audio; the neural network to be trained outputs sample predicted audio based on the real and imaginary parts of the predicted audio output by the encoder layer; and the device 1000 also includes: a third input unit (not shown in the figure), configured to input the sample predicted audio into at least one pre-trained discriminator, and each discriminator outputs a discrimination result for the sample predicted audio; a calculation unit (not shown in the figure), configured to calculate the loss based on at least one discrimination result, the sample predicted audio and the sample non-lost audio, and adjust the network parameters of the neural network to be trained based on the loss.

In some optional implementations of this embodiment, the at least one discriminator includes a first discriminator for determining the probability that the sample predicted audio is real audio and a second discriminator for determining the audio quality of the sample predicted audio.

In some optional implementations of this embodiment, the above-mentioned device 1000 also includes: a fourth input unit (not shown in the figure), configured to input the sample prediction audio output by the above-mentioned neural network to be trained and its corresponding sample non-packet-lost audio into a pre-trained speech recognition model respectively; a coding layer feature acquisition unit (not shown in the figure), configured to obtain the coding layer features of the above-mentioned sample prediction audio and its corresponding sample non-packet-lost audio in the above-mentioned speech recognition model; a network parameter adjustment unit (not shown in the figure), configured to adjust the network parameters of the above-mentioned neural network to be trained based on the difference loss of the two obtained coding layer features.

According to another embodiment, a neural network-based voice packet loss compensation device is provided. The neural network-based voice packet loss compensation device can be deployed on any device, equipment, platform, device cluster, etc. with computing and processing capabilities.

FIG11 shows a schematic block diagram of a speech packet loss compensation device based on a neural network according to an embodiment. The device shown in FIG11 is used to execute the method shown in FIG9 . As shown in FIG11 , the speech packet loss compensation device 1100 based on a neural network includes: a model acquisition unit 1101, configured to acquire a neural network for speech packet loss compensation trained according to the method described in FIG2 ; a receiving unit 1102, configured to receive audio to be processed and frame loss position information corresponding to the audio to be processed; and a feature input unit 1103, configured to input input features generated based on the audio to be processed and the frame loss position information corresponding to the audio to be processed into the neural network to obtain audio after packet loss compensation corresponding to the audio to be processed.

The above-mentioned device embodiments correspond to the method embodiments. For detailed descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For detailed descriptions, please refer to the corresponding method embodiments.

According to another embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in FIG. 2 or FIG. 9 .

According to yet another embodiment, an electronic device is provided, including a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, the method described in FIG. 2 or FIG. 9 is implemented.

The foregoing describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily need to be performed in the specific order shown or in a sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

12, which shows a schematic diagram of the structure of an electronic device 1200 suitable for implementing the embodiments of the present application. The electronic device shown in FIG12 is only an example and should not limit the functions and scope of use of the embodiments of the present application.

As shown in FIG12 , the electronic device 1200 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of the electronic device 1200 are also stored in the RAM 1203. The processing device 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input/output (I/O) interface 1205 is also connected to the bus 1204.

Typically, the following devices can be connected to the I/O interface 1205: an input device 1206 including, for example, a touch screen, a touchpad, a keyboard, a mouse, etc.; an output device 1207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1208 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1209. The communication device 1209 can allow the electronic device 1200 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 12 shows an electronic device 1200 with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively. Each box shown in Figure 12 may represent one device, or may represent multiple devices as needed.

In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 1209, or installed from the storage device 1208, or installed from the ROM 1202. When the computer program is executed by the processing device 1201, the above-mentioned functions defined in the method of the embodiment of the present application are performed.

An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the method provided in the present disclosure.

It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (Radio Frequency), or any suitable combination thereof.

The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a training sample set, wherein each training sample includes a sample packet loss audio, its corresponding sample frame loss position information, and a sample non-packet loss audio; generates input features based on the sample packet loss audio and its corresponding sample frame loss position information; inputs the input features into the neural network to be trained, wherein the neural network to be trained includes an encoder layer, an intermediate layer, and a decoder layer, wherein the intermediate layer is connected between the encoder layer and the decoder layer; inputs the features output by the intermediate layer into a pre-trained fundamental frequency prediction network, and the fundamental frequency prediction network outputs a predicted fundamental frequency; and adjusts the network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequency and the true fundamental frequency calculated based on the sample non-packet loss audio.

Computer program code for performing the operations of the embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or electronic device. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

The various embodiments of this disclosure are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences between the other embodiments. In particular, the storage medium and computing device embodiments are described briefly because they are generally similar to the method embodiments. For relevant portions, reference can be made to the description of the method embodiments.

Those skilled in the art will appreciate that, in one or more of the above examples, the functions described in the embodiments of the present disclosure may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

The specific implementation methods described above further illustrate the purpose, technical solutions, and beneficial effects of the embodiments of the present disclosure. It should be understood that the above description is only a specific implementation method of the embodiments of the present disclosure and is not intended to limit the scope of protection of the present disclosure. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims (14)

A method for compensating voice packet loss based on a neural network, comprising: Obtaining a pre-trained neural network for voice packet loss compensation; receiving audio to be processed and frame loss position information corresponding to the audio to be processed; and Input features generated based on the audio to be processed and the frame loss position information corresponding to the audio to be processed are input into the neural network to obtain audio after packet loss compensation corresponding to the audio to be processed. A method for training a neural network for voice packet loss compensation, wherein the neural network to be trained includes an encoder layer, an intermediate layer, and a decoder layer, wherein the intermediate layer is connected between the encoder layer and the decoder layer, and the method includes: Obtaining a training sample set, wherein each training sample includes sample packet loss audio, its corresponding sample frame loss position information, and sample non-packet loss audio; Generate input features based on the sample packet loss audio and its corresponding sample frame loss position information; Inputting the input features into the neural network to be trained; Inputting the features output by the intermediate layer into a pre-trained fundamental frequency prediction network, and having the fundamental frequency prediction network output a predicted fundamental frequency; and Based on the predicted fundamental frequency and the actual fundamental frequency calculated based on the sampled unpacked audio, network parameters of the encoder layer and the intermediate layer are adjusted. The method according to claim 2, wherein the neural network to be trained is a neural network with a U-Net structure, and wherein the intermediate layer is a bottleneck layer in the U-Net structure. The method according to claim 2, wherein the fundamental frequency prediction network comprises a bidirectional long short-term memory network. The method according to claim 2, wherein the features output by the intermediate layer include features of the frame corresponding to the frame loss position corresponding to the sample frame loss position information, and the predicted fundamental frequency output by the fundamental frequency prediction network includes the fundamental frequency of the frame corresponding to the frame loss position. The method according to claim 2, wherein generating input features based on the sample packet loss audio and its corresponding sample frame loss position information comprises: Performing sub-band decomposition on the sample packet loss audio to obtain multiple sub-bands; An input feature is generated based on the conversion results of the multiple sub-bands into the time-frequency domain and the sample frame loss position information. The method according to claim 2, wherein the encoder layer includes multiple encoders, each encoder includes a gated convolution layer and a time-frequency dilated convolution layer, and the time-frequency dilated convolution layer is used to extract features through dilated convolution in the time dimension and the frequency dimension. The method according to claim 2, wherein the decoder layer includes a plurality of decoders, each decoder includes a first branch and a second branch in parallel, the first branch is used to predict the real part of the audio, and the second branch is used to predict the imaginary part of the audio; the neural network to be trained outputs sample predicted audio based on the real part and imaginary part of the predicted audio output by the encoder layer; and the method further includes: Inputting the sample prediction audio into at least one pre-trained discriminator, and having each discriminator output a discrimination result for the sample prediction audio; Based on at least one discrimination result, the sample predicted audio and the sample non-packet-loss audio, a loss is calculated, and network parameters of the neural network to be trained are adjusted based on the loss. The method according to claim 8, wherein the at least one discriminator includes a first discriminator for discriminating the probability that the sample predicted audio is real audio and a second discriminator for discriminating the audio quality of the sample predicted audio. The method according to any one of claims 2 to 9, further comprising: Inputting the sample prediction audio output by the neural network to be trained and its corresponding sample unpacked audio into the pre-trained speech recognition model respectively; Obtaining coding layer features of the sample predicted audio and its corresponding sample unpacked audio in the speech recognition model; and Based on the obtained difference loss of the two coding layer features, the network parameters of the neural network to be trained are adjusted. A device for training a neural network for voice packet loss compensation, wherein the neural network to be trained includes an encoder layer, an intermediate layer, and a decoder layer, wherein the intermediate layer is connected between the encoder layer and the decoder layer, and the device includes: An acquisition unit is configured to acquire a training sample set, wherein each training sample includes a sample packet loss audio, its corresponding sample frame loss position information, and a sample non-packet loss audio; A generating unit configured to generate input features based on the sample packet loss audio and its corresponding sample frame loss position information; A first input unit is configured to input the input feature into the neural network to be trained; A second input unit is configured to input the features output by the intermediate layer into a pre-trained fundamental frequency prediction network, so that the fundamental frequency prediction network outputs a predicted fundamental frequency; and The adjustment unit is configured to adjust network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequency and the real fundamental frequency calculated based on the sampled unpacked audio. A speech packet loss compensation device based on a neural network, comprising: A model acquisition unit configured to acquire a pre-trained neural network for voice packet loss compensation; a receiving unit configured to receive audio to be processed and frame loss position information corresponding to the audio to be processed; and The feature input unit is configured to input input features generated based on the audio to be processed and the frame loss position information corresponding to the audio to be processed into the neural network to obtain audio after packet loss compensation corresponding to the audio to be processed. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 10. An electronic device comprises a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 10 is implemented.