CN115359779A - Speech synthesis method, apparatus, computer device and storage medium - Google Patents

Abstract

The present application relates to a speech synthesis method, device, computer equipment and storage medium. The method includes: determining the coded segment of the sound spectrum to be predicted by encoding the phoneme feature information corresponding to the Chinese text; in each round of iterative prediction, determining the current sound spectrum feature corresponding to the coded segment of the current round of prediction, and The attention weight used when predicting the current sound spectrum feature in the current round; when the attention weight reaches the preset weight threshold, the preset decoding corresponding to the coded segment is based on the length of the current sound spectrum feature The number of times is adjusted to obtain the adjusted preset decoding times; when the number of rounds of iterative prediction reaches the preset decoding times, the iteration is stopped, and the current sound spectrum feature is determined as the target corresponding to the coding segment Spectrum features; the target spectrum features are used for synthesizing the speech corresponding to the Chinese text. Adopting the method can improve the accuracy of speech synthesis.

Description

Speech synthesis method, device, computer equipment and storage medium

technical field

The present application relates to the technical field of speech processing, in particular to a speech synthesis method, device, computer equipment and storage medium.

Background technique

With the development of speech processing technology, speech synthesis technology has emerged. Speech synthesis technology can convert the text information generated by the computer itself or input from the outside into intelligible and fluent spoken Chinese output, so as to realize the voice communication of computer equipment.

The traditional autoregressive-based speech synthesis model requires a large amount of training data. If the training data is not enough, the speech synthesis model is prone to end early or fail to complete the decoding when performing sound spectrum prediction, resulting in wrong sound spectrum prediction, resulting in inaccurate Perform speech synthesis.

Contents of the invention

Based on this, it is necessary to provide a speech synthesis method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy for the above technical problems.

In a first aspect, the present application provides a speech synthesis method. The methods include:

Determine the coding segment of the sound spectrum to be predicted by encoding the phoneme feature information corresponding to the Chinese text;

In each round of iterative prediction, determine the current sound spectrum feature corresponding to the coding segment of the current round of prediction, and the attention weight used when predicting the current sound spectrum feature in the current round;

When the attention weight reaches the preset weight threshold, adjust the preset decoding times corresponding to the encoding segment based on the length of the current sound spectrum feature, to obtain the adjusted preset decoding times;

When the number of rounds of iterative prediction reaches the preset number of times of decoding, the iteration is stopped, and the current spectral feature is determined as the target spectral feature corresponding to the encoding segment; the target spectral feature is used for synthesis The voice corresponding to the Chinese text.

In one of the embodiments, the encoding of the phoneme feature information corresponding to the Chinese text to determine the coded segment of the sound spectrum to be predicted includes:

Determine the phoneme feature information corresponding to the Chinese text;

Encoding the phoneme feature information to obtain a text encoding sequence corresponding to the Chinese text;

According to the preset time step, the coding segment to be subjected to sound spectrum prediction is determined from the text coding sequence.

In one embodiment, the method also includes:

In the case that the number of rounds of iterative prediction does not reach the preset number of decoding times, the next round is used as the current round, and the current sound spectrum feature corresponding to the coding segment of the current round of prediction and the current sound spectrum feature corresponding to the current round of prediction are returned. The step of using attention weights for spectral features continues.

In one of the embodiments, the method also includes the step of calculating the attention weight; the step of calculating the attention weight includes:

In each round of iterative prediction, determine the current attention vector used when predicting the current sound spectrum feature in this round;

Based on the similarity between the current attention vector and the coding segment, calculate the attention weight corresponding to the current attention vector; the attention weight is used to indicate that the feature extraction is performed on the attention vector to obtain A current feature vector; the current feature vector is used to predict the current sound spectrum features.

In one of the embodiments, when the attention weight reaches the preset weight threshold, the preset decoding times corresponding to the coded segment is adjusted based on the length of the current sound spectrum feature, and the adjusted The preset decoding times for include:

When the attention weight reaches a preset weight threshold, determine a decoding round offset value corresponding to the coding segment;

According to the decoding round offset value and the length of the current sound spectrum feature, the preset decoding times corresponding to the coded segment is adjusted to obtain the adjusted preset decoding times.

In one embodiment, the method also includes:

predicting end identification information corresponding to the coded segment based on the current sound spectrum feature;

In the case that the end identification information cannot satisfy the preset end condition, compare the number of rounds of iterative prediction with the preset number of decoding times, and execute the number of rounds of iterative prediction and the preset number of decoding times In the case of a match, stop the iteration step.

In one embodiment, the method also includes:

When the end identification information satisfies the preset end condition, the iteration is stopped, and the current sound spectrum feature is determined as the target sound spectrum feature corresponding to the encoding segment.

In a second aspect, the present application also provides a speech synthesis device. The devices include:

The encoding module is used to determine the encoding segment of the sound spectrum to be predicted by encoding the phoneme feature information corresponding to the Chinese text;

The decoding module, in each round of iterative prediction, determines the current sound spectrum feature corresponding to the encoded segment of the current round of prediction, and the attention weight used when predicting the current sound spectrum feature in the current round; when the attention weight reaches In the case of a preset weight threshold, adjust the preset decoding times corresponding to the coded segment based on the length of the current sound spectrum feature to obtain the adjusted preset decoding times; when the number of rounds of iterative prediction reaches the preset When the number of decoding times is set, the iteration is stopped, and the current spectral feature is determined as the target spectral feature corresponding to the coded segment; the target spectral feature is used to synthesize the speech corresponding to the Chinese text.

In one of the embodiments, the encoding module is also used to determine the phoneme feature information corresponding to the Chinese text; encode the phoneme feature information to obtain the text encoding sequence corresponding to the Chinese text; according to the preset time step, The coding segment to be subjected to sound spectrum prediction is determined from the text coding sequence.

In one of the embodiments, the decoding module is further configured to use the next round as the current round when the number of rounds of iterative prediction does not reach the preset number of decoding times, and return to the code for determining the current round of prediction The steps of the current sound spectrum feature corresponding to the segment and the attention weight used when predicting the current sound spectrum feature in this round continue to be executed.

In one of the embodiments, the decoding module is also used to determine the current attention vector used when predicting the current sound spectrum feature in each round of iterative prediction; based on the current attention vector and the coding section The similarity between calculates the attention weight corresponding to the current attention vector; the attention weight is used to indicate that the feature extraction is performed on the attention vector to obtain the current feature vector; the current feature vector is used for prediction Current spectral features.

In one of the embodiments, the decoding module is further configured to determine the decoding round offset value corresponding to the coding segment when the attention weight reaches a preset weight threshold; according to the decoding round The offset value and the length of the current sound spectrum feature are used to adjust the preset decoding times corresponding to the encoded segment to obtain the adjusted preset decoding times.

In one of the embodiments, the decoding module is further configured to predict the end identification information corresponding to the encoded segment based on the current sound spectrum feature; if the end identification information cannot meet the preset end condition, the The number of rounds of iterative prediction is compared with the preset number of decoding times, and the step of stopping the iteration is performed when the number of rounds of iterative prediction matches the preset number of decoding times.

In one of the embodiments, the decoding module is further configured to stop iteration when the end identification information satisfies a preset end condition, and determine the current sound spectrum feature as the target corresponding to the encoding segment Spectral features.

In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps in each embodiment of the method described in this application when executing the computer program.

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the steps in each embodiment of the method described in this application are realized.

In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the steps in each embodiment of the method described in this application are implemented.

The above speech synthesis method, device, computer equipment, storage medium and computer program product, by encoding the phoneme feature information corresponding to the Chinese text, determine the coding segment of the sound spectrum to be predicted; in each round of iterative prediction, determine the current round of prediction The current spectral feature corresponding to the coding segment, and the attention weight used when predicting the current spectral feature in this round; when the attention weight reaches the preset weight threshold, the prediction corresponding to the coding segment is based on the length of the current spectral feature. Set the number of decodings to be adjusted to obtain the adjusted preset number of decodings; when the number of rounds of iterative prediction reaches the preset number of decodings, stop the iteration, and determine the current sound spectrum feature as the target sound spectrum feature corresponding to the encoding segment; The target spectral features are used to synthesize speech corresponding to Chinese text. By adjusting the preset decoding times according to the attention weight and the length of the current sound spectrum feature in each round of iterative prediction, when the number of rounds of iterative prediction reaches the preset decoding times, the iterative prediction is controlled to end, thereby Accurate target sound spectrum features can be obtained to improve the accuracy of speech synthesis.

Description of drawings

Fig. 1 is the application environment diagram of speech synthesis method in an embodiment;

Fig. 2 is a schematic flow chart of a speech synthesis method in an embodiment;

Fig. 3 is a schematic diagram of an acoustic spectrum prediction model in an embodiment;

Fig. 4 is the structural block diagram of speech synthesis device in an embodiment;

Fig. 5 is an internal structural diagram of a computer device in an embodiment;

Fig. 6 is an internal structure diagram of a computer device in another embodiment.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

The speech synthesis method provided in the embodiment of the present application may be applied in the application environment shown in FIG. 1 . Wherein, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that needs to be processed by the server 104 . The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The server 104 can determine the coded segment of the sound spectrum to be predicted by encoding the phoneme feature information corresponding to the Chinese text; the server 104 can determine the current sound spectrum feature corresponding to the coded segment of the current round of prediction in each round of iterative prediction, and this The attention weight used when predicting the current sound spectrum feature in turn; the server 104 can adjust the preset decoding times corresponding to the encoded segment based on the length of the current sound spectrum feature when the attention weight reaches the preset weight threshold, The adjusted preset decoding times are obtained; the server 104 can stop the iteration when the number of rounds of iterative prediction reaches the preset decoding times, and determine the current sound spectrum feature as the target sound spectrum feature corresponding to the encoding segment; the target sound spectrum Features are used to synthesize speech corresponding to Chinese text. It can be understood that the terminal 102 can display the synthesized speech based on the target feature.

Among them, the terminal 102 can be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. . Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, and the like. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

In one embodiment, as shown in FIG. 2 , a speech synthesis method is provided. This embodiment uses the method applied to a server as an example for illustration. It can be understood that this method can also be applied to a terminal or A system including a terminal and a server, and realized through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

Step 202, by encoding the phoneme feature information corresponding to the Chinese text, determine the code segment of the sound spectrum to be predicted; in each round of iterative prediction, determine the current sound spectrum feature corresponding to the code segment of the current round of prediction, and the current round of prediction. Attention weights to use when spectral features.

Wherein, the phoneme feature information is used to indicate the phonetic feature contained in the Chinese text. It can be understood that a phoneme is the smallest unit of speech divided according to the natural attributes of speech, and is the smallest pronunciation unit in speech. Chinese texts include a variety of phoneme features, such as initials, finals, punctuation marks, tones, and soft tones. These features Together, they determine the phonetic structure corresponding to the Chinese text, that is, how to emit the corresponding phonetics of the Chinese text. The coding segment is part of the coding information corresponding to the Chinese text. It can be understood that the encoding information corresponding to the Chinese text can be obtained by encoding the phoneme feature information corresponding to the Chinese text, and the encoding segment is essentially a piece of information in the encoding information. The current sound spectrum features include acoustic features corresponding to the coded segment, such as frequency information and amplitude information of sound waves. The attention weight is the intermediate data in the process of predicting the current spectral feature corresponding to the encoded segment. It can be understood that the attention mechanism is a problem-solving method proposed by imitating human attention. The core goal is to assign different weights to some nodes of the neural network in order to select from a large number of information that is more critical to the current task goal. Information. Attention weights can indicate the criticality of information to extract key features. Speech synthesis (text to speech) is to convert the input Chinese text into speech.

Specifically, the server may acquire phoneme feature information corresponding to the Chinese text, and determine the encoded segment of the sound spectrum to be predicted by encoding the phoneme feature information. It can be understood that in each round of iterative prediction, in order to predict the current spectral features, intermediate data will be used, and the attention weight is the intermediate data generated during the process of predicting the current spectral features. In each round of iterative prediction, the server may determine the current sound spectrum feature corresponding to the coding segment of the current round of prediction, and the attention weight used in the current round of prediction of the current sound spectrum feature.

In an embodiment, the current acoustic spectrum feature may be at least one of a Mel spectrum and a Mel cepstrum. Mel Bank Features, an acoustic feature of a speech signal, contains rich speech information and is often used in speech recognition and speech synthesis.

Step 204: When the attention weight reaches the preset weight threshold, adjust the preset decoding times corresponding to the coded segment based on the length of the current sound spectrum feature to obtain the adjusted preset decoding times.

Wherein, the preset weight threshold is a threshold set for the attention weight.

Specifically, the degree of alignment between the current spectral feature and the encoded segment is positively correlated with the attention weight. It can be understood that the greater the attention weight, the higher the alignment. When the attention weight reaches the preset weight threshold, it can be determined that the current spectral feature is aligned with the encoding segment to a certain extent. At this time, the server can adjust the preset decoding times corresponding to the encoding segment to obtain the adjusted Default decoding times. Wherein, the adjusted preset decoding times is larger than the previous preset decoding times.

In one embodiment, when the attention weight reaches the preset weight threshold and the length of the current sound spectrum feature matches the length of the coding segment, the corresponding prediction of the coding segment is based on the length of the current sound spectrum feature. Set the number of times of decoding to adjust to obtain the adjusted number of times of preset decoding.

In an embodiment, matching the length between the current sound spectrum feature and the coding segment may mean that the time domain length corresponding to the current sound spectrum feature is greater than the time domain length corresponding to the coding segment. It can be understood that the implementation manner is not limited to the length in the time domain, and the length in the frequency domain can also be used to determine the matching situation between the current sound spectrum feature and the length between the coding segments.

Step 206, when the number of rounds of iterative prediction reaches the preset number of decoding times, stop the iteration, and determine the current sound spectrum feature as the target sound spectrum feature corresponding to the coded segment.

Among them, the target sound spectrum feature is used to synthesize the speech corresponding to the Chinese text.

Specifically, although it can be considered that the current spectral feature is aligned with the encoded segment to a certain extent, it is not accurate enough to directly output the current spectral feature. The server can continue to predict the current spectral feature corresponding to the encoded segment, and iteratively predict When the number of rounds reaches the preset number of decoding times, the iteration is stopped, and the current sound spectrum feature is determined as the target sound spectrum feature corresponding to the encoding segment.

In the above speech synthesis method, by encoding the phoneme feature information corresponding to the Chinese text, the code segment of the sound spectrum to be predicted is determined; in each round of iterative prediction, the current sound spectrum feature corresponding to the code segment predicted in the current round, and the The attention weight used when predicting the current spectral feature in rounds; when the attention weight reaches the preset weight threshold, the preset decoding times corresponding to the coding segment are adjusted based on the length of the current spectral feature, and the adjusted prediction is obtained. Set the number of decoding times; when the number of rounds of iterative prediction reaches the preset number of decoding times, stop the iteration, and determine the current spectral feature as the target spectral feature corresponding to the encoding segment; the target spectral feature is used to synthesize the corresponding Chinese text voice. By adjusting the preset decoding times according to the attention weight and the length of the current sound spectrum feature in each round of iterative prediction, when the number of rounds of iterative prediction reaches the preset decoding times, the iterative prediction is controlled to end, thereby Accurate target sound spectrum features can be obtained to improve the accuracy of speech synthesis.

In one embodiment, by encoding the phoneme feature information corresponding to the Chinese text, determining the coded segment of the sound spectrum to be predicted includes: determining the phoneme feature information corresponding to the Chinese text; encoding the phoneme feature information to obtain the text corresponding to the Chinese text Coding sequence: according to the preset time step, determine the coding segment to be subjected to sound spectrum prediction from the text coding sequence.

Wherein, the text coding sequence is the intermediate data conforming to the corresponding format in the process of predicting the feature of the sound spectrum. It can be understood that the server cannot directly predict the sound spectrum feature from the phoneme feature information, and the sound spectrum feature can only be predicted by performing format conversion on the phoneme feature information multiple times to extract features. A coded segment is a partial coded sequence in a text coded sequence.

Specifically, the server can determine phoneme feature information in the Chinese text. The server can use the encoder to extract and compress the phoneme feature information into a fixed-length context vector to obtain a text encoding sequence corresponding to the Chinese text. The server can determine the code segment to be predicted from the text code sequence according to the preset time step. It can be understood that the text encoding sequence includes multiple encoding segments, and the server may perform sound spectrum prediction on the multiple encoding segments successively according to time sequence.

In one embodiment, the server can recognize 237 kinds of phonemes, including normal 1 to 5 tones, and common punctuation marks. Among them, tone 5 represents a soft voice.

In an embodiment, the server can obtain the phoneme feature information corresponding to the Chinese text by querying a standard phoneme dictionary.

In one embodiment, the encoder includes a word encoding layer, three convolutional layers, and a bidirectional LSTM network layer. Among them, the word encoding is to transform the text into a numerical matrix that can be recognized and processed by the computer through some data transformation or mapping. In the modeling process, the text information is represented by the numerical matrix and participates in training and calculation. The server can convert the phoneme feature information into a text vector through the word coding layer, and extract the coding features of the text vector through three convolutional layers and a bidirectional long-short memory network layer to obtain the text coding sequence corresponding to the Chinese text.

In this embodiment, by determining the phoneme feature information corresponding to the Chinese text and encoding the phoneme feature information, the text coding sequence corresponding to the Chinese text is obtained, and then according to the preset time step, the sound spectrum prediction to be performed is determined from the text coding sequence The coded section can be used to predict the sound spectrum of the coded section, so as to realize the speech synthesis of Chinese text.

In one embodiment, the method further includes: when the number of rounds of iterative prediction does not reach the preset number of decoding times, taking the next round as the current round, returning to determine the current sound spectrum feature corresponding to the coding segment of the current round of prediction, and The step of attention weights used when predicting the current spectral feature in this round continues.

Specifically, when the attention weight reaches the preset weight threshold, more prediction is required to improve the accuracy of sound spectrum prediction. It will be appreciated that this part of the multi-prediction can be a silent feature. When the number of rounds of iterative prediction does not reach the preset number of decoding times, the server can use the next round as the current round, and return the current sound spectrum feature corresponding to the coding segment of the current round of prediction, and the time when the current round of prediction of the current sound spectrum feature The step of using the attention weights continues.

In one embodiment, when the attention weight reaches the preset weight threshold and the current sound spectrum feature matches the length between the encoding segments, the server can directly predict another part of the current sound spectrum feature Mute feature to get the target sound spectrum feature.

In one embodiment, the length of the current spectral feature matches the number of rounds of iterative prediction. The server can judge whether the number of rounds of iterative prediction reaches the preset decoding times by comparing the length of the current sound spectrum feature with the preset decoding times. It can be understood that after each round of iterative prediction, the increased length of the current sound spectrum feature is fixed. Therefore, the server can obtain the number of iterative prediction rounds by determining the length of the current sound spectrum feature.

In this embodiment, when the number of rounds of iterative prediction does not reach the preset number of decoding times, the next round is used as the current round, and the current sound spectrum feature corresponding to the coding segment of the current round of prediction and the current sound spectrum feature corresponding to the current round of prediction are returned. The steps of attention weight used in spectral features are continued, and the accuracy of sound spectrum prediction is improved through multiple rounds of iterative prediction.

In one embodiment, the method further includes the step of calculating the attention weight; the step of calculating the attention weight includes: in each round of iterative prediction, determining the current attention vector used when predicting the current sound spectrum feature in this round; based on the current attention The similarity between the force vector and the encoding segment is used to calculate the attention weight corresponding to the current attention vector; the attention weight is used to indicate the feature extraction of the attention vector to obtain the current feature vector; the current feature vector is used to predict the current sound spectrum feature.

Specifically, in each round of iterative prediction, the server can determine the current attention vector used when predicting the current sound spectrum features in this round, and calculate the similarity between the current attention vector and the coded segment through the attention loop neural network layer, and obtain The similarity score is used as the input of the normalized exponential function (softmax function) to obtain the attention weight corresponding to the current attention vector. Among them, the recurrent neural network (Recurrent Neural Network, RNN) is a kind of recurrent neural network that takes sequence data as input, recurses in the evolution direction of the sequence, and connects all nodes (recurrent units) in a chain.

In one embodiment, the server can calculate the similarity score between the current attention vector and the coded segment by combining the two dimensions of context information and location. It can be understood that the text coding sequence includes complete context information, and the text coding sequence includes corresponding sequence position information. The text content information and state position information contained in the attention vector should be consistent with the context information and sequence position information in the encoding segment, the higher the consistency between the two, the higher the similarity score.

In an embodiment, the server may perform sound spectrum prediction on the coded segment through the decoder. Wherein, the decoder may include a preprocessing layer, an attention recurrent neural network layer and a decoding recurrent neural network layer. The server may determine the last sound spectrum feature predicted in the last round, and the last attention vector and the last feature vector used in the last round of prediction. The server can input the previous sound spectrum feature to the preprocessing layer, and after splicing the output of the preprocessing layer with the previous attention vector, input it to the attention loop neural network layer to obtain the current attention vector needed for the current round of prediction. The server can input the current attention vector and the previous feature vector to the decoding cycle neural network layer to obtain the current feature vector to be used in the current round of prediction.

In an embodiment, the server may input the current feature vector into the frame prediction network to obtain the current sound spectrum feature. The current sound spectrum feature may include at least one frame of mel spectrum. It can be understood that the server may predict the mel spectrum of the preset number of spectrum frames in each round of iterative prediction according to the preset number of spectrum frames. For example, the preset frequency spectrum frame number may be 3 frames, 4 frames or 5 frames.

In one embodiment, the server can determine the current coding segment to be predicted by the sound spectrum. When performing the first round of sound spectrum prediction on the current encoding segment, the server may determine the last attention vector corresponding to the last target sound spectrum feature predicted in the last round. The server can input the current encoding segment and the previous attention vector to the attention recurrent neural network layer to obtain the current attention vector needed for the current round of prediction.

In this embodiment, in each round of iterative prediction, the current attention vector used when predicting the current sound spectrum feature in this round is determined; based on the similarity between the current attention vector and the coding segment, the attention corresponding to the current attention vector is calculated. Power weight; the preset decoding times can be adjusted based on the attention weight in the future, so as to predict the sound spectrum more accurately and improve the accuracy of speech synthesis.

In one embodiment, when the attention weight reaches the preset weight threshold, the preset decoding times corresponding to the coding segment are adjusted based on the length of the current sound spectrum feature, and the adjusted preset decoding times include: When the attention weight reaches the preset weight threshold, determine the decoding round offset value corresponding to the coding segment; according to the decoding round offset value and the length of the current sound spectrum feature, perform the preset decoding times corresponding to the coding segment Adjust to obtain the adjusted preset decoding times.

Wherein, the decoding round offset value is used to indicate the offset of the preset decoding times.

Specifically, when the attention weight reaches the preset weight threshold, the server can determine the decoding round offset value corresponding to the encoded segment, and add the decoding round offset value to the preset decoding times to obtain the adjusted predicted Set the decoding times.

In an embodiment, the preset decoding times before adjustment corresponding to all the coding segments may be a preset fixed value. The server may determine the length of the current sound spectrum feature, and calculate the sum of the decoding round offset value and the length of the current sound spectrum feature to obtain the adjusted preset decoding times.

In this embodiment, when the attention weight reaches the preset weight threshold, the decoding round offset value corresponding to the coding segment is determined; based on the decoding round offset value, the preset decoding times corresponding to the coding segment are adjusted to obtain The adjusted preset decoding times, so as to control the iteration stop based on the preset decoding times, so as to perform sound spectrum prediction more accurately.

In one embodiment, the method further includes: predicting the end identification information corresponding to the encoded segment based on the current sound spectrum feature; in the case that the end identification information cannot meet the preset end condition, the number of rounds of iterative prediction is compared with the preset number of decoding times Compare, and execute the step of stopping the iteration when the number of rounds of iterative prediction reaches the preset number of decoding times.

Wherein, the end identification information is used to indicate the probability that the sound spectrum prediction ends.

Specifically, the server may determine the current feature vector corresponding to the current sound spectrum feature, and input the current feature vector into the end identification information corresponding to the linear projection layer predictive coding segment. The server may determine the length of the current sound spectrum feature when the end probability indicated by the end identification information cannot reach the preset end threshold. It can be understood that the length of the current sound spectrum feature may indicate the number of iterative prediction rounds the current sound spectrum feature is in. The server can determine the number of rounds up to the current iterative prediction based on the length of the current sound spectrum feature, compare the number of iterative prediction rounds with the adjusted preset decoding times, and perform the operation when the iterative prediction rounds reach the preset decoding times In the case of , stop the iteration step.

In an embodiment, the length value of the current sound spectrum feature may be consistent with the iteratively predicted round value. The server can directly compare the length of the current spectral feature with the adjusted preset decoding times. When the length of the current spectral feature matches the adjusted preset decoding times, the server can perform the rounds of iterative prediction. When the preset number of times of decoding is reached, the step of iterating is stopped. It can be understood that the matching situation may be the situation that the length value of the current sound spectrum feature is consistent with the preset decoding times value.

In an embodiment, in the case that the end identification information cannot satisfy the preset end condition, the server may return to perform an operation based on the length of the current sound spectrum feature when the attention weight reaches the preset weight threshold. The step of adjusting the preset decoding times corresponding to the coding segment to obtain the adjusted preset decoding times is to continue the iteration.

In an embodiment, the server may input the current feature vector to the frame prediction network and the linear projection layer respectively, so as to obtain the current sound spectrum feature and the end identification information corresponding to the predictive coding segment.

In one embodiment, the server may determine whether the attention weight reaches a preset weight threshold and whether the current sound spectrum feature matches the length between the encoded segments when the end identification information fails to meet the preset end condition.

In this embodiment, when the end identification information cannot meet the preset end condition, compare the number of rounds of iterative prediction with the preset number of decoding times, and perform , the step of stopping the iteration, the stop of the iteration is controlled by the criteria of the two dimensions of the end identification information and the preset number of decoding times, and the accuracy of the sound spectrum prediction is improved.

In an embodiment, the method further includes: when the end identification information satisfies the preset end condition, stop the iteration, and determine the current sound spectrum feature as the target sound spectrum feature corresponding to the encoding segment.

Specifically, when the end identification information satisfies the preset end condition and the length between the current sound spectrum feature and the current encoding segment matches, the server can control the iterative prediction to stop, and determine the current sound spectrum feature as the length corresponding to the encoding segment. Spectral characteristics of the target.

In one embodiment, when the end identification information satisfies the preset end condition and the length between the current sound spectrum feature and the current encoding segment matches, the server can stop the iteration and determine the current sound spectrum feature as the encoding segment corresponding to target sound spectrum features.

In this embodiment, when the end identification information satisfies the preset end condition, the iteration is stopped, and the current sound spectrum feature is determined as the target sound spectrum feature corresponding to the encoding segment, which can avoid the length of the sound spectrum when the preset end condition is met. too short question.

In one embodiment, a schematic diagram of an acoustic spectrum prediction model is provided as shown in FIG. 3 . The server can predict the sound spectrum of the Chinese text through the sound spectrum prediction model. The Spectral Prediction Model is a sequence-to-sequence generative model with a locality-sensitive attention mechanism. It can be understood that in the task of Chinese text-to-speech synthesis, the local sensitive attention mechanism reasonably distributes the weight of each element in the sequence, so that the model pays different attention to each part of the sequence, and pays more attention to the synthesis. The section with more closely related content. The spectral prediction model includes an encoder, a decoder, and a post-processing network. The encoder consists of a word encoding layer, three convolutional layers, and a bidirectional LSTM network layer. The decoder includes a preprocessing layer, an attention recurrent neural network layer, and a decoding recurrent neural network layer. A post-processing network is used to improve the results of spectral reconstruction. The result of spectral reconstruction is the target sound spectrum feature output by the decoder.

The server can respectively input the current feature vector output by the decoded cyclic neural network layer to the frame prediction network and the linear projection layer to obtain three frames of mel spectrum and corresponding end identification information. The server may input at least one frame of the three frames of mel spectrum to the preprocessing layer for a next round of iteration until the end of the iteration. The server can finally input the predicted mel spectrum to the post-processing network, predict a residual term, and superimpose the residual term into the mel spectrum to obtain an improved mel spectrum.

It should be understood that although the steps in the flow charts involved in the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flow charts involved in the above-mentioned embodiments may include multiple steps or stages, and these steps or stages are not necessarily executed at the same time, but may be performed at different times For execution, the execution order of these steps or stages is not necessarily performed sequentially, but may be executed in turn or alternately with other steps or at least a part of steps or stages in other steps.

Based on the same inventive concept, an embodiment of the present application further provides a speech synthesis device for implementing the speech synthesis method involved above. The solution to the problem provided by the device is similar to the implementation described in the above method, so for the specific limitations in one or more embodiments of the speech synthesis device provided below, please refer to the definition of the speech synthesis method above, I won't repeat them here.

In one embodiment, as shown in FIG. 4 , a speech synthesis device 400 is provided, including: an encoding module 402 and a decoding module 404, wherein:

The coding module 402 is used to determine the coding segment of the sound spectrum to be predicted by coding the phoneme feature information corresponding to the Chinese text;

The decoding module 404, in each round of iterative prediction, determines the current sound spectrum feature corresponding to the encoded segment of the current round of prediction, and the attention weight used when predicting the current sound spectrum feature in the current round; when the attention weight reaches the preset weight threshold In this case, adjust the preset decoding times corresponding to the code segment based on the length of the current sound spectrum feature to obtain the adjusted preset decoding times; when the number of iterative prediction rounds reaches the preset decoding times, stop the iteration, and The current spectral feature is determined as the target spectral feature corresponding to the coding segment; the target spectral feature is used to synthesize the speech corresponding to the Chinese text.

In one embodiment, the encoding module 402 is also used to determine the phoneme feature information corresponding to the Chinese text; encode the phoneme feature information to obtain the text encoding sequence corresponding to the Chinese text; determine from the text encoding sequence according to the preset time step The coded segment to be subjected to spectrum prediction.

In one embodiment, the decoding module 404 is further configured to use the next round as the current round when the number of rounds of iterative prediction does not reach the preset number of decoding times, and return to determine the current sound spectrum corresponding to the encoded segment of the current round of prediction features, and the attention weights used to predict the current spectral feature in this round continue to execute.

In one embodiment, the decoding module 404 is also used to determine the current attention vector used in the current round of iterative prediction in each round of iterative prediction; based on the similarity between the current attention vector and the coding segment, The attention weight corresponding to the current attention vector is calculated; the attention weight is used to instruct to perform feature extraction on the attention vector to obtain the current feature vector; the current feature vector is used to predict the current sound spectrum feature.

In one embodiment, the decoding module 404 is further configured to determine the decoding round offset value corresponding to the encoded segment when the attention weight reaches the preset weight threshold; according to the decoding round offset value and the current sound spectrum feature length, adjust the preset decoding times corresponding to the coding segment, and obtain the adjusted preset decoding times.

In one embodiment, the decoding module 404 is also used to predict the end identification information corresponding to the coded segment based on the current sound spectrum feature; when the end identification information cannot meet the preset end condition, the number of rounds of iterative prediction and the preset The number of decoding times is compared, and the step of stopping the iteration is performed when the number of iteration prediction rounds matches the preset number of decoding times.

In one embodiment, the decoding module 404 is further configured to stop the iteration when the end identification information satisfies the preset end condition, and determine the current sound spectrum feature as the target sound spectrum feature corresponding to the encoding segment. Each module in the above-mentioned speech synthesis device can be fully or partially realized by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

In one embodiment, a computer device is provided, and the computer device may be a server, and its internal structure may be as shown in FIG. 5 . The computer device includes a processor, a memory, an input/output interface (Input/Output, I/O for short), and a communication interface. Wherein, the processor, the memory and the input/output interface are connected through the system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs and databases. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store the sound spectrum features predicted by each round of iterations. The input/output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech synthesis method is realized.

In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure may be as shown in FIG. 6 . The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit and an input device. Wherein, the processor, the memory and the input/output interface are connected through the system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input/output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a speech synthesis method is realized. The display unit of the computer equipment is used to form a visually visible picture, and may be a display screen, a projection device or a virtual reality imaging device, the display screen may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment may be a display screen The touch layer covered on the top can also be a button, a trackball or a touch pad arranged on the computer equipment casing, or an external keyboard, touch pad or mouse.

Those skilled in the art can understand that the structures shown in Fig. 5 and Fig. 6 are only block diagrams of partial structures related to the solution of this application, and do not constitute a limitation on the computer equipment on which the solution of this application is applied, specifically The computer device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the foregoing method embodiments when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented.

In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented.

It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all It is information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the computer programs can be stored in a non-volatile computer-readable memory In the medium, when the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any reference to storage, database or other media used in the various embodiments provided in the present application may include at least one of non-volatile and volatile storage. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive variable memory (ReRAM), magnetic variable memory (Magnetoresistive Random Access Memory, MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (Phase Change Memory, PCM), graphene memory, etc. The volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory. As an illustration and not a limitation, the RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM). The databases involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a blockchain-based distributed database, etc., but is not limited thereto. The processors involved in the various embodiments provided by this application can be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., and are not limited to this.

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above-mentioned embodiments only express several implementation modes of the present application, and the description thereof is relatively specific and detailed, but should not be construed as limiting the patent scope of the present application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be determined by the appended claims.

Claims (10)

1. a speech synthesis method, is characterized in that, described method comprises: Determine the coding segment of the sound spectrum to be predicted by encoding the phoneme feature information corresponding to the Chinese text; In each round of iterative prediction, determine the current sound spectrum feature corresponding to the coding segment of the current round of prediction, and the attention weight used when predicting the current sound spectrum feature in the current round; When the attention weight reaches the preset weight threshold, adjust the preset decoding times corresponding to the encoding segment based on the length of the current sound spectrum feature, to obtain the adjusted preset decoding times; When the number of rounds of iterative prediction reaches the preset number of times of decoding, the iteration is stopped, and the current spectral feature is determined as the target spectral feature corresponding to the encoding segment; the target spectral feature is used for synthesis The voice corresponding to the Chinese text. 2. The method according to claim 1, characterized in that, by encoding the phoneme feature information corresponding to the Chinese text, determining the coded segment of the sound spectrum to be predicted comprises: Determine the phoneme feature information corresponding to the Chinese text; Encoding the phoneme feature information to obtain a text encoding sequence corresponding to the Chinese text; According to the preset time step, the coding segment to be subjected to sound spectrum prediction is determined from the text coding sequence. 3. The method according to claim 1, characterized in that the method further comprises: In the case that the number of rounds of iterative prediction does not reach the preset number of decoding times, the next round is used as the current round, and the current sound spectrum feature corresponding to the coding segment of the current round of prediction and the current sound spectrum feature corresponding to the current round of prediction are returned. The step of using attention weights for spectral features continues. 4. method according to claim 1, is characterized in that, described method also comprises the step of calculating attention weight; The step of described calculation attention weight comprises: In each round of iterative prediction, determine the current attention vector used when predicting the current sound spectrum feature in this round; Based on the similarity between the current attention vector and the coding segment, calculate the attention weight corresponding to the current attention vector; the attention weight is used to indicate that the feature extraction is performed on the attention vector to obtain A current feature vector; the current feature vector is used to predict the current sound spectrum features. 5. The method according to claim 1, wherein when the attention weight reaches a preset weight threshold, the length corresponding to the coding segment is preset based on the length of the current sound spectrum feature. The decoding times are adjusted, and the adjusted preset decoding times include: When the attention weight reaches a preset weight threshold, determine a decoding round offset value corresponding to the coding segment; According to the decoding round offset value and the length of the current sound spectrum feature, the preset decoding times corresponding to the coded segment is adjusted to obtain the adjusted preset decoding times. 6. The method according to any one of claims 1 to 5, characterized in that the method further comprises: predicting end identification information corresponding to the coded segment based on the current sound spectrum feature; When the end identification information cannot satisfy the preset end condition, compare the number of rounds of iterative prediction with the preset number of decoding times, and perform In the case of decoding times, stop the iterative step. 7. The method according to claim 6, further comprising: When the end identification information satisfies the preset end condition, the iteration is stopped, and the current sound spectrum feature is determined as the target sound spectrum feature corresponding to the encoding segment. 8. A speech synthesis device, characterized in that the device comprises: The encoding module is used to determine the encoding segment of the sound spectrum to be predicted by encoding the phoneme feature information corresponding to the Chinese text; The decoding module, in each round of iterative prediction, determines the current sound spectrum feature corresponding to the encoded segment of the current round of prediction, and the attention weight used when predicting the current sound spectrum feature in the current round; when the attention weight reaches In the case of a preset weight threshold, adjust the preset decoding times corresponding to the coded segment based on the length of the current sound spectrum feature to obtain the adjusted preset decoding times; when the number of rounds of iterative prediction reaches the preset When the number of decoding times is set, the iteration is stopped, and the current spectral feature is determined as the target spectral feature corresponding to the coded segment; the target spectral feature is used to synthesize the speech corresponding to the Chinese text. 9. A computer device, comprising a memory and a processor, the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program . 10. A computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are realized.

Images