WO2022121169A1 - Emotional speech synthesis method, apparatus, and device, and storage medium - Google Patents

Abstract

An emotional speech synthesis method, apparatus, and device, and a storage medium, for use in solving the problems that a synthesized speech is flat and lacks emotions, and increasing the diversity of the synthesized speech. The emotional speech synthesis method comprises: obtaining speech data to be recognized and corresponding text data (101); inputting said speech data into a pretrained emotion recognition network to generate a Mel spectrum feature and a position code, and performing processing in the emotion recognition network in combination with the Mel spectrum feature and the position code to generate an emotion embedding feature (102); inputting the emotion embedding feature and the text data into a pre-trained speech synthesis network to generate target Mel spectrum data (103); and performing speech conversion on the target Mel spectrum data by using a neural vocoder to generate a target emotional speech (104). In addition, the method also relates to a blockchain technology, and said speech data and the text data can be stored in a blockchain.

Description

Emotional speech synthesis method, device, device and storage medium

This application claims the priority of the Chinese patent application with the application number 202011432589.4 and the invention titled "Method, Apparatus, Equipment and Storage Medium for Emotional Speech Synthesis" filed with the China Patent Office on December 10, 2020, the entire contents of which are by reference incorporated in the application.

technical field

The present application relates to the technical field of speech synthesis, and in particular, to a method, apparatus, device and storage medium for synthesizing emotional speech.

Background technique

With the development of technology, artificial intelligence services such as smart customer service centers, chat robots, and smart speakers have entered our daily lives and are playing an increasingly important role. This kind of artificial intelligence server usually involves speech synthesis technology, so speech synthesis technology has also been more widely used.

In the prior art, speech synthesis methods are mainly based on hidden Markov speech synthesis methods or neural network-based speech synthesis methods. The inventor realized that although these two speech synthesis methods can obtain good synthesized speech, the resulting Synthesized speech is flat and lacking emotion, making it impossible to get emotional speech.

SUMMARY OF THE INVENTION

The present application provides a method, device, device and storage medium for synthesizing emotional speech, which are used to solve the problem of dullness and lack of emotion in the synthesized speech, and increase the diversity of the synthesized speech.

A first aspect of the present application provides a method for synthesizing emotional speech, including: acquiring speech data to be recognized and corresponding text data; inputting the speech data to be recognized into a pre-trained emotion recognition network to generate Mel spectrum features and position encoding, and process in the emotion recognition network in combination with the Mel spectrum feature and the position encoding to generate an emotion embedding feature; input the emotion embedding feature and the text data into the pre-trained speech synthesis In the network, target mel-spectrum data is generated; a neural vocoder is used to perform speech conversion on the target mel-spectrum data to generate target emotional speech.

A second aspect of the present application provides a device for synthesizing emotional speech, comprising: an acquisition module for acquiring to-be-recognized speech data and corresponding text data; an embedded feature generation module for inputting the to-be-recognized speech data into pre-training In a good emotion recognition network, the mel spectrum feature and the position code are generated, and the mel spectrum feature and the position code are processed in the emotion recognition network to generate the emotion embedded feature; the mel spectrum data generation module , for inputting the emotion embedded feature and the text data into the pre-trained speech synthesis network to generate target mel-spectrum data; the speech conversion module is used for using a neural vocoder to analyze the target mel-spectrum data Perform speech conversion to generate target emotional speech.

A third aspect of the present application provides a device for synthesizing emotional speech, including: a memory and at least one processor, where instructions are stored in the memory; the at least one processor invokes the instructions in the memory, so that The device for synthesizing emotional speech performs the following method for synthesizing emotional speech:

Obtain the speech data to be recognized and the corresponding text data; input the speech data to be recognized into a pre-trained emotion recognition network, generate mel spectrum features and position coding, and combine the mel spectrum features and the position coding Perform processing in the emotion recognition network to generate emotion embedded features; input the emotion embedded features and the text data into a pre-trained speech synthesis network to generate target mel spectrum data; use a neural vocoder to The target mel spectrum data is used for speech conversion to generate the target emotional speech.

A fourth aspect of the present application provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, when the computer-readable storage medium runs on a computer, the computer executes the following method for synthesizing emotional speech:

Obtain the speech data to be recognized and the corresponding text data; input the speech data to be recognized into a pre-trained emotion recognition network, generate mel spectrum features and position coding, and combine the mel spectrum features and the position coding Perform processing in the emotion recognition network to generate emotion embedded features; input the emotion embedded features and the text data into a pre-trained speech synthesis network to generate target mel spectrum data; use a neural vocoder to The target mel spectrum data is used for speech conversion to generate the target emotional speech.

In the technical solution provided by this application, the speech data to be recognized and the corresponding text data are obtained; the speech data to be recognized is input into a pre-trained emotion recognition network to generate Mel spectrum features and location codes, and combined with the Mel spectrum features and the position encoding are processed in the emotion recognition network to generate emotion embedded features; input the emotion embedded features and the text data into a pre-trained speech synthesis network to generate target Mel spectrum data ; Use a neural vocoder to convert the target Mel spectrum data to generate target emotional speech. In the embodiment of the present application, through the pre-trained emotion recognition network, combined with the Mel spectrum feature and the position encoding, the emotion embedded feature is generated, and then the emotion embedded feature and the text data are spliced to generate the target emotional speech, which solves the problem of dull synthetic speech, The problem of lack of emotion increases the variety of synthesized speech.

Description of drawings

1 is a schematic diagram of an embodiment of a method for synthesizing emotional speech in an embodiment of the present application;

2 is a schematic diagram of another embodiment of a method for synthesizing emotional speech in an embodiment of the present application;

3 is a schematic diagram of an embodiment of an apparatus for synthesizing emotional speech in an embodiment of the present application;

4 is a schematic diagram of another embodiment of an apparatus for synthesizing emotional speech in an embodiment of the present application;

FIG. 5 is a schematic diagram of an embodiment of a device for synthesizing emotional speech in an embodiment of the present application.

Detailed ways

Embodiments of the present application provide a method, apparatus, device, and storage medium for synthesizing emotional speech, which are used to solve the problem that the synthesized speech is dull and lack emotion, and increase the diversity of the synthesized speech.

The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that data so used can be interchanged under appropriate circumstances so that the embodiments described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed steps or units, but may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

For ease of understanding, the specific flow of the embodiment of the present application is described below, referring to FIG. 1 , an embodiment of the method for synthesizing emotional speech in the embodiment of the present application includes:

101. Acquire speech data to be recognized and corresponding text data;

The server obtains the speech data to be recognized and the text data corresponding to the text data to be recognized. It should be emphasized that, in order to further ensure the privacy and security of the above-mentioned voice data and text data to be recognized, the above-mentioned voice data and text data to be recognized can also be stored in a node of a blockchain.

The to-be-recognized speech data is the to-be-recognized speech data with emotion, which may be the to-be-recognized speech data with happy emotion, the to-be-recognized speech data with surprised emotion, and/or the to-be-recognized speech data with anger emotion. When the server obtains the speech data to be recognized with emotion, it also obtains the corresponding text data. For example, the speech data to be recognized with emotion is "really! Congratulations!", the server is obtaining "really! Congratulations!" !", the text data of "Really! Congratulations!" is also obtained.

It can be understood that the execution subject of the present application may be a device for synthesizing emotional speech, and may also be a terminal or a server, which is not specifically limited here. The embodiments of the present application take the server as an execution subject as an example for description.

102, input the speech data to be recognized in the pre-trained emotion recognition network, generate mel spectrum feature and position coding, and process in emotion recognition network in conjunction with mel spectrum feature and position coding, generate emotion embedded feature;

The server inputs the speech data to be recognized into the pre-trained emotion recognition network, firstly generates the mel spectrum feature and the position code, and then processes the mel spectrum feature and the position code in the emotion recognition network to generate the emotion embedded feature .

The server inputs the speech data of "really! Congratulations!" into the pre-trained emotion recognition network for calculation, and generates Mel spectrum features [B ₁ , T ₁ , D ₁ ] and position code P, where, The location code P is generated based on the mel spectrum feature. The location code P is actually a hidden layer output. The server combines the mel spectrum features [B ₁ , T ₁ , D ₁ ] and the location code P for calculation to generate emotional embedding features. [B ₂ , T ₂ , D ₂ ].

103. Input the emotion embedded feature and text data into the pre-trained speech synthesis network to generate target mel spectrum data;

The server inputs the emotion embedded features and text data into the pre-trained speech synthesis network for calculation, and generates the target mel spectrum data.

The server inputs the emotional embedding features of [B ₂ , T ₂ , D ₂ ] and the text data of "Really! Congratulations!" into the pre-trained speech synthesis network for calculation. In this embodiment, the speech synthesis network It includes an encoder, in which the text data of "Really! Congratulations!" is feature-lifted, an extraction result is generated, and the extraction result is combined with the emotion embedding feature [B ₂ ,T ₂ ,D ₂ ] for splicing to generate target Mel spectrum data [B ₂ , T ₂ , D ₂ +D].

104. Use a neural vocoder to perform speech conversion on the target Mel spectrum data to generate a target emotional speech.

The server uses a neural vocoder to convert the target mel-spectral data into target emotional speech.

It should be noted that, in this embodiment, the neural vocoder is Wave Glow, the target Mel spectrum data is the input of the neural vocoder, the input frame length is 1024, and the frame shift is 256. The Er spectrum data is input into the affine coupling layer of the neural vocoder for scaling and transformation to generate emotional speech features, and then reversible convolution is performed on the emotional speech features to generate the target emotional speech "Really! (surprised emotion) Congratulations to you. ! (happy emotion)".

In the embodiment of the present application, through the pre-trained emotion recognition network, combined with the Mel spectrum feature and the position encoding, the emotion embedded feature is generated, and then the emotion embedded feature and the text data are spliced to generate the target emotional speech, which solves the problem of dull synthetic speech, The problem of lack of emotion increases the variety of synthesized speech.

Referring to FIG. 2, another embodiment of the method for synthesizing emotional speech in the embodiment of the present application includes:

201. Obtain emotional speech training data, emotional label data, and text training data;

The server obtains emotional speech training data, emotional label data and text training data from the big data platform or database.

It should be noted that the emotional speech training data can be divided into emotional speech training data including noise and emotional speech training data not including noise.

The emotional voice training data can be voice training data with emotion such as "too much", "really" or "too good", and obtain emotional label data and text training data, among which "too much" The emotional voice training data corresponds to the emotional label data of "anger" and the text training data of "too much"; the emotional voice training data of "really" corresponds to the emotional label data of "surprise" and corresponds to the text of "really" Training data; "too good" emotional speech training data corresponds to "happy" emotional label data, corresponding to "too good" text training data.

202. Use emotional speech training data and emotional label data, and combine layer regularization mechanism to perform model training to generate a pre-trained emotion recognition network, and use emotional speech training data and text training data for model training to generate pre-trained speech synthetic network;

The server performs training based on the emotional voice training data and emotional label data, combined with the regularization mechanism, to generate a pre-trained emotion recognition network, and then performs model training based on the emotional voice training data and text training data to generate a pre-trained speech synthesis network.

It should be noted that the pre-trained emotion recognition network is used to extract emotional features, so emotional speech training data and emotion label data are used to train the emotion recognition network; the pre-trained speech synthesis network is used to synthesize emotional speech, so emotional speech is used. Speech training data and text training data train the speech synthesis network. Although the above training process includes emotional speech training data, when training the emotion recognition network, the emotional speech training data can be either the emotional speech training data including noise or the emotional speech training data that does not include noise, but training speech synthesis When the network is used, it is necessary to call high-quality training data for training, that is, emotional speech training data that does not include noise.

In the process of training the emotion recognition network, the server combines the layer regularization mechanism to train the emotion recognition network, mainly adding a layer regularization mechanism after each sub-layer, and the layer regularization mechanism is to calculate the output of the i layer in the channel dimension. and variance, and then subtract the mean from the output of layer i and divide by the variance, so that the output of layer i has a mean of 0 and a variance of 1. The layer regularization mechanism can make the distribution of training data consistent and make the training process stable.

203. Obtain speech data to be recognized and corresponding text data;

The server obtains the speech data to be recognized and the text data corresponding to the text data to be recognized. It should be emphasized that, in order to further ensure the privacy and security of the above-mentioned voice data and text data to be recognized, the above-mentioned voice data and text data to be recognized can also be stored in a node of a blockchain.

The to-be-recognized speech data is the to-be-recognized speech data with emotion, which may be the to-be-recognized speech data with happy emotion, the to-be-recognized speech data with surprised emotion, and/or the to-be-recognized speech data with anger emotion. When the server obtains the speech data to be recognized with emotion, it also obtains the corresponding text data. For example, the speech data to be recognized with emotion is "really! Congratulations!", the server is obtaining "really! Congratulations!" !", the text data of "Really! Congratulations!" is also obtained.

204. Input the speech data to be recognized into a pre-trained emotion recognition network, generate mel spectrum features and positional codes, and process them in the emotion recognition network in combination with the mel spectrum features and positional codes to generate emotion embedded features;

The server inputs the speech data to be recognized into the pre-trained emotion recognition network, firstly generates the mel spectrum feature and the position code, and then processes the mel spectrum feature and the position code in the emotion recognition network to generate the emotion embedded feature .

The server inputs the speech data of "really! Congratulations!" into the pre-trained emotion recognition network for calculation, and generates Mel spectrum features [B ₁ , T ₁ , D ₁ ] and position code P, where, The location code P is generated based on the mel spectrum feature. The location code P is actually a hidden layer output. The server combines the mel spectrum features [B ₁ , T ₁ , D ₁ ] and the location code P for calculation to generate emotional embedding features. [B ₂ , T ₂ , D ₂ ].

Specifically, the server inputs the speech data to be recognized into a pre-trained emotion recognition network to generate mel spectrum features; the server generates a position code according to the mel spectrum features and a preset position conversion formula; the server converts the mel spectrum features and The positional encoding is input into the encoder of the emotion recognition network for encoding to generate emotion embedding features.

The server inputs the speech data of "really! Congratulations!" into the pre-trained emotion recognition network, and firstly inputs "really! Congratulations!" into the trained emotion recognition network for feature extraction. Generate mel spectrum features [B ₁ , T ₁ , D ₁ ]; then the server calculates the position encoding of the Mel spectrum features of [B ₁ , T ₁ , D ₁ ] according to the preset position conversion formula, and generates the position code P; Finally, the server inputs the Mel spectrum features [B ₁ , T ₁ , D ₁ ] and the position encoding P into the encoder of the emotion recognition network for encoding to generate emotion embedding features, where the encoder of the emotion recognition network includes five identical module and a layer of long short-term memory artificial neural network layer, in which each module includes two sub-layers, namely multi-head self-attention layer and forward propagation layer, in the encoder [B ₁ , T ₁ , D ₁ ] and P are encoded to generate sentiment embedding features [B ₂ , T ₂ , D ₂ ].

The server inputs the speech data to be recognized into the pre-trained emotion recognition network, and the generated Mel spectrum features include:

First, the server performs windowing processing on the speech data to be recognized to generate windowed speech data; then, the server performs short-time Fourier transform on the windowed speech data to generate Fourier transformed speech data; finally, The server uses a Mel filter bank to process the Fourier transformed speech data to generate Mel spectrum features.

The server uses a window function to perform windowing processing on the speech data to be recognized for "really! Congratulations!" to generate windowed speech data; then the server performs Fourier transform on the windowed speech data, and determines to add The frequency and phase of the speech data after the window are obtained, thereby generating the speech data after Fourier transformation; finally, the server uses a Mel filter bank to process the speech data after Fourier transformation into mel spectrum features.

According to the Mel spectrum feature and the preset position conversion formula, the server generates the position code including:

The server reads the length of the mel spectrum feature and reads the position of the mel spectrum feature; the service generates a position input value based on the length of the mel spectrum feature and the position of the mel spectrum feature; the server inputs the position input vector into the preset Position conversion formula to generate position codes.

In this embodiment, the preset position conversion formula is

Among them, pos is the position of the mel spectral feature, 2i represents the even dimension, 2i+1 represents the odd dimension, and d _mode represents the preset dimension vector corresponding to the location of the mel spectral feature, such as 256.

For example, the server reads the length of the mel spectral feature as 5 and the position of the mel spectral feature as 0, then the server determines the position based on the length of the mel spectral feature of "5" and the position of the mel spectral feature of "0" The input value is [0,1,2,3,4], and then the position input value [0,1,2,3,4] is calculated based on the above formula to generate the position code P. It should be noted that, in this embodiment, P is only a pronoun, and does not represent specific position encoding data.

The server inputs the mel spectrum feature and position encoding into the encoder of the emotion recognition network for encoding, and the generated emotion embedding features include:

The server inputs the position encoding of the mel spectrum feature sum into the multi-head self-attention layer of the emotion recognition network, and combines the residual connections to generate the initial emotion feature vector; the server inputs the initial emotion feature vector into the forward propagation layer of the emotion recognition network for processing. Convolution to generate sentiment embedding features.

The server first inputs the Mel spectrum features [B ₁ , T ₁ , D ₁ ] into the multi-head self-attention layer for calculation, and combines the residual connections to generate the initial emotion feature vector. The formula for the multi-head self-attention layer design is as follows:

head _i =Attention(QW _i ^Q ,KW _i ^K ,KW _i ^V )

c _t =MultiHead(Q,K,V)=Concat(head ₁ ,...,head _h )W ^O ;

Among them, Q, K, V are the input, that is, the mel spectrum feature, d _k is the preset dimension vector, such as 256, head _i is the ith head, and each calculation in the multi-head self-attention layer is a head, if Perform 4 Attention (QW _i ^Q , KW _i ^K , KV ^V ) calculations, then generate 4 heads, W _i ^Q , KW _i ^K , KW _i ^V are the weights, which are generated during the training process, and Concat refers to how many The heads are spliced together along the last dimension. For example, the dimension vectors of these four heads are [B _t , T _t , D _t ], and the server splices them together to generate the initial emotion feature vector [B _t , T _{t ]} , 4D _t ], W ^O is the parameter learned in advance. After generating the initial emotion feature vector, the initial emotion feature vector and the Mel spectrum feature input are convolved in the corresponding forward propagation layer to generate the first module emotion feature vector. Since the encoder includes five identical modules, Therefore, five calculations are performed according to the above calculation method, and the output result of the last module is input into a layer of long short-term memory artificial neural network layer, thereby generating emotional embedded features [B ₂ , T ₂ , D ₂ ].

It should be noted that the residual connection is to add the input of each multi-head self-attention layer to the output as the input of the next forward propagation layer. In the first multi-head self-attention layer, it is the initial emotion. Mel spectrum features are added to the feature vector to generate the input of the forward propagation layer. Thereby improving the relevance of generating emotional embedding features.

205. Input the emotion embedded feature and text data into the pre-trained speech synthesis network to generate target mel spectrum data;

The server inputs the emotion embedded features and text data into the pre-trained speech synthesis network for calculation, and generates the target mel spectrum data.

The server inputs the emotion embedding features of [B ₂ , T ₂ , D ₃ ] and the text data of "Really! Congratulations!" into the pre-trained speech synthesis network for calculation. In this embodiment, the speech synthesis network It includes an encoder, in which the text data of "Really! Congratulations!" is feature-lifted, an extraction result is generated, and the extraction result is combined with the emotion embedding feature [B ₂ ,T ₂ ,D ₂ ] for splicing to generate target Mel spectrum data [B ₂ , T ₂ , D ₂ +D].

Specifically, the server converts text data into text embedding features in the pre-trained speech synthesis network; the server splices the text embedding features and emotion embedding features in order of time to generate target Mel spectrum data.

In the pre-trained speech synthesis network, the server first converts the text data into text embedding features in the same form as the emotional embedding features in the order of time, and then the server splices the text embedding features and emotional embedding features in the order of time to generate the target Mei spectral data. _In this embodiment, for example, the emotion embedding feature is [B ₂ , T ₂ , D ₂ ], and the text embedding feature is [B ₂ , T ₂ , D ₃ ], and the server _will D ₂ ] is spliced with [B ₂ , T ₂ , D] to generate target mel spectrum data as [B ₂ , T ₂ , D ₂ +D]. In other embodiments, if the emotion embedding feature dimension is [B ₂ , D ₂ ], the server expands the emotion embedding feature to [B ₂ , 1, D ₂ ], and then the emotion embedding feature and cost embedding feature Do stitching.

206. Use a neural vocoder to perform speech conversion on the target Mel spectrum data to generate a target emotional speech.

The server uses a neural vocoder to convert the target mel-spectral data into target emotional speech.

It should be noted that, in this embodiment, the neural vocoder is Wave Glow, the target Mel spectrum data is the input of the neural vocoder, the input frame length is 1024, and the frame shift is 256. The Er spectrum data is input into the affine coupling layer of the neural vocoder for scaling and transformation to generate emotional speech features, and then reversible convolution is performed on the emotional speech features to generate the target emotional speech "Really! (surprised emotion) Congratulations to you. ! (happy emotion)".

In the embodiment of the present application, through the pre-trained emotion recognition network, combined with the Mel spectrum feature and the position encoding, the emotion embedded feature is generated, and then the emotion embedded feature and the text data are spliced to generate the target emotional speech, which solves the problem of dull synthetic speech, The problem of lack of emotion increases the variety of synthesized speech.

The method for synthesizing emotional speech in the embodiment of the present application has been described above, and the device for synthesizing emotional speech in the embodiment of the present application is described below. Please refer to FIG. 3 . An embodiment of the device for synthesizing emotional speech in the embodiment of the present application includes:

A to-be-recognized data acquisition module 301, configured to acquire to-be-recognized voice data and corresponding text data;

The embedded feature generation module 302 is used to input the speech data to be recognized into the pre-trained emotion recognition network, generate a mel spectrum feature and a position code, and combine the mel spectrum feature and the position code in the described mel spectrum feature and the position code. Processed in the emotion recognition network to generate emotion embedded features;

Mel spectrum data generation module 303, for inputting the emotion embedded feature and the text data into a pre-trained speech synthesis network to generate target Mel spectrum data;

The speech conversion module 304 is configured to perform speech conversion on the target mel spectrum data by using a neural vocoder to generate a target emotional speech.

In the embodiment of the present application, through the pre-trained emotion recognition network, combined with the Mel spectrum feature and the position encoding, the emotion embedded feature is generated, and then the emotion embedded feature and the text data are spliced to generate the target emotional speech, which solves the problem of dull synthetic speech, The problem of lack of emotion increases the variety of synthesized speech.

Referring to FIG. 4 , another embodiment of the apparatus for synthesizing emotional speech in the embodiment of the present application includes:

A to-be-recognized data acquisition module 301, configured to acquire to-be-recognized voice data and corresponding text data;

The embedded feature generation module 302 is used to input the speech data to be recognized into the pre-trained emotion recognition network, generate a mel spectrum feature and a position code, and combine the mel spectrum feature and the position code in the described mel spectrum feature and the position code. Processed in the emotion recognition network to generate emotion embedded features;

Mel spectrum data generation module 303, for inputting the emotion embedded feature and the text data into a pre-trained speech synthesis network to generate target Mel spectrum data;

The speech conversion module 304 is configured to perform speech conversion on the target mel spectrum data by using a neural vocoder to generate a target emotional speech.

Optionally, the embedded feature generation module 302 includes:

Mel spectrum feature generation unit 3021, for inputting the speech data to be recognized into the pre-trained emotion recognition network to generate Mel spectrum features;

a position code generation unit 3022, configured to generate a position code according to the mel spectrum feature and a preset position conversion formula;

The encoding unit 3023 is configured to input the mel spectrum feature and the position code into the encoder of the emotion recognition network for encoding, and generate an emotion embedded feature.

Optionally, the mel spectrum feature generating unit 3021 can also be specifically used for:

Windowing is performed on the to-be-recognized speech data to generate windowed speech data;

Carry out short-time Fourier transform to the voice data after the windowing, and generate the voice data after the Fourier transform;

The Fourier-transformed speech data is processed by using a Mel filter bank to generate Mel spectrum features.

Optionally, the location code generation unit 3022 can also be specifically used for:

Read the length of the mel spectrum feature, and read the position of the mel spectrum feature;

generating a position input value based on the length of the mel spectral feature and the position of the mel spectral feature;

Input the position input vector into a preset position conversion formula to generate a position code.

Optionally, the encoding unit 3023 can also be specifically used for:

Inputting the Mel spectrum feature and the position encoding into the multi-head self-attention layer of the emotion recognition network, and combining the residual connections to generate an initial emotion feature vector;

The initial emotion feature vector is input into the forward propagation layer of the emotion recognition network for convolution to generate emotion embedded features.

Optionally, the Mel spectrum data generation module 303 can also be specifically used for:

In a pre-trained speech synthesis network, convert the text data into text embedding features;

According to the time sequence, the text embedding feature and the emotion embedding feature are spliced to generate target mel spectrum data.

Optionally, the device for synthesizing emotional speech further includes:

A training data acquisition module 305, configured to acquire emotional speech training data, emotional label data and text training data;

The training module 306 is configured to use the emotional voice training data and the emotional label data, and perform model training in combination with a layer regularization mechanism, generate a pre-trained emotional recognition network, and use the emotional voice training data and the text The training data is used for model training to generate a pre-trained speech synthesis network.

In the embodiment of the present application, through the pre-trained emotion recognition network, combined with the Mel spectrum feature and the position encoding, the emotion embedded feature is generated, and then the emotion embedded feature and the text data are spliced to generate the target emotional speech, which solves the problem of dull synthetic speech, The problem of lack of emotion increases the variety of synthesized speech.

3 and 4 above describe the device for synthesizing emotional speech in the embodiment of the present application in detail from the perspective of modular functional entities, and the following describes the device for synthesizing emotional speech in the embodiment of the present application in detail from the perspective of hardware processing.

5 is a schematic structural diagram of a device for synthesizing emotional speech provided by an embodiment of the present application. The device 500 for synthesizing emotional speech may vary greatly due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (eg, one or more processors) and memory 520, one or more storage media 530 (eg, one or more mass storage devices) that store application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 may be short-term storage or persistent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the device 500 for synthesizing emotional speech. Furthermore, the processor 510 may be configured to communicate with the storage medium 530, and execute a series of instruction operations in the storage medium 530 on the emotional speech synthesis device 500.

The emotional speech synthesis device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and/or, one or more operating systems 531, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, and more. Those skilled in the art can understand that the structure of the device for synthesizing emotional speech shown in FIG. 5 does not constitute a limitation on the device for synthesizing emotional speech, and may include more or less components than those shown in the figure, or combine some components, or Different component arrangements.

The present application also provides a device for synthesizing emotional speech. The computer device includes a memory and a processor, and computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by the processor, the processor executes the steps in the foregoing embodiments. The steps of the emotion speech synthesis method.

The present application also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium may also be a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to execute the steps of the method for synthesizing emotional speech.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above may refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

The blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes .

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: The technical solutions described in the embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims (22)

A method for synthesizing emotional speech, wherein the method for synthesizing emotional speech comprises: Obtain the speech data to be recognized and the corresponding text data; Input the speech data to be recognized into the pre-trained emotion recognition network, generate mel spectrum features and position codes, and process them in the emotion recognition network in combination with the mel spectrum features and the position codes, and generate emotion embedded features; Inputting the emotion embedded feature and the text data into a pre-trained speech synthesis network to generate target mel spectrum data; A neural vocoder is used to perform speech conversion on the target Mel spectrum data to generate target emotional speech. The method for synthesizing emotional speech according to claim 1, wherein said inputting said to-be-recognized speech data into a pre-trained emotion recognition network, generating mel spectrum features and position codes, and combining said mel spectrum The feature and the location encoding are processed in the emotion recognition network, and generating emotion embedding features includes: Inputting the speech data to be recognized into a pre-trained emotion recognition network to generate Mel spectrum features; generating a position code according to the mel spectrum feature and a preset position conversion formula; The mel spectrum feature and the position code are input into the encoder of the emotion recognition network for encoding to generate emotion embedded features. The method for synthesizing emotional speech according to claim 2, wherein the inputting the speech data to be recognized into a pre-trained emotion recognition network, and generating a Mel spectrum feature comprises: Windowing is performed on the to-be-recognized speech data to generate windowed speech data; Carry out short-time Fourier transform to the voice data after the windowing, and generate the voice data after the Fourier transform; The Fourier-transformed speech data is processed by using a Mel filter bank to generate Mel spectrum features. The method for synthesizing emotional speech according to claim 2, wherein the generating a position code according to the Mel spectrum feature and a preset position conversion formula comprises: Read the length of the mel spectrum feature, and read the position of the mel spectrum feature; generating a position input value based on the length of the mel spectral feature and the position of the mel spectral feature; Input the position input vector into a preset position conversion formula to generate a position code. The method for synthesizing emotional speech according to claim 2, wherein the inputting the Mel spectrum feature and the position code into an encoder of the emotion recognition network for encoding, and generating the emotion embedded feature comprises: Inputting the Mel spectrum feature and the position encoding into the multi-head self-attention layer of the emotion recognition network, and combining the residual connections to generate an initial emotion feature vector; The initial emotion feature vector is input into the forward propagation layer of the emotion recognition network for convolution to generate emotion embedded features. The method for synthesizing emotional speech according to claim 1, wherein the inputting the emotional embedding feature and the text data into a pre-trained speech synthesis network, and generating target mel spectrum data comprises: In a pre-trained speech synthesis network, convert the text data into text embedding features; According to the time sequence, the text embedding feature and the emotion embedding feature are spliced to generate target mel spectrum data. The method for synthesizing emotional speech according to any one of claims 1-6, wherein, before acquiring the speech data to be recognized and the corresponding text data, the method for synthesizing emotional speech comprises: Obtain emotional speech training data, emotional label data and text training data; Using the emotional voice training data and the emotional label data, combined with the layer regularization mechanism, model training is performed to generate a pre-trained emotion recognition network, and the emotional voice training data and the text training data are used for model training, Generate a pretrained speech synthesis network. A device for synthesizing emotional speech, wherein the device for synthesizing emotional speech comprises: an acquisition module for acquiring the speech data to be recognized and the corresponding text data; The embedded feature generation module is used to input the speech data to be recognized into the pre-trained emotion recognition network, generate mel spectrum features and position codes, and combine the mel spectrum features and the position codes in the emotion recognition network. Process in the recognition network to generate emotional embedded features; Mel spectrum data generation module, for inputting the emotion embedded feature and the text data into the pre-trained speech synthesis network to generate target Mel spectrum data; The speech conversion module is used to perform speech conversion on the target mel spectrum data by using a neural vocoder to generate target emotional speech. A device for synthesizing emotional speech, wherein the device for synthesizing emotional speech comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor invokes the instructions in the memory, so that the device for synthesizing emotional speech executes the following method for synthesizing emotional speech: Obtain the speech data to be recognized and the corresponding text data; Input the speech data to be recognized into the pre-trained emotion recognition network, generate mel spectrum features and position codes, and process them in the emotion recognition network in combination with the mel spectrum features and the position codes, and generate emotion embedded features; Inputting the emotion embedded feature and the text data into a pre-trained speech synthesis network to generate target mel spectrum data; A neural vocoder is used to perform speech conversion on the target Mel spectrum data to generate target emotional speech. The device for synthesizing emotional speech according to claim 9, wherein the device for synthesizing emotional speech is executed by the processor by inputting the to-be-recognized speech data into a pre-trained emotion recognition network to generate Mel spectral features and positional encoding, and combined with the mel spectral features and the positional encoding for processing in the emotion recognition network, the steps of generating emotional embedded features include: Inputting the speech data to be recognized into a pre-trained emotion recognition network to generate Mel spectrum features; generating a position code according to the mel spectrum feature and a preset position conversion formula; The mel spectrum feature and the position code are input into the encoder of the emotion recognition network for encoding to generate emotion embedded features. The device for synthesizing emotional speech according to claim 10, wherein the device for synthesizing emotional speech is executed by the processor by inputting the to-be-recognized speech data into a pre-trained emotion recognition network to generate Mel The steps for spectral features include: Windowing is performed on the to-be-recognized speech data to generate windowed speech data; Carry out short-time Fourier transform to the voice data after the windowing, and generate the voice data after the Fourier transform; The Fourier-transformed speech data is processed by using a Mel filter bank to generate Mel spectrum features. The device for synthesizing emotional speech according to claim 10, wherein the device for synthesizing emotional speech is executed by the processor of the step of generating a position code according to the Mel spectrum feature and a preset position conversion formula , including: Read the length of the mel spectrum feature, and read the position of the mel spectrum feature; generating a position input value based on the length of the mel spectral feature and the position of the mel spectral feature; Input the position input vector into a preset position conversion formula to generate a position code. The device for synthesizing emotional speech according to claim 10, wherein the device for synthesizing emotional speech is executed by the processor to input the encoding of the mel spectrum feature and the positional code into the emotion recognition network The steps of generating emotion embedded features include: Inputting the Mel spectrum feature and the position encoding into the multi-head self-attention layer of the emotion recognition network, and combining the residual connections to generate an initial emotion feature vector; The initial emotion feature vector is input into the forward propagation layer of the emotion recognition network for convolution to generate emotion embedded features. The device for synthesizing emotional speech according to claim 9, wherein the device for synthesizing emotional speech is executed by the processor to input the emotion embedding feature and the text data into a pre-trained speech synthesis network , the steps to generate target mel spectrum data include: In a pre-trained speech synthesis network, convert the text data into text embedding features; According to the time sequence, the text embedding feature and the emotion embedding feature are spliced to generate target mel spectrum data. The device for synthesizing emotional speech according to any one of claims 9 to 14, wherein before the device for synthesizing emotional speech is executed by the processor the step of acquiring the speech data to be recognized and the corresponding text data ,include: Obtain emotional speech training data, emotional label data and text training data; Using the emotional voice training data and the emotional label data, combined with the layer regularization mechanism, model training is performed to generate a pre-trained emotion recognition network, and the emotional voice training data and the text training data are used for model training, Generate a pretrained speech synthesis network. A computer-readable storage medium storing instructions on the computer-readable storage medium, wherein, when the instructions are executed by a processor, the following method for synthesizing emotional speech is implemented: Obtain the speech data to be recognized and the corresponding text data; Input the speech data to be recognized into the pre-trained emotion recognition network, generate mel spectrum features and position codes, and process them in the emotion recognition network in combination with the mel spectrum features and the position codes, and generate emotion embedded features; Inputting the emotion embedded feature and the text data into a pre-trained speech synthesis network to generate target mel spectrum data; A neural vocoder is used to perform speech conversion on the target Mel spectrum data to generate target emotional speech. The computer-readable storage medium according to claim 16, wherein the synthesizing instruction of the emotional speech is executed by the processor, and the to-be-recognized speech data is input into a pre-trained emotion recognition network to generate a Mel spectral features and positional encoding, and combined with the mel spectral features and the positional encoding for processing in the emotion recognition network, the steps of generating emotional embedded features include: Inputting the speech data to be recognized into a pre-trained emotion recognition network to generate Mel spectrum features; generating a position code according to the mel spectrum feature and a preset position conversion formula; The mel spectrum feature and the position code are input into the encoder of the emotion recognition network for encoding to generate emotion embedded features. The computer-readable storage medium according to claim 17, wherein the synthesizing instruction of the emotional speech is executed by the processor and the inputting the to-be-recognized speech data into a pre-trained emotion recognition network to generate a Mel The steps for spectral features include: Windowing is performed on the to-be-recognized speech data to generate windowed speech data; Carry out short-time Fourier transform to the voice data after the windowing, and generate the voice data after the Fourier transform; The Fourier-transformed speech data is processed by using a Mel filter bank to generate Mel spectrum features. The computer-readable storage medium according to claim 17, wherein the step of generating a position code according to the mel spectrum feature and a preset position conversion formula is executed by the processor for the synthetic instruction of the emotional speech , including: Read the length of the mel spectrum feature, and read the position of the mel spectrum feature; generating a position input value based on the length of the mel spectral feature and the position of the mel spectral feature; Input the position input vector into a preset position conversion formula to generate a position code. 18. The computer-readable storage medium of claim 17, wherein the synthesizing instructions of the emotional speech are executed by the processor and the encoding to input the mel spectral features and the positional encoding into the emotion recognition network The steps of generating emotion embedded features include: Inputting the Mel spectrum feature and the position encoding into the multi-head self-attention layer of the emotion recognition network, and combining the residual connections to generate an initial emotion feature vector; The initial emotion feature vector is input into the forward propagation layer of the emotion recognition network for convolution to generate emotion embedded features. 17. The computer-readable storage medium of claim 16, wherein the emotional speech synthesis instructions are executed by the processor and the inputting the emotion embedding feature and the textual data into a pre-trained speech synthesis network , the steps to generate target mel spectrum data include: In a pre-trained speech synthesis network, convert the text data into text embedding features; According to the time sequence, the text embedding feature and the emotion embedding feature are spliced to generate target mel spectrum data. The computer-readable storage medium according to any one of claims 16-21, wherein before the step of obtaining the speech data to be recognized and the corresponding text data is performed by the processor before the synthesizing instruction of the emotional speech ,include: Obtain emotional speech training data, emotional label data and text training data; Using the emotional voice training data and the emotional label data, combined with the layer regularization mechanism, model training is performed to generate a pre-trained emotion recognition network, and the emotional voice training data and the text training data are used for model training, Generate a pretrained speech synthesis network.

Images