WO2024127472A1 - Emotion recognition learning method, emotion recognition method, emotion recognition learning device, emotion recognition device, and program - Google Patents

Abstract

The present invention involves: learning data that, for a plurality of speakers, includes a plurality of ways of input utterances by the speakers, a plurality of ways of ordinary-state emotion utterances corresponding to the input utterances, and correct-answer labels for emotion corresponding to the input utterances; and a model that receives inputs of an input utterance and a corresponding ordinary-state emotion utterance to output an emotion recognition result. The invention causes a computer to execute a learning procedure for training the model on the basis of a first loss function for minimizing a difference (error) between a correct-answer label corresponding to the input utterance input to the model and the recognition result output by the model on the basis of the learning data, and a second loss function for achieving, for every speaker, a constant vector representing the quality of emotional expression calculated for the ordinary-state emotion utterance in the course of the model outputting the recognition result. In this way, the invention contributes to an increase in the accuracy of recognizing the emotion of a speaker from an utterance.

Description

Emotion recognition learning method, emotion recognition method, emotion recognition learning device, emotion recognition device, and program

The present invention relates to an emotion recognition learning method, an emotion recognition method, an emotion recognition learning device, an emotion recognition device, and a program.

　Recognizing a speaker's emotions from speech is an important technology. For example, by recognizing a speaker's emotions during counseling, it is possible to visualize the patient's feelings of anxiety or sadness, which is expected to deepen the counselor's understanding and improve the quality of guidance. Furthermore, by recognizing human emotions in human-machine dialogue, it is possible to build a more friendly dialogue system that can share in the human's happiness if the human is happy, or encourage the human if he or she is sad. Hereafter, the technology that takes an utterance as input and estimates which emotion class the speaker's emotion contained in the utterance belongs to (e.g., neutral, anger, joy, sadness, etc.) will be referred to as "emotion recognition".

Patent Document 1 and Non-Patent Document 1 propose a technology (hereinafter referred to as "conventional technology") for improving the accuracy of emotion recognition by using speech when speaking with "normal" emotions (emotions that are neither positive emotions such as joy nor negative emotions such as anger or sadness, but are normal emotions in a normal state) (hereinafter referred to as "normal emotion speech"). The conventional technology is based on the hypothesis that "if you know how a person normally speaks (= normal emotion speech), the accuracy of emotion recognition for that person will improve."

Figure 1 is a diagram for explaining the conventional technology. In the conventional technology, in addition to the input utterance to be recognized, normal emotional utterance of the same speaker as the input utterance is required for estimation, and emotion recognition results are obtained by inputting these into an emotion recognition model. Inside the emotion recognition model, an "emotion expression vector extraction block" is first used for each of the input utterance and the normal emotional utterance to extract an emotion expression vector that represents the nature of the emotional expression of the entire utterance. Then, based on the emotion expression vectors of the input utterance and the normal emotional utterance, an "emotion estimation block for input utterance using the emotion expression vector of normal emotional utterance" is used to estimate the emotion of the input utterance. A statistical model based on deep learning is used for the emotion recognition model, and each block in the emotion recognition model is simultaneously trained using a set of labeled data consisting of the input utterance, normal emotional utterance, and the correct emotion label of the input utterance (the correct value for emotion recognition of the input utterance) prior to estimation.

International Publication No. 2021/171552

Andreas Triantafyllopoulos, Shuo Liu, Bjoern W. Schuller, "DEEP SPEAKER CONDITIONING FOR SPEECH EMOTION RECOGNITION", Proc. of ICME, pp.1-6, 2021

In conventional technologies, there is a problem that the recognition result changes when different normal emotion utterances are used during recognition (for example, when input utterance X and normal emotion utterance A are used, the recognition result of input utterance X may be "joy", and when X and normal emotion utterance B are used, the recognition result of input utterance X may be "sadness".). This is because when training the emotion recognition model, a set of input utterance, normal emotion utterance, and correct emotion label for the input utterance is used, so each block in the emotion recognition model is optimized to output recognition results specialized for combinations of input utterance and normal emotion utterance. In conventional technologies, a method is used in which combinations with various normal emotion utterances are included in the training data to prevent the problem of over-specialization in combinations, but this method does not explicitly reduce specialization in combinations and does not adequately address the problem.

The present invention was made in consideration of the above points, and aims to contribute to improving the accuracy of recognizing a speaker's emotions from speech.

In order to solve the above problem, a computer executes a learning procedure for learning the model based on training data including, for multiple speakers, multiple input utterances by a speaker, multiple normal emotion utterances corresponding to each of the input utterances, and correct emotion labels corresponding to each of the input utterances, and based on a first loss function for minimizing an error in the emotion recognition result output by a model to which any of the input utterances and the normal emotion utterance corresponding to that input utterance are input, relative to the correct emotion label corresponding to that input utterance, and a second loss function for making constant for each speaker a vector representing the nature of the emotional expression calculated for the normal emotion utterance in the process in which the model outputs the recognition result.

This can contribute to improving the accuracy of recognizing a speaker's emotions from speech.

FIG. 1 is a diagram for explaining a conventional technique. FIG. 1 is a diagram illustrating an example of a hardware configuration of an emotion recognition device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a functional configuration during learning of the emotion recognition device 10 according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a configuration of an emotion recognition model m1 according to an embodiment of the present invention. FIG. 2 is a diagram for explaining an outline of a learning method for an emotion recognition model m1. FIG. 2 is a diagram illustrating an example of a functional configuration of the emotion recognition device 10 according to the embodiment of the present invention when recognizing emotions. 10 is a flowchart illustrating an example of a process performed by the emotion recognition device 10 during learning. 11 is a diagram for explaining calculation of an average value of emotion expression vectors of normal emotion utterances for each speaker. FIG. FIG. 13 is a diagram for explaining calculation of a distance S _ji,k between each emotional expression vector e _ji and each speaker average c _k .

In this embodiment, when training an emotion recognition model using normal emotion utterances (utterances expressing emotions in a normal state that are neither positive emotions such as joy, nor negative emotions such as anger or sadness), in addition to the loss function used in conventional methods for minimizing the error between the correct emotion label and the emotion recognition result (error in the emotion recognition result for the correct emotion label), a new loss function (speaker identity loss function for normal emotion utterances) is introduced to ensure that the emotion expression vector of normal emotion utterances shows the same vector value if the speaker is the same (in other words, even if the normal emotion utterance changes, it shows a constant vector value if the speaker is the same). This optimizes the emotion expression vector extraction block so that a constant emotion expression vector of normal emotion utterance is obtained for different normal emotion utterances, solving the problem of specialization to the combination of input utterance and normal emotion utterance.

The following describes an embodiment of the present invention with reference to the drawings.

FIG. 2 is a diagram showing an example of the hardware configuration of an emotion recognition device 10 according to an embodiment of the present invention. The emotion recognition device 10 in FIG. 2 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are interconnected via a bus B.

The program that realizes the processing in the emotion recognition device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 via the drive device 100 into the auxiliary storage device 102. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

When an instruction to start a program is received, the memory device 103 reads out and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the emotion recognition device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

FIG. 3 is a diagram showing an example of the functional configuration of the emotion recognition device 10 during learning in an embodiment of the present invention. As shown in FIG. 3, the emotion recognition device 10 (emotion recognition learning device) during learning has two acoustic feature extraction units 11 (acoustic feature extraction unit 11-1, acoustic feature extraction unit 11-2) and one learning unit 12. These units are realized by processing in which one or more programs installed in the emotion recognition device 10 are executed by the processor 104. The emotion recognition device 10 also uses a learning data storage unit 121. The learning data storage unit 121 can be realized, for example, using the auxiliary storage device 102, or a storage device that can be connected to the emotion recognition device 10 via a network.

[Learning Data Storage Unit 121]
The learning data storage unit 121 stores a large amount of learning data in advance, which is a set of an input utterance, a correct emotion label (correct emotion label) of the input utterance, a normal emotion utterance of the same speaker as the input utterance, and a speaker label that is identification information of the speaker of the normal emotion utterance. The input utterance and the normal emotion utterance of each learning data are different utterances. The correct emotion label refers to a correct value of emotion recognition of the input utterance. Emotion recognition refers to estimating, using a certain utterance as input, which emotion class (e.g., normal, anger, joy, sadness, etc.) the speaker's emotion contained in the utterance belongs to. The normal emotion utterance refers to an utterance when speaking with a "normal" emotion. In addition, "utterance" refers to a voice (voice data) generated by an action that expresses a linguistic expression. The input utterance and the normal emotion utterance of each learning data have different emotions and utterance contents (text to be spoken). The learning data storage unit 121 stores such learning data for multiple speakers. The contents of speech by each speaker may be different or the same. It is also preferable that multiple sets of training data with different utterance contents for the same speaker are included. This is to avoid the possibility that an emotion recognition model that returns a specific emotion recognition result from the utterance contents (phonological bias) may be constructed if each speaker speaks a fixed content. In this case, it is preferable that normal emotion utterances are also different in each set of training data for the same speaker. Therefore, it can be said that the training data storage unit 121 stores, as training data for multiple speakers, multiple input utterances by the speaker, multiple normal emotion utterances corresponding to each input utterance, and correct emotion labels corresponding to each input utterance.

[Acoustic feature extraction unit 11]
The acoustic feature extraction unit 11 receives an utterance (audio data of the utterance) as input, extracts an acoustic feature sequence from the utterance, and outputs the acoustic feature sequence. During training, the acoustic feature extraction unit 11-1 extracts an acoustic feature sequence from the input utterance of each training data, and the acoustic feature extraction unit 11-2 extracts an acoustic feature sequence from normal emotion utterance of each training data.

An acoustic feature series is data in which an input utterance is divided into short-time windows, acoustic features are obtained for each short-time window, and the acoustic feature vectors are arranged in chronological order. The acoustic features include one or more of the power spectrum, Mel filter bank output, MFCC, fundamental frequency, logarithmic power, HNR (Harmonics-to-Noise Ratio), speech probability, number of zero-crossings, and their first or second derivatives. The speech probability is obtained, for example, from the likelihood ratio of a pre-trained speech/non-speech GMM model. HNR can be calculated, for example, using a cepstrum-based method (Peter Murphy, Olatunji Akande, "Cepstrum-Based Harmonics-to-Noise Ratio Measurement in Voiced Speech", Lecture Notes in Artificial Intelligence, Nonlinear Speech Modeling and Applications, Vol. 3445, Springer-Verlag, 2005). By using more acoustic features, it is possible to express various characteristics contained in speech, which tends to improve the accuracy of emotion recognition.

[Learning unit 12]
The learning unit 12 inputs the acoustic feature sequences of the input utterance and normal emotional utterance into an emotion recognition model (hereinafter simply referred to as "emotion recognition model m1") based on the acoustic feature sequences output from the acoustic feature extraction unit 11 for each of the input utterance and normal emotional utterance of each piece of learning data, and the correct emotion label and speaker label of each piece of learning data, and learns the emotion recognition model m1 using the correct emotion label and speaker label of the input utterance as teacher data.

FIG. 4 is a diagram showing an example of the configuration of emotion recognition model m1 in an embodiment of the present invention. As shown in FIG. 4, emotion recognition model m1 includes two emotion expression vector extraction blocks m11 (emotion expression vector extraction block m11-1, emotion expression vector extraction block m11-2) and one emotion probability estimation block m12.

The emotional expression vector extraction block m11 converts the acoustic feature sequence into a fixed-length vector (hereinafter referred to as "emotion expression vector") that represents the nature (or features) of the emotional expression of the entire utterance during the process in which the emotion recognition model m1 recognizes the speaker's emotion. The emotional expression vector extraction block m11 uses a deep learning model (for example, a model composed of a Transformer and Self Attentive Pooling) that extracts a fixed-length expression from an input vector sequence of any length. In Figure 4, the emotional expression vector extraction block m11-1 converts the acoustic feature sequence extracted from the input utterance into an emotional expression vector. The emotional expression vector extraction block m11-2 converts the acoustic feature sequence extracted from normal emotional utterance into an emotional expression vector.

The emotion probability estimation block m12 uses a deep learning model (e.g., a model with one or more layers of fully connected layers and activation functions) that projects a vector representing the posterior probability of each emotion (a vector in which each dimension indicates the posterior probability of a different emotion) from a fixed-length vector (emotion expression vector).

As shown in FIG. 5, the learning unit 12 updates the model parameters of all models based on a loss function L (L=αL a +(1-α)L _p ) obtained by weighting and summing a speaker identity loss function L p (FIG. _A3 ) for normal emotional utterances and a loss function L _a (a loss function for minimizing the error between the correct emotion label of the input utterance and the emotion recognition result) based on the error ( _error ) between the output of the emotion recognition model m1 and the correct emotion label, where α is a weighting coefficient that is determined manually. The learning unit 12 updates the model parameters using the stochastic gradient descent method, as in the conventional technology. After updating the model parameters a certain number of times, the learning unit 12 outputs the finally obtained emotion recognition model m1.

FIG. 6 is a diagram showing an example of the functional configuration of emotion recognition device 10 during emotion recognition in an embodiment of the present invention. In FIG. 6, the same parts as in FIG. 3 are given the same reference numerals, and their explanations are omitted.

As shown in FIG. 6, when recognizing emotions, the emotion recognition device 10 has an emotion recognition unit 13 instead of a learning unit 12. The emotion recognition unit 13 is realized by a process in which one or more programs installed in the emotion recognition device 10 are executed by the processor 104. Furthermore, when recognizing emotions, the emotion recognition device 10 does not use the learning data storage unit 121. Note that different computers may be used for learning and emotion recognition.

[Emotion Recognition Unit 13]
The emotion recognition unit 13 receives as input an acoustic feature sequence extracted by the acoustic feature extraction unit 11-1 from the input utterance of a person to be recognized for emotion, and an acoustic feature sequence extracted by the acoustic feature extraction unit 11-2 from the person's normal emotional utterance, and outputs an emotion recognition result (hereinafter simply referred to as "emotion recognition result") by comparing the acoustic feature sequence with the normal emotional utterance by forward propagating the acoustic feature sequence to a trained emotion recognition model m1. The emotion recognition result, which is the output of the emotion recognition unit 13, includes a posterior probability vector of each emotion (output of the forward propagation of the emotion recognition model m1), and an emotion class with the maximum posterior probability in the posterior probability vector. The emotion class with the maximum posterior probability is used as the final emotion recognition result.

The processing steps executed by the emotion recognition device 10 are explained below.

[Study]
FIG. 7 is a flowchart illustrating an example of a process performed by the emotion recognition device 10 during learning.

In step S101, the learning unit 12 randomly selects a number of (N) pieces of learning data from the group of learning data stored in the learning data storage unit 121 to obtain one mini-batch (hereinafter referred to as the "target mini-batch"). More precisely, the learning unit 12 randomly rearranges the learning data to generate the mini-batch.

The emotion recognition device 10 then executes a loop process including steps S102 to S104 for each of the N pieces of training data included in the target mini-batch. The training data being processed in this loop process is hereinafter referred to as the "target training data."

In step S102, the acoustic feature extraction unit 11-1 extracts an acoustic feature sequence from the input utterance of the target learning data, and the acoustic feature extraction unit 11-2 extracts an acoustic feature sequence from the normal emotion utterance of the target learning data.

Then, the learning unit 12 inputs the two extracted acoustic feature sequences into emotion expression model a, and acquires the recognition result (posterior probability vector for each emotion) and the emotion expression vector of normal emotion utterance output by emotion expression model a (S103). That is, in the process in which emotion expression model a outputs the recognition result, the emotion expression vector extraction block m11-2 calculates the emotion expression vector related to normal emotion utterance, and the learning unit acquires this emotion expression vector. The learning unit 12 stores the acquired emotion expression vector in association with the speaker label of the target training data.

Next, the learning unit 12 calculates a loss function L _a (the cross-entropy function between the correct emotion label of the input utterance and the posterior probability vector of each emotion) based on the correct emotion label of the target training data and the recognition result (the posterior probability vector of each emotion) to calculate a loss based on the error for the correct emotion label of the recognition result (hereinafter referred to as "loss L _a ") (S104).

When steps S102 to S104 have been executed for all training data in the target mini-batch, the training unit 12 calculates the average value for each speaker for the emotional expression vectors (of normal emotional utterances) obtained for each training data in the target mini-batch in step S103 (S105).

Fig. 8 is a diagram for explaining calculation of the average value of emotion expression vectors of normal emotion utterances for each speaker. In Fig. 8, e _ji indicates the emotion expression vector of the i-th normal emotion utterance of speaker j in the target mini-batch. The speaker of a certain emotion expression vector can be identified based on the speaker label associated with the emotion expression vector in step S103. Fig. 8 shows an example in which emotion expression vectors of normal emotion utterances are obtained for each speaker whose speaker label is 1, 2, or 3.

The learning unit 12 calculates the average value of each set of emotion expression vectors associated with the same (common) speaker label. Hereinafter, the average value for each speaker is referred to as the "speaker average c _k " (k is the speaker label). In the example of Fig. 8, the speaker labels are 1, 2, and 3, so speaker averages c ₁ , c ₂ , and c ₃ are calculated.

Next, the learning unit 12 calculates the distance S _ji,k between each emotional expression vector e _ji and the speaker average c _k (S106).

FIG. 9 is a diagram for explaining the calculation of the distance S _ji,k between each emotional expression vector e _ji and each speaker average c _k .

9 shows a matrix in which each row corresponds to each e _ji and each column corresponds to each c _k . Each element S _ji,k of the matrix corresponds to the distance between e _ji and c _k . The learning unit 12 calculates the distance based on, for example, the following formula.

但し、ｗ、ｂは学習可能なパラメータであり、感情認識モデルｍ１を構成する各ブロック（各モデル）のパラメータと同時に損失関数に基づいて更新される。

Here, w and b are learnable parameters, and are updated based on the loss function simultaneously with the parameters of each block (each model) constituting the emotion recognition model m1.

Next, the learning unit 12 calculates a loss based on the distance (hereinafter, referred to as "loss _Lp ") using the following loss function _Lp (S107).

但し、ｊ＝ｋのとき（ｅ_ｊｉとｃ_ｋとの話者が同じ場合）ｔ_ｊｉ，ｋ＝１，ｊ≠ｋのとき（ｅ_ｊｉとｃ_ｋとの話者が異なる場合）ｔ_ｊｉ，ｋ＝０とし、Ｓ_ｊｉ，ｋとｔ_ｊｉ，ｋが一致するように（Ｌ_ｐが最小化されるように）学習が行われる。なお、図９に示した行列において、網掛けが付与されている要素は、ｊ＝ｋである要素に対応する。すなわち、損失関数Ｌ_ｐは、ｊ＝ｋの場合（感情表現ベクトルと、感情表現ベクトルの話者平均が同じ話者の場合）は１に近づき、そうでない場合は０に近づくような損失関数である。

However, when j=k (when the speaker of _eji and _ck is the same), _tji,k =1, and when j≠k (when the speaker of _eji and _ck is different), _tji,k =0, and learning is performed so that _Sji ,k and _tji ,k match (so that _Lp is minimized). In the matrix shown in FIG. 9, the shaded elements correspond to the elements where j=k. In other words, the loss function _Lp approaches 1 when j=k (when the emotional expression vector and the speaker average of the emotional expression vector are the same speaker), and approaches 0 otherwise.

In other words, the loss function _Lp is a loss function for making the emotional expression vector calculated for normal emotional utterances constant for each speaker in the process in which the emotion recognition model m1 outputs the recognition result (for minimizing the error of the emotional expression vector for each speaker).

Next, the learning unit 12 calculates the weighted sum of the loss _Lp based on the loss function _Lp and the loss _{La based on the loss function La as} the loss L of the entire emotion recognition model m1 (S108). Here, the calculation formula for the loss L is, for example, as follows:
L = αΣ(L _a /N) + (1-α)L _p
Here, α is a hyperparameter (weighting coefficient) for adjusting the priority between the role of L _a in "reducing emotion recognition errors" and the role of L _p in "making the normal emotion vector of the same speaker take a similar value." The larger the weight of L _a is, the weaker the effect of L _p is, and the larger the weight of L _p is, the more emotion recognition errors are tolerated to a certain extent.

In addition, N is the number of training data in the target mini-batch, i.e., Σ(L _a /N) is the average value of L _a in the target mini-batch.

Next, the learning unit 12 simultaneously updates the parameters of each block in the emotion recognition model m1 and the loss function L _P using the backpropagation algorithm based on L (S109). By repeating steps S101 to S109, the parameters of the blocks are simultaneously optimized.

When steps S101 to S109 have been repeated a predetermined number of times (Yes in S110), the learning unit 12 ends the processing procedure in FIG. 7.

[When recognized]
The process executed by the emotion recognition device 10 during recognition is as described with reference to Fig. 6. That is, the emotion recognition device 10 receives as input an utterance of a speaker to be subjected to emotion recognition and the speaker's normal emotional utterance. The acoustic feature extraction unit 11-1 extracts an acoustic feature series from the utterance, and the acoustic feature extraction unit 11-2 extracts an acoustic feature series from the normal emotional utterance. The emotion recognition unit 13 inputs these two acoustic feature series to a trained emotion recognition model m1, and recognizes the speaker's emotion based on the probability for each emotion label output by the emotion recognition model m1.

As described above, according to this embodiment, the emotional expression vector extraction block m11-2 learns based on _Lp so as to be more likely to output similar (relatively small distance) emotional expression vectors for multiple normal emotion utterances by the same speaker. By including the weighted sum of _Lp in L, it is possible to alleviate the problem that in the conventional technology, there is no guarantee that normal emotion utterances by the same speaker will have similar emotional expression vectors (= recognition results often change when normal emotion utterances change). In other words, it is possible to eliminate the phenomenon in which an estimated result is output by specializing in a combination of an input utterance and a normal emotion utterance in emotion recognition using normal emotion utterances. This increases the possibility of obtaining the same recognition result for different normal emotion utterances, which contributes to improving the recognition accuracy of the speaker's emotion from the utterance.

In this embodiment, the loss function L _a is an example of a first loss function, and the loss function L _p is an example of a second loss function.

The above describes in detail the embodiments of the present invention, but the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

10 Emotion recognition device 11-1 Acoustic feature extraction unit 11-2 Acoustic feature extraction unit 12 Learning unit 13 Emotion recognition unit 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device 121 Learning data storage unit B Bus m1 Emotion recognition model m11 Emotion expression vector extraction block m11-1 Emotion expression vector extraction block m11-2 Emotion expression vector extraction block m12 Emotion probability estimation block

Claims (8)

a learning procedure for learning the model based on learning data including, for a plurality of speakers, a plurality of input utterances by a speaker, a plurality of normal emotion utterances corresponding to each of the input utterances, and a correct answer label of the emotion corresponding to each of the input utterances, and based on: a first loss function for minimizing an error, relative to the correct answer label corresponding to the input utterance, of an emotion recognition result output by a model to which any of the input utterances and the normal emotion utterance corresponding to the input utterance have been input; and a second loss function for making constant, for each speaker, a vector representing the nature of the emotional expression calculated for the normal emotion utterance in a process in which the model outputs the recognition result;
The emotion recognition learning method is characterized in that the emotion recognition learning method is executed by a computer. the second loss function is based on a distance between an average value of the vectors calculated by the model for each of the normal emotion utterances corresponding to the same speaker and each of the vectors;
2. The emotion recognition learning method according to claim 1 . the learning procedure includes learning the model based on a weighted sum of the first loss function and the second loss function.
3. The emotion recognition learning method according to claim 1 or 2. an emotion recognition step of recognizing the emotion of a speaker related to an input utterance and a normal emotional utterance by using a model trained by the emotion recognition training method according to claim 1;
The emotion recognition method is characterized in that the emotion recognition method is executed by a computer. a learning unit configured to learn the model based on learning data including, for a plurality of speakers, a plurality of input utterances by a speaker, a plurality of normal emotion utterances corresponding to each of the input utterances, and a correct answer label of the emotion corresponding to each of the input utterances, based on: a first loss function for minimizing an error, with respect to the correct answer label corresponding to the input utterance, of an emotion recognition result output by a model to which any of the input utterances and the normal emotion utterance corresponding to the input utterance have been input; and a second loss function for making constant, for each speaker, a vector representing the nature of the emotional expression calculated for the normal emotion utterance in a process in which the model outputs the recognition result;
An emotion recognition learning device comprising: an emotion recognition unit configured to recognize emotions of a speaker related to an input utterance and a normal emotional utterance by using a model trained by the emotion recognition training method according to claim 1;
An emotion recognition device comprising: A program that causes a computer to execute the emotion recognition learning method described in claim 1. A program that causes a computer to execute the emotion recognition method described in claim 4.

Images