JP2005352311A - Speech synthesis apparatus and speech synthesis program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a speech synthesizer and a speech synthesis program capable of easily correcting a synthesized speech when there is an error in the inflection or prosody of the synthesized speech or when it is desired to adjust the inflection or prosody.
SOLUTION: A voice signal receiving means 110 for receiving a voice signal obtained from a voice of a speaker, a phrase conversion means 120 for converting a voice signal received by the voice signal receiving means 110 into a phrase, and a voice signal receiving means. An emotion estimation unit 130 for estimating the speaker's emotion based on the audio signal received by 110, and a synthesized speech change unit 140 for changing the synthesized speech information corresponding to the phrase according to the emotion estimated by the emotion estimation unit 130. It comprises and comprises.
[Selection] Figure 1

Description

  The present invention relates to a speech synthesizer and a speech synthesis program that output synthesized speech based on synthesized speech information expressed in text or the like.

As a conventional speech synthesizer, an operator uses an editor to insert a prosody control command into a text described in MSCL (Multi-layered Speech / Sound Synthesis Control Language), so that characters that require prosody correction are inserted. It is known that prosodic parameters including phonetic string information, accent information, and the like are generated for a sequence and a synthesized speech is output according to the generated prosodic parameters (see, for example, Patent Document 1).
Japanese Patent Laid-Open No. 11-202844 (pages 9 and 15) JP 2002-230598 A JP-A-5-289691 “Acoustics and Speech Engineering”, Sadayoshi Yoshii, Modern Science, 1992 "Voice coding", Takehiro Moriya, IEICE, 1998 “Digital Audio Processing”, Sadayoshi Yoshii, Tokai University Press, 1985 Y. Linde, A. Buzo and RMGray. “An algorithm for vector Quantizer design”, IEEE Trans.Commun., Vol. Com-28, pp.84-95, 1980

  However, in such a conventional speech synthesizer, it is necessary to manually correct the synthesized speech information using an editor when there is an error in the insufflation or prosody of the synthesized speech. The problem of making it feel was left.

  The present invention has been made to solve such a problem, and in the case where there is an error in the inflection or prosody of the synthesized speech, or when it is desired to adjust the inflection or prosody, etc., the synthesized speech is simply corrected. An object of the present invention is to provide a speech synthesizer and a speech synthesis program.

The speech synthesizer of the present invention includes a speech signal receiving unit that receives a speech signal obtained from a speech of a speaker, a phrase conversion unit that converts a speech signal received by the speech signal receiving unit, and the speech Emotion estimation means for estimating the speaker's emotion based on the speech signal received by the signal reception means, and synthetic speech change for changing the synthesized speech information corresponding to the phrase according to the emotion estimated by the emotion estimation means And means.
With this configuration, the synthesized speech information corresponding to the phrase obtained from the speech of the speaker is changed according to the emotion of the speaker, so there is an error in the inflection or prosody of the synthesized speech, or the inflection or prosody etc. The synthesized speech can be easily corrected when it is desired to adjust the sound.

The speech synthesizer of the present invention further includes a translation unit that translates the phrase converted by the phrase conversion unit into a phrase of another language, and the synthesized speech change unit is configured to respond to the emotion estimated by the emotion estimation unit. The synthetic speech information corresponding to the words translated by the translation means is changed.
With this configuration, since the phrase obtained from the speech of the speaker is translated into another language, the synthesized speech of the speech synthesizer used as a translator can be easily corrected.

The speech synthesis program of the present invention includes a computer that receives a speech signal received from a speaker's speech, a speech signal reception step, and a phrase conversion step that converts the speech signal received in the speech signal reception step into a phrase; An emotion estimation step for estimating the emotion of the speaker based on the audio signal received in the audio signal reception step, and a synthetic speech for changing the synthetic audio information corresponding to the phrase according to the emotion estimated in the emotion estimation step The change step is executed.
This program changes the synthesized speech information corresponding to the phrase obtained from the speech of the speaker according to the emotion of the speaker, so if there is an error in the inflection or prosody of the synthesized speech, or the inflection or prosody The synthesized speech can be easily corrected when it is desired to adjust the sound.

The speech synthesis program according to the present invention translates the phrase converted in the phrase conversion step into a phrase in another language and the translation means step according to the emotion estimated in the emotion estimation step. Changing the synthesized speech information corresponding to the phrase.
This program translates a phrase obtained from the speech of the speaker into a phrase in another language, so that the synthesized speech of a speech synthesizer used as a translator can be easily corrected.

  The present invention provides a speech synthesizer and a speech synthesis program that can easily correct a synthesized speech when there is an error in the inflection or prosody of the synthesized speech or when it is desired to adjust the inflection or prosody. Is.

  Hereinafter, the best mode for carrying out the present invention will be described.

  FIG. 1 is a block configuration diagram of a speech synthesizer according to an embodiment of the present invention. As shown in FIG. 1, the speech synthesizer 100 includes a speech signal receiving unit 110, a phrase conversion unit 120, an emotion estimation unit 130, a synthesized speech changing unit 140, a speech synthesis unit 150, and a translation unit 160. Yes.

  The audio signal receiving means 110 receives an audio signal obtained from the user's voice via a microphone or the interface unit 106. The phrase converter 120 converts the voice signal received by the voice signal receiver 110 into a phrase. The phrase conversion unit 120 converts the voice signal into, for example, a text that is a text string using a known voice recognition technique.

  The emotion estimation unit 130 is configured to estimate the speaker's emotion based on the audio signal received by the audio signal reception unit 110. The emotion estimation means 130 outputs emotion information representing the estimated emotion to the synthesized voice changing means 140. A detailed description of the emotion estimation means 130 will be described later.

  The synthesized voice changing unit 140 changes the synthesized voice information corresponding to the phrase according to the emotion estimated by the emotion estimating unit 130. For example, the synthesized speech changing unit 140 may change the synthesized speech information based on a correspondence table in which emotion information output from the emotion estimating unit 130 corresponds to a command based on MSCL. Here, an example of the correspondence table is shown in Table 1.

As a more detailed example, when the phrase “Today is sunny” is converted from the audio signal by the phrase conversion unit 120 and emotion information indicating “sadness” is output from the emotion estimation unit 130, Based on the correspondence table shown in Table 1, the synthesized voice changing means 140 is for outputting a voice with a feeling of mourning to the synthesized voice information in the text format representing the phrase “Today is sunny” stored in advance. [@Naki] which is a command of the S layer is inserted. That is, the synthesized speech changing means 140 changes to synthesized speech information described in MSCL as shown below.
“@Naki {Sunny today}”

  The voice synthesizer 150 synthesizes voice according to the synthesized voice information. For example, the hard disk 105 stores synthesized speech information in a text format that conforms to MSCL, and the speech synthesizer 150 synthesizes speech in accordance with the synthesized speech information in a text format that conforms to MSCL, thereby synthesizing the synthesized speech into a speaker. The data is output via 108. Of course, if the synthesized speech information has been changed by the synthesized speech changing means 140, the speech synthesizing means 150 synthesizes speech according to the changed synthesized speech information.

  The translation unit 160 translates the phrase converted by the phrase conversion unit 120 into a phrase in another language. In addition, the synthesized speech changing unit 140 changes the synthesized speech information corresponding to the phrase translated by the translating unit 160 according to the emotion estimated by the emotion estimating unit 130.

  For example, when translating from English to Japanese, the user utters the word “How are you?” To the speech synthesizer 100, and the speech signal receiving means 110 receives the speech signal representing this word. Then, the phrase conversion means 120 converts the received voice signal into the phrase “How are you?”. Here, the translation means 160 is adapted to translate the phrase "Hello" from the phrase "How are you?". It should be noted that, when the user had been speaking with great sadness the word "How are you?", Synthesized speech change means 140, in text format synthetic speech information of which represents the phrase say "Hello" which is stored in advance, Based on the correspondence table shown in Table 1, a command for outputting a voice having a feeling of grief is inserted.

  Next, FIG. 2 is a schematic diagram showing a hardware configuration of the speech synthesizer according to the embodiment of the present invention. A CPU (Central Processing Unit) 101 is configured to execute a program related to the speech synthesizer 100. Note that the speech signal receiving unit 110, the phrase conversion unit 120, the emotion estimation unit 130, the synthesized speech changing unit 140, the speech synthesis unit 150, and the translation unit 160 may be modules of a program executed by the CPU 101.

  A ROM (Read Only Memory) 102 stores a program read by the CPU 101, a program for starting up the CPU 101, other programs, control parameters, and the like. A RAM (Random Access Memory) 1033 stores programs, data, and the like required for the operation of the CPU 101 during the operation of the CPU 101. An EEPROM (Electrically Erasable Programmable Read-Only Memory) 104 stores a program and predetermined data in a nonvolatile and rewritable manner. The hard disk 105 stores various data. The interface unit 106 communicates with a device connected to the network according to a predetermined communication protocol.

  In addition, the speech synthesizer 100 includes a display 107 such as a liquid crystal display, input devices such as a microphone, a keyboard, and a mouse, and output devices such as a speaker 108. Note that the speech synthesizer 100 can also be realized using a computer including a personal computer.

  For example, in the speech synthesizer 100, text-to-speech synthesized speech information representing phrases described in MSCL or the like is stored in advance on the hard disk 105, and the speech synthesizer 100 synthesizes according to the stored synthesized speech information. It is designed to emit sound. The synthesized voice information may be an electronic file representing text format data.

  For example, since it has the synthesized speech information “Today is sunny” described in MSCL or the like, the speech synthesizer 100 can emit the synthesized speech with the word “Today is sunny”. The user who has heard this utters the same word “today is fine” to the speech synthesizer 100 when he wants to adjust the inflection or prosody of this word.

  Next, FIG. 3 is a block diagram of emotion estimation means according to the embodiment of the present invention. As shown in FIG. 3, the emotion estimation unit 130 includes a storage unit 131, a voice feature amount extraction unit 132, an emotion expression likelihood calculation unit 133, a calm state likelihood calculation unit 134, and an emotion expression determination unit 135. Configured.

  The storage means 131 is a codebook generated using the learning speech, and is a vector quantized speech feature vector generated from a speech feature amount set included in the learning speech and a code and speech corresponding thereto. , The emotional expression probability that is the appearance probability of the speech feature vector when the speaker's emotion is expressed, and the speech feature vector when there is no speaker's emotional expression The code book 131CB that stores the calm state probability, which is the appearance probability, is stored in advance. For example, as shown in FIG. 9, the code book is created corresponding to emotions such as “laughter”, “anger”, and “sadness”. The code book corresponding to each emotion is not limited to three such as “laughter”, “anger”, and “sadness”, and a large number of codebooks may be prepared.

  The audio feature amount extraction unit 132 extracts an audio feature amount vector from audio data included in the input content received by the audio signal reception unit 12.

  The emotion expression likelihood calculating unit 133 detects a speech feature amount vector corresponding to the speech feature amount vector extracted by the speech feature amount extracting unit 132 from the code book, and corresponds to the speech feature amount vector detected from the code book. Based on the emotional expression probability of the emotion, the emotional expression state likelihood, which is the likelihood of the speaker's emotional expression, is calculated. For example, the emotion expression likelihood calculating means 133 calculates each emotion expression state likelihood based on a codebook corresponding to each emotion.

  The calm state likelihood calculating unit 134 detects a speech feature amount vector corresponding to the speech feature amount vector extracted by the speech feature amount extracting unit 132 from the code book, and corresponds to the speech feature amount vector detected from the code book. Based on the calm state probability, the calm state likelihood, which is the likelihood of the speaker's calm state, is calculated. For example, the calm state likelihood calculating unit 134 calculates each calm state likelihood based on a codebook corresponding to each emotion.

  The emotional expression determination unit 135 extracts the voice feature amount based on the emotional expression state likelihood calculated by the emotional expression likelihood calculation unit 133 and the calm state likelihood calculated by the calm state likelihood calculation unit 134. The means 132 is configured to determine whether or not the speaker's emotional expression is present in each section of the voice including the voice feature vector extracted from the voice data. When there is an emotion expression, the emotion expression determination means 135 outputs emotion information representing the determined emotion to the image extraction means 14.

  For example, when it is determined whether or not the speaker has expressed “laughter” as an emotion, the emotion corresponding to the “laughter” calculated by the emotion expression likelihood calculating unit 133 using the codebook of “laughter”. Based on the expression likelihood and the calm state likelihood corresponding to the “laughter” calculated by the calm state likelihood calculation unit 134 using the code book of “laughter”, the emotional expression determination unit 135 indicates “laughter”. Judgment is made on whether or not there was an emotional expression.

  Although not shown in FIG. 3, the audio feature quantity extraction unit 132 includes buffer memory unit, temporarily stores the input audio content, and stores audio data included in the content in the buffer memory unit. The speech feature vector is extracted by analysis.

  Hereinafter, a program executed by the speech synthesizer according to the embodiment of the present invention will be described with reference to the drawings. FIG. 4A is a flowchart showing a flow of an operation in which the speech synthesizer according to the embodiment of the present invention changes the synthesized speech information.

  First, an audio signal obtained from audio via a microphone or the interface unit 106 is received by the audio signal receiving unit 110 (S101), and the received audio signal is converted into a phrase by the phrase conversion unit 120 ( S102).

  Here, the speech synthesizer 100 has determination information for determining whether or not to translate the converted word / phrase in advance, and determines whether translation is possible based on this determination information (S103). As a result of the determination, in the case of translation, the converted word / phrase is translated into a word / phrase in another language by the translation means 160 (S104).

  Next, emotions of the user or the like are estimated by the emotion estimation means 130 based on the received audio signal (S105). Note that step S105 and steps S102 to S104 may be executed in parallel. The synthesized speech information corresponding to the phrase converted by the phrase converting unit 120 is changed by the synthesized speech changing unit 140 according to the emotion estimated by the emotion estimating unit 130 (S106).

  FIG. 4B is a flowchart showing a flow of operations in which the speech synthesizer according to the embodiment of the present invention outputs synthesized speech.

  For example, a request signal for outputting synthesized speech in response to a request from a user or the like is input to the speech synthesizer 100 (S107). In response to the request signal, the synthesized speech is output by the speech synthesizer 150 according to the synthesized speech information (S108).

  Further, the operation of the emotion estimation means according to the embodiment of the present invention will be described. FIG. 5A is a diagram for explaining the codebook in detail.

  A code book generated using learning speech is created in advance. For example, it is the appearance probability of the voice feature amount when there is a predetermined voice feature amount included in the learning voice, the emotion of the speaker who uttered the voice, and the emotional expression of the speaker (hereinafter referred to as emotion expression state) The codebook that stores the emotion expression probability and the calm state probability that is the appearance probability of the voice feature amount when there is no speaker emotion expression is stored in advance by the storage unit 131 (S110). . Details of the codebook creation will be described later.

  Next, the operation | movement of the emotion estimation means 130 which concerns on embodiment of this invention is demonstrated. FIG. 5 is a flowchart for explaining the operation of emotion estimation means 130 according to the embodiment of the present invention.

  First, a codebook generated using learning speech, which is a speech feature vector that is a set of predetermined speech features (a set of parameters) included in the learning speech, the emotion of the speaker who has spoken, Emotional expression probability, which is the appearance probability of a voice feature vector when a person's emotional expression is expressed (hereinafter referred to as emotional expression state), and a voice feature amount when there is no speaker's emotional expression A code book that associates and holds a calm state probability that is the appearance probability of a vector is created in advance and stored in the storage unit 131 of FIG. 1 (S310). This code book creation process is performed in advance as necessary for configuring the apparatus of the present invention. The details of the codebook creation will be described later, but the speech feature vector is at least the fundamental frequency, the average power, and the time change characteristics of the dynamic feature detected as per speech frame as disclosed in Patent Document 2. Is a vector including at least one parameter set of at least one of the above and / or at least one of the inter-frame differences.

  Steps S320 to S340 are emotion expression detection processing. First, the entire input content is taken into the storage means 131, and a plurality of predetermined sets of voice feature quantities (speech feature quantity vectors) are extracted from the fetched voice data (S320).

  In S320, a speech feature vector closest to the series of speech feature vectors extracted from the predetermined section (determination section) of the speech data is detected from the code book, and the emotion of the detected speech feature vector from the code book is detected. The appearance probability in the expression state is read, and the emotion expression state likelihood, which is the likelihood of the speaker's emotion expression, is calculated based on the series of application probabilities. The calculation of the emotional expression state likelihood is performed for each determination section of a series of voice feature vectors of the voice data (for example, for each voice sub-paragraph described later or for each fixed section length).

  Similarly, the appearance probability of the speech feature tatami vector extracted from the speech data in the calm state is read from the codebook, and based on this probability, the calm state likelihood, which is the likelihood of the speaker's calm state, is expressed as the emotion. It is calculated for each determination section that is the same as the calculation of the exposed state likelihood (S330).

  Next, based on the emotional expression state likelihood and the calm state likelihood calculated in step S330, the speaker is included in the determination section of the audio data including the predetermined audio feature amount set extracted from the audio data in step S320. It is determined whether or not there is an emotional expression (S340).

Hereinafter, the processing in each of the above steps will be described in detail. First, prior to a detailed description of the processing in each step, the above-described set of audio feature values will be described. As the speech feature amount, a speech feature amount that is obtained stably even in a noisy environment and that is less dependent on the speaker is used to determine whether or not it is in an emotional expression state compared to information such as a speech spectrum. In the embodiment of the present invention, as the audio feature amount satisfying such conditions, the fundamental frequency f ₀ , power p, dynamic feature amount d (t). Extracting the unvoiced _{T S} and the like.

  These voice feature extraction methods are known, and for details, see Non-Patent Document 1, Non-Patent Document 2, Non-Patent Document 3, and the like.

  Here, the dynamic feature amount d (t) is defined by the following equation (1), and the temporal change amount is a parameter serving as a measure of the speech rate.

Here, t is the time, C _k (t) is the kth-order LPC cepstrum coefficient at time t, and ± F ₀ is the number of frames before and after the target frame (hereinafter referred to as the current frame) (not necessarily an integer number of frames). It may be a fixed time interval). As the dynamic feature amount d (t), one defined in Patent Document 3 may be used.

  The order k of the LPC cepstrum coefficient is any integer from 1 to K. The number of local maximum points per unit time of the dynamic feature quantity d (t) or the rate of change per unit time is a measure of the speech rate.

In the following, it is assumed that the length of one frame (hereinafter referred to as the frame length) is 100 ms, and the next frame is formed with a shift of 50 ms from the start time of this frame. Further, the average fundamental frequency f ₀ ′ and the average power P are calculated for each frame. The average fundamental frequency f ₀ ′ and the average power p ′ are calculated using only the frames for which the fundamental frequency f ₀ is reliable. For example, it is possible to utilize the autocorrelation coefficients of the extraction of the fundamental frequency f _0. Furthermore, _'and the fundamental frequency f ₀ of the previous i-th frame from the current _frame' fundamental frequency f ₀ of the current frame _{'respectively} and a difference Delta] f _0' of the fundamental frequency f ₀ after i frames from and the current frame (-i), Delta] f ₀ ′ (i). Similarly for the average power p ′, the average power p ′ of the current frame, the average power p ′ before i frames from the current frame, and the average power p ′ after i frames from the current frame, and the difference Δp ′ (−i ), Δp ′ (i).

Next, for each frame, fundamental frequency f ₀ ′, fundamental frequency differences Δf ₀ ′ (−i), Δf ₀ ′ (i), average power p ′, average power differences Δp ′ (− i), Δp '(I) is normalized. In the following, the fundamental frequency f ₀ ′, fundamental frequency differences Δf ₀ ′ (−i), Δf ₀ ′ (i), average power p ′, average power differences Δp ′ (− i), Δp ′ (i) Each is simply expressed as f ₀ ′, Δf ₀ ′ (−i), Δf ₀ ′ (i), p ′, Δp ′ (−i), Δp ′ (i), and the normalized ones are respectively , F ₀ ″, Δf ₀ ″ (−i), Δf ₀ ″ (i), p ″, Δp ″ (− i), Δp ″ (i).

This normalization may be performed, for example, by dividing each of f ₀ ′, Δf ₀ ′ (−i), and Δf ₀ ′ (i) by, for example, the average fundamental frequency of the entire audio data to be processed. , It may be standardized to have an average of 0 and a variance of 1. Further, instead of the average fundamental frequency of all audio data to be processed, an average fundamental frequency for each of the audio sub-paragraphs and audio paragraphs described later, or an average fundamental frequency within a time such as several seconds or minutes may be used. .

  Similarly, p ′ is also normalized or standardized by dividing by the average power of all the audio data to be processed. Further, instead of the average power of the entire audio data to be processed, an average power for each audio sub-paragraph or audio paragraph, which will be described later, or an average power within a time such as several seconds or several minutes may be used. Here, the value of i is set to 4, for example.

The number of dynamic feature (dynamic measure) peaks is calculated as follows. First, a section having a sufficiently long time width (2T ₁ , where T ₁ is, for example, about 10 times the frame length) is provided around the start time of the current frame. Next, the maximum point of the time variation of the dynamic feature amount d (t) in this section is calculated, and the number of maximum points d _p (hereinafter simply referred to as d _p ) is counted.

Also, the difference value of the number of dynamic major peaks is calculated as described below. That is, the difference component Δd _p (−T ₂ ) obtained by subtracting d _p of the current frame from d _p in the section within the width 2T ₁ centered on the time before T ₂ of the start time of the current frame is obtained. Similarly, the _{d p} in the interval in the width 2T ₁ around the time after _{T 3} of the end time of the current frame, determining a difference component [Delta] d _p was subtracted from _{d p} of the current frame _{(T 3).}

The values of T ₁ , T ₂ , and T ₃ described above are sufficiently longer than the frame length, and in the following, T ₁ = T ₂ = T ₃ = 450 ms. However, it is not restricted to these values. Also, let the lengths of the silent sections before and after the frame be t _SB and t _SF , respectively. In step S320, the above-mentioned f ₀ ″, Δf ₀ ″ (− i), Δf ₀ ″ (i), p ″, Δp ″ (− i), Δp ″ (i), d _p , Δd _p (−T ₂ ), Δd _p (T ₃ ), etc. (hereinafter referred to as parameters) for each frame.

When creating the code book, the above f ₀ ″, Δf ₀ ″ (−i), Δf ₀ ″ (i), p ″, Δp ″ (−i), Δp ″ (i), d _p , Δd _p ″ (-T ₂ ), Δd _p ″ (T ₃ ) and other parameters selected from parameters such as (f ₀ ″, p ″, d _p ) (voice feature vector) The expression probability and the calm state probability are calculated, and the selected parameter is associated with the emotion expression probability and the calm state probability and recorded in the codebook. In the codebook, the same set of parameters as the above set of parameters is recorded as a speech feature vector.

In step S320, the target input speech, said speech feature quantity parameters _{f 0 ", Δf 0" (} -i), Δf 0 "(i), p", Δp "(- i), Δp" (i) , D _p , Δd _p ″ (−T ₂ ), Δd _p ″ (T ₃ ), etc., parameters used in the speech feature vector stored in the codebook, for example, (f ₀ ″ described above) , P ″, d _p ) are calculated for each frame to obtain a series of audio feature vectors over the entire audio content. As a result, the speech feature vector of the codebook corresponding to each speech feature vector of speech data can be specified, and the emotional expression probability and the calm state probability can be determined.

  Details of the processing in step S330 will be described with reference to FIG. In step S330, first, in steps S331 to S333, a small audio paragraph and an audio paragraph are extracted. Next, in steps S334, S335, and S336, the emotion expression state likelihood and the calm state likelihood are calculated. In this embodiment, an audio sub-paragraph is used as a unit for determining whether or not it is in an emotional expression state, and the audio paragraph is, for example, at least one audio sub-audio that is sandwiched in an unvoiced section of about 400 ms or more. It shall be a section including a paragraph. FIG. 7 conceptually shows the relationship between audio sub-paragraphs and audio paragraphs.

  To extract a voice paragraph, first, unvoiced and voiced sections of voice data are extracted (S331). The determination of whether it is a voiced section or an unvoiced section (hereinafter simply referred to as voiced / unvoiced judgment) is regarded as equivalent to the determination of the presence or absence of periodicity, and the peak of the autocorrelation function or modified correlation function Often done based on value.

Specifically, the spectral envelope is removed from the short-time spectrum of the input signal, the autocorrelation function of the obtained prediction residue (hereinafter referred to as a modified correlation function) is calculated, and the peak value of the modified correlation function is greater than a predetermined threshold value. The voiced / unvoiced judgment is made depending on whether the value is large. Further, the pitch period 1 / f ₀ is extracted based on the delay time of the correlation process that can obtain such a peak value.

  In the above description, the case where each voice feature amount is extracted from the voice data for each frame has been described. However, the voice data has already been encoded for each frame by CELP (Code-Excited Linear Prediction), for example, analysis. The audio feature quantity may be generated using a coefficient or code obtained by this encoding. A code obtained by CELP (hereinafter referred to as a CELP code) generally includes a linear prediction coefficient, a gain coefficient, a pitch period, and the like. Therefore, the speech feature tatami can be obtained by decoding the CELP code.

  Specifically, the absolute value or square value of the decoded gain coefficient can be used as power, and voiced / wireless determination can be performed based on the ratio between the gain coefficient of the pitch component and the gain coefficient of the aperiodic component. Further, the reciprocal of the decoded pitch period can be used as the pitch frequency, that is, the fundamental frequency. Further, the LPC cepstrum coefficient used for the calculation of the dynamic feature amount described in the above equation (l) can be obtained by converting the one obtained by decoding the CELP code.

  Further, if the CELP code includes an LSP (Line Spectrum Pair) coefficient, the LSP coefficient may be once converted into an LPC cepstrum coefficient and obtained from the LPC cepstrum coefficient obtained by conversion. Thus, since the CELP code includes speech feature values that can be used in the present invention, it is possible to decode the CELP code and extract a set of necessary speech feature values for each frame.

Returning to FIG. 6, when the times t _SM and t _SF of the unvoiced segments on both sides of the voiced segment are equal to or greater than the predetermined t _S , the signal portion including the voiced segment surrounded by the unvoiced segment is represented as the audio sub-paragraph S. (S332). Hereinafter, the value of the time t _S of the silent section is set to t _S = 400 ms, for example.

Next, the audio sub-paragraphs in S, preferably compares the average power p voiced in a section of the rear faculties, and a constant β times the average power value P _S of the audio sub-paragraph S, p <.beta.P _S If there is, the audio sub-paragraph S is set as the end audio sub-paragraph, and the audio sub-paragraph after the immediately preceding end audio sub-paragraph to the current end audio sub-paragraph is determined as the audio paragraph and extracted (S333).

The extraction of the audio sub-paragraph is performed under the condition that the time of the unvoiced section surrounding the voiced section is t _S or more. FIG. 7 shows S _j−1 , S _j , and S _{j + 1} as the audio sub-paragraphs. In the following, the audio sub-paragraph S _j is set as the audio sub-paragraph to be processed. The audio sub-paragraph S _j is composed of Q _j voiced sections, and the average power of the audio sub-paragraph S _j is P _j .

Further, the average power of the q-th voiced section V _q (q = 1, 2,..., Q) included in the audio sub-paragraph S _j is expressed as p _q . Whether or not the audio sub-paragraph S _j is the audio sub-paragraph at the end of the audio sub-paragraph B is determined based on the average power of the voiced section in the latter half of the audio sub-paragraph S _j . Specifically, the determination is made based on whether or not a condition shown in the following expression (2) is satisfied.

When this condition is satisfied, it is determined that the audio sub-paragraph S _j is the last audio sub-paragraph of the audio paragraph B.

Here, α is a constant that takes a value of Q _j / 2 or less, and β is a constant that takes a value of about 0.5 to 1.5, for example. These values are determined in advance by experiments so as to optimize the extraction of speech paragraphs. However, the average power p _q of the voiced section is the average power of all frames in the voiced section. In the embodiment of the present invention, α = 3 and β = 0.8. As described above, a set of audio sub-paragraphs between adjacent end audio sub-paragraphs can be determined as an audio paragraph. Alternatively, the small audio paragraph may be determined as a fixed length t (s) and a shift width S (s). For example, a fixed length and a shift width of t (s) = S (s) = 1 msec may be used. The voice paragraph may also be a section surrounded by a silent section of ΔS.

Next, returning to FIG. 6, processing (S334, S335) for calculating the emotion expression state likelihood will be described (hereinafter, this processing is referred to as emotion expression determination processing). First, in accordance with the speech feature quantity vector recorded in the code book created in advance in step S310, the speech feature quantity pairs included in the input speech extracted in step S320 are vector quantized, and the code strings C ₁ , C ₂ , C ₃ ,... Are obtained (S334).

  Prior to the calculation of the emotional expression state likelihood in step S335, a codebook creation method will be described with reference to FIG. First, a large number of learning voices are collected from the subject and labeled so that the utterances with emotional expression and the utterances in a calm state can be identified (S311). For example, the labeling is performed on the sections where the voice is determined to be laughing, angry, or sad.

  On the contrary, the reason for determining the calm state does not correspond to any of the above laughter, anger, and sadness, and it is assumed that the utterance is felt calm.

When the labeling is performed in step S311, the speech feature amount of a predetermined parameter set, for example, (f ₀ ″, p ″, d _p ), is determined from the labeled speech data in the same manner as the processing in step S320. Is extracted for each frame as a speech feature vector value (S312). LBG algorithm using emotion expression state or calm state information obtained by labeling and speech feature vector obtained for label section (labeled speech section) in emotion expression state or calm state A codebook is created according to (S313).

  The LBG algorithm is known and the details thereof are described in, for example, Non-Patent Document 4.

The number of entries recorded in the codebook (hereinafter referred to as code length size) is 2 ^m (m is an integer equal to or greater than 1) and is variable, and the code C is used as an entry index. Is an m-bit quantization vector (C = 00... 0 to 11... 1) corresponding to the code C.

  The code length is determined by the LBG algorithm using a total speech feature vector obtained in a desired section sufficiently longer than the frame length, for example, a label section of the learning speech, corresponding to the quantization vector (code C). The representative vector is recorded as a voice feature amount representative vector. In that case, you may normalize each audio | voice feature-value with the average value and standard deviation, for example. In the following description, the speech feature amount representative vector of the codebook is also simply referred to as a speech feature amount vector.

  Of the speech feature value parameters extracted from the input speech data, the set of parameters used for the emotion expression determination process is the same as the set of parameters used for the above codebook creation. In order to specify a voice sub-paragraph in the emotional expression state or the calm state, the appearance probability in each emotional expression state and the appearance probability in the calm state are associated with the code C (entry index) in the voice sub-paragraph. Are calculated respectively. At that time, the emotions are classified into “laughter”, “anger”, “sadness”, etc., and the appearance probabilities of the emotion expression state and the calm state are calculated for each emotion and recorded in one codebook. To do. Therefore, in the codebook, the code C, the voice feature vector, the appearance probability in the emotional expression state, and the appearance probability in the calm state are recorded correspondingly. These may be classified for each type of emotion and recorded in separate codebooks.

Below, an example of the calculation method of the emotion expression likelihood which is the likelihood about the speaker's emotion expression performed at step S335 and the method of calculating the calm state likelihood which is the likelihood about the calm state performed at step S336 will be described. Will be described. First, n is the number of frames included in the label section in the learning speech, and the codes corresponding to the speech feature amount sets obtained from the respective frames are C ₁ , C ₂ ,..., C _{n in} time series. It shall be.

As described above, the label section is one voice section labeled in step S311 of the process for creating the codebook. At this time, the emotional expression likelihood P _Aemo and the calm state likelihood P _{Anrm in} the label section A calculated in steps S335 and S336 are expressed as shown in the following expressions (3) and (4), respectively. Is done.

Here, P _emo (C _i | C ₁ ... C _i−1 ) is a conditional probability that the code C _i is in an emotional expression state next to the code string C ₁ ,..., C _i−1 , P _nrm ( _{C i | C 1 ... C i} -1) , as well as the code string _{C 1,} ..., is the conditional probability that next to the code _{C i} of _{C i-1} is the calm state. In addition, P _emo (C _i ) counts the total number of codes C _i corresponding to the speech feature vector existing in the part in which the speech is labeled as the emotional expression state in the process of creating the codebook. It is a value obtained by dividing the total number by the total number of codes (= number of frames) of the voice data labeled as the emotional expression state. On the other hand, P _nrm (C _i ) is a value obtained by dividing the number of codes C _i existing in the portion labeled as calm and _divided by the total number of codes of the voice data labeled as calm.

In the following, each conditional probability is approximated by an N-gram (N <i) model to simplify the calculation of the emotional expression state likelihood and the calm state likelihood. The N-gram model is a model that approximates that the appearance of a certain event at a certain time depends on the appearance of N−1 events immediately before that. Here, when N = 3, it is called trigram, when N = 2, it is called bigram, and when N = 1, it is called unigram. In this model, for example, the probability P (C _i ) that the code C _i appears in the i-th frame is P (C _i ) = P (C _i | C _{i−N + 1} ... C _i−1 ).

Each conditional appearance probability P _emo (C _i | C _i ... C _i-1 ), P _nrm (C _i | C _i ... C _i-1 ) in the above formulas (3) and (4) When the gram model is applied, each conditional appearance probability is approximated as shown in the following equations (5) and (6).

P _emo (C _i | C _i ... C _i−1 ) = P _emo (C _i | C _{i−N + 1} ... C _i−1 ) (5)
P _nrm (C _i | C _i ... C _i−1 ) = P _nrm (C _i | C _{i−N + 1} ... C _i−1 ) (6)

P _emo (C _i | C _{i−N + 1} ... C _i−1 ) in the above equation (5) and P _nrm (C _i | C _{i−N + 1} ... C _i−1 ) in the equation (6) are usually signed. All can be obtained from the book, but there are some that cannot be obtained from the learning speech. In that case, it may be obtained by interpolation from other conditional appearance probabilities or single appearance probabilities. For example, it is possible to interpolate a high-order (that is, code string is long) conditional appearance probability from a low-order (that is, code string is short) conditional appearance probability and a single appearance probability.

The interpolation method will be described below. In the following, the above-described trigram (N = 3), bigram (N = 2), and unigram (N = 1) will be described as examples. Each occurrence probability is trigram (N = 3), P _emo (C _i | C _i-2 C _i-1 ), Pnrm (C _i | C _i-2 C _i-1 ), bigram (N = 2) Is represented as P _emo (C _i | C _i-1 ), Pnrm (C _i | C _i-1 ), and unigram (N = 1) as P _emo (C _i ) and Pnrm (C _i ). The

In this interpolation method, P _emo (C _i | C _i−2 C _i−1 ) and P _nrm (C _i | C _i−2 C _i−1 ) are expressed as three occurrences in the emotional expression state described above. Using the probability or the three appearance probabilities in a calm state, the calculation is performed based on the following equations (7) and (8).

Here, the above-described _λ emo1, λ emo2, λ emo3 is the number of frames of the learning data expressed emotion state and labeling trigram is n, when the code _C 1 in _series, C 2, ..., is _{C n} When obtained, it is expressed as follows.

However, the audio data for _obtaining λ _emo1 , λ _emo2 , and λ _emo3 is _assumed to be other than the audio data for creating the codebook. This is because if the same voice data as that used when creating the codebook is used, a trivial solution of [lambda] _emo1 = 1 and [lambda] _emo2 = [lambda] _emo3 = 0 is obtained. Similarly, λ _nrm1 , λ _nrm2 , and λ _nrm3 are also obtained.

Next, using trigram, when the number of frames in the label section A is F _A and the obtained codes are C ₁ , C ₂ ,..., C _FA , the emotional expression state likelihood P _{Aemo of} this label section A The calm state likelihood P _Anrm is expressed as shown in the following equations (9) and (10), respectively.

P _Aemo = P _emo (C ₃ | C ₁ C ₂ )... P _emo (C _FA | C _FA-2 C _FA-1 ) (9)
P _Anrm = P _nrm (C ₃ | C ₁ C ₂ )... P _nrm (C _FA | C _FA-2 C _FA-1 ) (10)

In the embodiment of the present invention, in the above example, trigram (N = 3), so that interpolation and calculation of emotion expression state likelihood P _Aemo and calm state likelihood P _Anrm can be performed as described above. bigram (N = 2) and unigram (N = 1) are calculated for each code and stored in the codebook. That is, the codebook stores a set of the speech feature vector, the appearance probability in the emotional expression state, and the appearance probability in the calm state corresponding to each code.

  Appearance probabilities in the emotional expression state include the probability that each code appears in the emotional expression state independently of the code that appeared in the past frame (single appearance probability), and can take the predetermined number of consecutive frames immediately before. Next to the sequence of codes, use the conditional probability that the code appears in the emotional expression state, or both. Similarly, the appearance probability in a calm state, the single appearance probability that the code appears in a calm state regardless of the code that appeared in the past frame, the sequence of codes that can be taken by the immediately preceding predetermined number of frames, Use the conditional appearance probability that the code appears in a calm state, or both.

FIG. 9 shows an example of contents recorded in the code book. In the creation of each of the following codebooks, the total number of frames in the calm state used from the learning speech and the total number of frames in the expression state of the corresponding emotion (for example, laughter) are selected equally. In this example, a code book CB-1 created by analyzing the laughing label section and the calm label section in the learning speech, and a code book CB-2 created by analyzing the anger label section and the calm label section. 2 shows a codebook CB-3 created by analyzing the sad label section and the calm label section. As shown in FIG. 9, in the codebook, for each code C ₁ , C ₂ ,..., The speech feature vector and its single appearance probability are stored for the emotional expression state and the calm state, and the conditional appearance Probabilities are stored in pairs for the emotional expression state and the calm state. Here, the codes C ₁ , C ₂ , C ₃ ,... Represent codes (indexes) corresponding to the speech feature vectors of the codebook, and m-bit values “00... 00” and “00... 01”, respectively. , “00... 10”,.

It represents a h-th code in the code book in C _h, for example C ₁ denote the 1st code. In the following, an emotion table when parameters f ₀ ″, p ″, d _p are used as an example of a set of speech feature values suitable for the present invention and the codebook size (number of speech feature value vectors) is ^26. An example in which the conditional appearance probability in the outgoing state and the calm state is approximated by trigram will be described.

FIG. 10 is a schematic diagram for explaining audio data processing. Of the audio sub-paragraphs starting from time t, the first to fourth frames are denoted by reference signs i to i + 3. The frame length and frame shift were set to 100 ms and 50 ms, respectively, as described above. Here, code code _{C 1} for the frame of the frame number i (time t~t + 100) is, for a frame of the frame number i + 1 (time t + 50~t + 150) code _{C 2} is the frame of the frame number i + 2 (time t + 100~t + 200) C ₃ is then assumed that the code _{C 4} is obtained for the frame of the frame number i + 1 (time t + 50~t + 150). That is, it is assumed that the codes are C ₁ , C ₂ , C ₃ , and C ₄ in the frame order.

In this case, the trigram can be calculated in the frame having the frame number i + 2 or more. _Assuming that the emotion expression state likelihood of the speech sub-paragraph S is P _Semo and the calm state likelihood is P _Snrm , the respective likelihoods up to the fourth frame are expressed by the following equations (11) and (12), respectively. Given.

P _Semo = P _emo (C ₃ | C ₁ C ₂ ) P _emo (C ₄ | C ₂ C ₃ ) (11)
P _Snrm = P _nrm (C ₃ | C ₁ C ₂ ) P _nrm (C ₄ | C ₂ C ₃ ) (12)

In this example, the individual appearance probabilities of the emotion expression state and the calm state of the codes C ₃ and C ₄ are obtained from the code book, and the condition that the code C ₃ appears in the emotion expression state and the calm state next to the code C ₂ seeking with probability, further next emotional expression status code C ₂ C ₃ which appeared in emotional expression state and undisturbed state to the next code C ₁ C ₂ code C ₃ is continuous, the code C ₄ are continuous And the conditional probability of appearing in a calm state is as follows.

P _emo (C ₃ | C ₁ C ₂ ) = λ _emo1 P _emo (C ₃ | C ₁ C ₂ ) + λ _emo2 P _emo (C ₃ | C ₂ ) + λ _emo3 P _emo (C ₃ ) (13)
P _emo (C ₄ | C ₂ C ₃ ) = λ _emo1 P _emo (C ₄ | C ₂ C ₃ ) + λ _emo2 P _emo (C ₄ | C ₃ ) + λ _emo3 P _emo (C ₄ ) (14)
P _nrm (C ₃ | C ₁ C ₂ ) = λ _nrm1 P _nrm (C ₃ | C ₁ C ₂ ) + λ _nrm2 P _nrm (C ₃ | C ₂ ) + λ _nrm3 P _emo (C ₃ ) (15)
P _nrm (C ₄ | C ₂ C ₃ ) = λ _nrm1 P _nrm (C ₄ | C ₂ C ₃ ) + λ _nrm2 P _nrm (C ₄ | C ₃ ) + λ _nrm3 P _nrm (C ₄ ) (16)

By using the above equations (13) to (16), the emotion expression state likelihood P _Semo and the calm state likelihood up to the third frame represented by equations (11) and (12) are obtained as P _Snrm. . Here, the conditional application probabilities P _emo (C ₃ | C ₁ C ₂ ) and P _nrm (C ₃ | C ₁ C ₂ ) can be calculated from frame number i + 2.

The above explanation is about the calculation up to the fourth frame i + 3, but the same applies to the audio sub-paragraph S having the number of frames F _S. For example, when the code obtained from each frame of the audio sub-paragraph S with the number of frames F _S is C ₁ , C ₂ ,..., C _FA , the likelihood P _Aemo that the audio sub-paragraph S is in the emotional expression state Likelihood P _Anrm is calculated as shown in the following equations (17) and (18).

If the likelihood calculated as described above is P _Aemo > P _Anrm , it is determined that the utterance state of the audio sub-paragraph S is an emotional expression state. On the other _hand , if P _Aemo ≦ P _Anrm , the state is substantially determined to be calm. Similarly, P _Aemo / P _Anrm > 1 may be set as a condition for determining an emotional expression state. Also, on condition that W ^L P _Aemo / P _Anrm is satisfied with _respect to the positive weighting factor W, or R _E = (log P _Aemo −log P _Anrm ) / L> W (19)
If the condition is satisfied, the influence of weighting may be increased or decreased according to the number of frames L of the small paragraph. Here, L may be, for example, L = FA−2.

  In each emotion expression state determination method of “laughter”, “anger”, and “sadness”, the voice feature amount used is the same as in the above-described method. It is preferable to include at least one or more of the time-varying characteristics of power and dynamic features and / or at least one or more of these inter-frame differences. The appearance probability may be a single appearance probability or a combination of this and a conditional appearance probability, and when this combination is used, it is preferable to use a linear interpolation method for calculating the conditional appearance probability. Also in this emotion expression state determination method, each voice feature is standardized or standardized for each sub-speech or longer appropriate section, or by the average value of each voice feature of the entire voice signal. Thus, it is preferable to form a set of audio feature values for each frame and perform processing after vector quantization.

As the emotion expression state determination method, for example, the likelihoods P _Aau , P _Aang , and P _Asad for “laughter”, “anger”, and “sadness” with respect to the audio sub-paragraph are expressed by the following equations as in equation (17):

In this case, for example, when it is determined whether it is “laughter” or “calm”, the condition (a1) is _calculated from the laughing expression likelihood P _Alau and the generation state likelihood P _Anrm as described above. ) P _Alau > P _Anrm ,
( _B1 ) W ^L P _Aau > P _Anrm ,
( _C1 ) R _L = (logP _Aau −logP _Anrm ) / L> W,
Any one of the above conditions is used, and if the condition is satisfied, it is determined that the state of laughter is expressed. To determine whether it is “anger” or “calm”, the likelihood P _Aang is calculated using equation (21),

(A2) P _Aang > P _Anrm ,
(B2) W ^L P _Aang > P _Anrm ,
(C2) R _A = (log P _Aang −log P _Anrm ) / L> W,
Any one of the above conditions is used, and if the condition is satisfied, it is determined that the state of anger is expressed. Similarly, to determine whether it is “sadness” or “calm”, the likelihood P _Asad is calculated using equation (22),

( _A3 ) P _Asad > P _Anrm ,
( _B3 ) W ^L P _Asad > P _Anrm ,
(C3) R _S = (logP _Asad −logP _Anrm ) / L> W,
Any one of the predetermined conditions may be used to determine whether the condition is satisfied. Various other judgment conditions can be easily considered.

When determining whether the expression of emotion is “laughter”, “anger”, or “sadness”, for example, the ratio of likelihood of laughter according to the above conditional expressions (c1), (c2), and (c3) R _L , anger likelihood ratio R _A , and sadness likelihood ratio R _S can be calculated and determined by comparing these likelihood ratios.

  According to the principle of the present invention, as described above, at least one of the fundamental frequency, the power, and the time change characteristic of the dynamic feature quantity and / or at least one of the inter-frame differences is used as the voice feature quantity. It is sufficient to use at least two, but it is preferable to include a time change characteristic of the dynamic feature amount among these voice feature amounts. Furthermore, the accuracy of emotion detection can be improved by using at least the fundamental frequency, the power, the time change characteristic of the dynamic feature quantity, or the difference between frames as the voice feature quantity. At least the fundamental frequency, power, and time change characteristics of the dynamic feature amount are particularly preferable as practical feature amounts.

  The creation of the code book used in the emotion expression detection method according to the present invention and the detection of the emotion expression using the code book have been described above in detail. In the following, an embodiment is described in which the present invention is used to extract a desired voice expression of desired emotion expression, here, laughter, anger, and sadness.

First Embodiment In this embodiment, three emotions, “laughter”, “anger”, and “sadness” are not distinguished, and any emotional expression is detected as “emotion”.

  Labels all “feelings” without distinguishing between “Laughter”, “Anger”, and “Sad” in the learning speech, and all other intervals are “Silence” As shown in FIG. 11, one codebook is created.

FIG. 12 shows an emotion expression section detection processing procedure according to the first embodiment.
Step S1: A predetermined determination section S is fetched from the audio data of the input content. The determination section may be the above-described audio sub-paragraph, or may be a fixed-length audio section including at least one predetermined frame.

Step S2: Analyzing the fetched determination section to obtain a speech feature vector for each frame, and referring to the code book of FIG. 11, for example, calmly by the equations (17), (18) or (19), (20) The state likelihood P _Anrm and the emotion expression state likelihood P _Aemo are calculated.
Step S3: It is determined whether there is a remaining determination section. If there is, the process returns to Step S1, and the same process is performed for the next determination section.

Step S4: When the emotion expression state likelihood P _Aemo and the calm state likelihood P _Anrm are obtained as conceptually shown in FIG. 13 for all the determination sections, the section S ′ satisfying W ^L P _Aemo > P _Anrm is determined. The position of each detection section S ′ (for example, the start and end frame numbers of the detection section, or the start time and end time of the detection section from the beginning of the content) is stored in the storage means. W is a positive constant determined in advance, and L is the number of frames for each section S. Although W ^L P _Aemo and P _Anrm are shown as continuous curves in FIG. 13, they are actually discontinuous curves for each determination section S.

Step S5: A section corresponding to the position of the section S ′ detected in step S4 is extracted from the content as an emotion expression section.

Second Embodiment In this embodiment, the emotion expression section S ′ detected in the first embodiment is further selected from “laughter”, “anger”, and “sadness” in step S5 of FIG. Determine if there is. In the second embodiment, the following code book is created in advance in addition to the emotion expression detection code book of FIG. 11 used in the first embodiment.

  Label “Laughter” in the laughter expression section in the learning voice section labeled “Emotion”, label “Anger” in the anger expression section, and “Sadness” in the grief expression section. 14 is created on the basis of the speech sections labeled “laughter”, “anger”, and “sadness”.

  FIG. 15 shows a processing procedure for detecting emotional expression intervals of “laughter”, “anger”, and “sadness” according to the second embodiment. Steps S1 to S4 are the same as the emotion expression section detection processing according to the first embodiment shown in FIG. 12 using the code book of FIG. 11, and thereby, “laughter”, “anger”, “sadness” An emotion expression section S ′ including any of them is detected. In subsequent steps S5 to S8, it is determined whether the emotion expression section S 'is "laughter", "anger", or "sadness".

Step S5: A series of speech feature quantity vectors in the emotion expression section S 'detected in Step S4 is obtained. In this case, since speech feature vectors for all speech sections have already been obtained in steps S1 to S3, a series of speech feature vectors corresponding to the section S 'may be extracted from the speech feature vectors.

Step S6: Referring to the code book of FIG. 14, the laughter expression likelihood P _Aauu , the angry expression likelihood P _Aang , and the sadness expression likelihood P _Asad of the detected emotion expression section S ′ are calculated.
Step S7: Among these likelihoods P _Aau , P _Aang , and P _Asad , the maximum likelihood is determined, and a mark indicating the emotion of the maximum likelihood, for example, Lau for laughter, Ang for anger, and Sad for sadness Stored in correspondence with the position of the detection section S ′.

Step S8: It is determined whether or not an unprocessed emotion expression detection section S ′ remains, and if it remains, the process returns to step S5, and the same process is executed for the next emotion expression detection section S ′.
Step S9: If the determination of the maximum likelihood has been completed for all the emotional expression detection sections S ′, the mark Lau, Ang, Sad among all the emotional expression detection sections S ′, for example, by the user A section corresponding to the detection section of the designated emotion mark is extracted from the content.

  As described above, according to the second embodiment, if the user designates one or more types of emotional expressions, the part corresponding to the designated emotional expression can be extracted from the content.

Third Embodiment In the second embodiment described above, an emotion expression section is first detected from voice data, and then it is determined whether each emotion expression section is “laughter”, “anger”, or “sadness”. In this third embodiment, any emotional expression of “laughter”, “anger”, and “sadness” is directly detected from the audio data. The code book shown in FIG. 14 is used. FIG. 16 shows an emotion expression section detection processing procedure according to the third embodiment.

Step S1: The determination section S is taken from the input audio content.
Step S2: A series of frame speech feature amount vectors in the determination section S are obtained, and the laughing expression likelihood P _Alau , the anger expression likelihood P _Aang , and the sadness expression likelihood P _Asad are obtained with reference to the code book of FIG. Calculate each.

Step S3: Among these likelihoods P _Aau , P _Aang , P _Asad , the maximum likelihood is determined, and a mark representing the emotion of the maximum likelihood, for example, Lau for laughter, Ang for anger, and Sad for sadness Stored in correspondence with the position of the determination section S.

Step S4: It is determined whether or not an unprocessed determination section S remains. If it remains, the process returns to step S1, and the same process is performed for the next determination section S.
Step S5: If the determination of the maximum likelihood has been completed for all the determination sections S, among the determination sections S, for example, the mark detection section designated by the user among the marks Lau, Ang, and Sad. The corresponding section is extracted from the content.

  As described above, according to the third embodiment, if the user designates one or more types of emotion expressions, a portion corresponding to the designated emotion expressions can be extracted from the content. In the case of the third embodiment, since the codebook in the first embodiment is not used, the calm state likelihood is not used. That is, the detection of the emotional expression according to the present invention does not necessarily require the calculation of the calm state likelihood.

Fourth Embodiment This embodiment also makes it possible to extract any one (one or more) of three types of emotion expressions, such as “laughter”, “anger”, and “sadness”, for example, The following three code books are created in advance (similar to the example of FIG. 9).

(1) “Laughter” is labeled in all the laughing expression sections in the learning voice, and “Silence” is labeled in all the calm state sections to create a laughter detection codebook.
(2) An anger detection codebook is created by labeling “anger” in all the angry expression sections in the learning voice and labeling “seduce” in all the calm state sections.
(3) A sadness detection codebook is created by labeling “sadness” in all the sadness expression intervals in the learning speech and labeling “seduce” in all the calm state intervals.

FIG. 17 shows a processing procedure of the fourth embodiment. Also in this embodiment, any one or more of the three types of emotion expressions can be detected.
Step S1: The determination section S is fetched from the audio data of the content. The determination section S may be an audio sub-paragraph as described above, or may be a predetermined length section.

Step S2: Analyzing the determination section S to obtain a speech feature amount for each frame, referring to the laughter detection codebook to obtain a laughing expression likelihood P _Alau and a calm state likelihood P _{Anrm corresponding thereto} , and laughing likelihood degrees ratio _{R L = (logP Alau -logP Anrm} ) / L
Calculate The anger expression likelihood P _Aang and the calm state likelihood PAAnrm are obtained with reference to the anger detection codebook, and the anger likelihood ratio R _A = (log P _Aang −log P _Anrm ) / L
Calculate Moreover, it obtains a sadness exposed likelihood _{P Asad} and normal state likelihood PAnrm thereto with reference to the codebook for the sorrow detection, sadness likelihood ratio _{R S = (logP Asad -logP Anrm} ) / L
Calculate The calculated likelihood ratios R _L , R _A and R _S are stored.

Step S3: It is determined whether there is a remaining determination section S. If there is, the process returns to Step S1 and the same process is executed for the next determination section S. If all the sections of the audio data have been completed, any one or more of “laughter”, “anger”, and “sadness” designated by the user is designated among the following steps S4, S5, and S6. Execute what corresponds to the emotion.

Steps S4, S5, and S6: As shown conceptually in FIG. 18 by the processing of steps S1, S2, and S3, for example, the ordinate represents the likelihood ratio R, the laughing expression likelihood ratio R _L , and the anger expression likelihood ratio. Curves of R _A and sadness expression likelihood ratio R _S are respectively obtained, and these are compared with a predetermined threshold value Rth to detect sections larger than Rth, and their positions and emotion marks Lau, Ang , Sad are stored in correspondence.

Step S7: Among the “laughter”, “anger”, and “sadness”, the detection section of the one designated by the user is extracted from the content.
As described above, also in the third embodiment, it is possible to select any emotion expression of “laughter”, “anger”, and “sadness” and extract it from the content.

Fifth Embodiment This embodiment is a modification of the fourth embodiment. In the fourth embodiment, each emotion expression state likelihood ratio is compared with a certain threshold value Rth in order to detect an emotion expression section, but here each emotion expression state likelihood is compared with a common calm state likelihood. Then, each emotion expression section is detected. For that purpose, “Laughter”, “Anger”, and “Sadness” are labeled in the laughter expression section, anger expression section, and grief expression section in the learning voice, respectively, and “Silence” is indicated in the section where the voice is calm The code book shown in FIG. 19 is created by labeling. As shown in FIG. 19, the single appearance probability (unigram) and conditional appearance probability (bigram, trigram) of the code in each expression of laughter, anger, sadness, and calm are obtained and written in the codebook. It is.

FIG. 20 shows a processing procedure of the fifth embodiment.
Step S1: The determination section S is fetched from the audio data of the content.
Step S2: Analyzing the determination section S to obtain a voice feature amount for each frame, referring to the code book of FIG. 19, the laughing expression likelihood P _Aau , the anger expression likelihood P _Aang , and the sadness expression likelihood P _Asad , calm state likelihood P _Anrm is calculated and stored.

Step S3: It is determined whether there is a remaining determination section. If there is, the process returns to Step S1, and the same process is executed for the next determination section. If there is no remaining determination section, one corresponding to steps S4, S5, and S6 is executed for one or more designated by the user among “laughter”, “anger”, and “sadness”.

Steps S4, S5, and S6: At the stage where the processes of Steps S1, S2, and S3 are completed, as shown conceptually in FIG. 21, for example, the laughing expression likelihood P _Aau , the anger expression likelihood P _Aang , and the sadness expression _Curves of likelihood P _Asad and normal state likelihood P _Anrm are obtained. However, the emotional expression likelihood _P AlAu interval of the number of frames in FIG. 21 _FA, P _Aang, shows a curve obtained by multiplying the weight ^{W L} to _{P Asad.} These likelihood curves W ^L P _Aau , W ^L P _Aang , W ^L P _Asad and the curve P _Anrm are compared, and W ^L P _Aau > P _Anrm , W ^L P _Aang > P _Anrm , W ^L P _Asad > P _Anrm The section of the largest one of P _Alau , P _Aang , and P _Asad is detected, and the position of the detected section and the mark are stored in association with each other.
Step S7: Of the “laughter”, “anger”, and “sadness”, an interval corresponding to the emotion detection interval designated by the user is extracted from the audio content.

Sixth Embodiment In this embodiment, labels corresponding to the “laughter”, “anger”, and “sadness” speech segments in the learning speech are assigned in advance, and the “laughter” and “anger” speech segments are labeled. The appearance probability of each quantized speech feature vector for laughter expression and the appearance probability of the quantized speech feature vector for anger expression are obtained from the speech feature spectrum of all frames, and the codebook shown in FIG. CB-1 is created, and similarly, each occurrence probability of each quantized speech feature vector for anger expression and sadness from speech feature vectors of all frames of the speech segment of “anger” and “sadness” The respective appearance probabilities of the quantized speech feature vectors for the expression are obtained, and the code book CB-2 shown in FIG. 22 is created, and the speech feature values of all frames of the speech section of “sadness” and the speech section of “laughter” vector 22 to determine the appearance probability of each quantized speech feature quantity vector for sadness expression and the appearance probability of the quantized speech feature quantity vector for laughter expression, and create codebook CB-3 shown in FIG. deep.

FIG. 23 shows an emotion expression detection process procedure according to the sixth embodiment.
Steps S1 to S4 are the same as the processing procedure in the case of not distinguishing each emotion in FIG. 12, and the emotion expression state likelihood W ^L P _Aemo and the calm state likelihood obtained for all speech sections using the codebook of FIG. From the curve of degree P _Anrm , all the sections where W ^L P _Aemo > P _Anrm are detected as emotion expression sections S ′ and temporarily stored.

Step S5: The emotion expression section S ′ is captured.
Step S6: The laughter expression likelihood P _Alau1 and the anger expression likelihood P _Aang2 are obtained from the series of speech feature amount spectra in the emotion expression section S ′ with reference to the code book CB-1 in FIG. 22, and the code book CB. -2 is obtained, the anger expression likelihood P _Aang1 and the sadness expression likelihood P _Asad2 are obtained, and the sadness expression likelihood P _Asad1 and the laughter expression likelihood P _Alau3 are obtained by referring to the code book CB-3. .
Step S7: Two likelihoods for laughter, anger, and sadness are determined from the likelihood as follows.

Laughter likelihood: P _LAU1 = P _Alau1 / P _Aang2 ; P _LAU2 = P _Alau2 / P _Asad1
Anger likelihood: P _ANG1 = P _Aang1 / P _Asad2 ; P _ANG2 = P _Aang2 / P _Alau1
Sadness likelihood: P _SAD1 = P _Asad1 / P _Alau2 ; P _SAD2 = P _Asad2 / P _Aang1

Step S8: Degree of laughter, anger, and sadness are determined as follows.
Laughter level: LAU = (PLAU1 + PLAU2) / 2
Angry degree: ANG = (PANG1 + PANG2) / 2
Sadness: SAD = (PSAD1 + PSAD2) / 2

Step S9: As shown in FIG.
The section of LAU> ANG and LAU> SAD is detected and marked with Lau.
The section of ANG> SAD and ANG> LAU is detected, and Ang is marked.
A section of SAD> LAU and SAD> ANG is detected, and a Sad mark is attached.

Step S10: It is determined whether or not the processing has been completed for all the detection sections S ′. If not, the process returns to Step S5 and the same processing is performed in Steps S6 to S9 for the next emotion expression detection detection section S ′.
Step S11: If all the detection sections S ′ have been completed, the section of the emotion mark designated by the user is extracted from the audio content. Alternatively, it is possible to extract a section _{equal to} or greater than the threshold value _Rth that satisfies the requirement that the user wants to view the summary for a specified length of time desired by the user, or to view only a smiling place (see the broken line in FIG. 22).

Each of the emotion expression state likelihoods P _Aau , P _Ang , and P _Asad in the first to sixth embodiments described above may be calculated using any of the above formulas (17) or (19).

  As described above, the speech synthesizer and the speech synthesis program according to the embodiment of the present invention change the synthesized speech information corresponding to the phrase obtained from the speech of the speaker according to the emotion of the speaker. When there is an error in the inflection or prosody of the synthesized speech, or when it is desired to adjust the inflection or prosody, the synthesized speech can be easily corrected.

  In addition, since the phrase obtained from the speech of the speaker is translated into another language, the synthesized speech of the speech synthesizer used as a translator can be easily corrected.

  Also, quantize the set of speech features of the input speech, obtain the probability that the corresponding speech feature vector in the codebook will appear in the emotional expression state from the codebook, and determine whether the emotional expression is based on this appearance probability Therefore, the emotion of the speaker can be specified without depending on the voice quality of the speaker, and the synthesized speech can be easily corrected.

  Further, in the embodiment of the present invention, the voice processing operation has been described in which the voice processing apparatus performs the processes in the above steps S310 to S340. However, in order to execute the voice processing operation including these steps S310 to S340. It is also possible to implement using a predetermined computer in which the voice processing program is installed.

  The speech synthesizer and the speech synthesis program according to the present invention are capable of easily correcting synthesized speech when there is an error in the inflection or prosody of the synthesized speech or when it is desired to adjust the inflection or prosody. And can be used for applications such as personal computers or terminals installed in public facilities.

1 is a block diagram showing a configuration of a speech synthesizer according to an embodiment of the present invention. 1 is a schematic diagram showing a hardware configuration of a speech synthesizer according to an embodiment of the present invention. The block diagram which shows the structure of the emotion estimation means which concerns on embodiment of this invention. The flowchart which shows the flow of the operation | movement which the speech synthesizer which concerns on embodiment of this invention changes synthetic | combination speech information, and the output. The flowchart for demonstrating operation | movement of the emotion estimation means 130 which concerns on embodiment of this invention. The flowchart for demonstrating the detail of the process in step S330. The conceptual diagram for demonstrating an audio | voice small paragraph, an audio | voice paragraph, etc. The flowchart for demonstrating the detail of the process in step S310. The figure which shows the example of a code book recording. The schematic diagram for demonstrating the process of audio | voice data. The figure which shows the example of the code book used for 1st Embodiment. The flowchart which shows the process sequence of 1st Embodiment. The conceptual diagram for demonstrating the detection of the emotion expression area by the comparison of likelihood. The figure which shows the example of the codebook used by 1st Embodiment. The flowchart which shows the process sequence of 2nd Embodiment. The flowchart which shows the process sequence of 3rd Embodiment. The flowchart which shows the process sequence of 4th Embodiment. The conceptual diagram for demonstrating the detection of the emotion expression area based on likelihood ratio. The figure which shows the example of the codebook used by 4th Embodiment. The flowchart which shows the process sequence of 5th Embodiment. The conceptual diagram for demonstrating the detection of the emotion expression area based on likelihood comparison. The figure which shows the example of the codebook used by 6th Embodiment. The flowchart which shows the process sequence of 6th Embodiment. The conceptual diagram for demonstrating the emotional expression by comparison, such as laughter, anger, and sadness.

Explanation of symbols

100 speech synthesizer 101 CPU
102 ROM
103 RAM
104 EEPROM
DESCRIPTION OF SYMBOLS 105 Hard disk 106 Interface part 107 Display 108 Speaker 110 Audio | voice signal receiving means 120 Phrase conversion means 130 Emotion estimation means 131 Storage means 132 Speech feature-value extraction means 133 Emotion expression likelihood calculation means 134 Quiet state likelihood calculation means 135 Emotion expression Determining means 140 Synthetic speech changing means 150 Speech synthesizing means 160 Translation means

Claims (4)

Voice signal receiving means for receiving a voice signal obtained from the voice of the speaker;
A phrase conversion means for converting the voice signal received by the voice signal receiving means into a phrase;
Emotion estimating means for estimating the emotion of the speaker based on the audio signal received by the audio signal receiving means;
A speech synthesizer comprising: synthesized speech changing means for changing synthesized speech information corresponding to the word or phrase according to the emotion estimated by the emotion estimating means. The speech synthesizer
A translation means for translating the phrase converted by the phrase conversion means into a phrase of another language;
2. The speech synthesizer according to claim 1, wherein the synthesized speech changing unit changes the synthesized speech information corresponding to the words translated by the translating unit according to the emotion estimated by the emotion estimating unit. . On the computer,
A voice signal receiving step for receiving a voice signal obtained from the voice of the speaker;
A phrase conversion step of converting the voice signal received in the voice signal reception step into a phrase;
An emotion estimation step of estimating the emotion of the speaker based on the audio signal received in the audio signal reception step;
A speech synthesis program, comprising: executing a synthesized speech changing step of modifying synthesized speech information corresponding to the phrase according to the emotion estimated in the emotion estimation step. A translation means step for translating the phrase converted in the phrase conversion step into a phrase of another language;
4. The speech synthesis program according to claim 3, wherein the step of changing the synthesized speech information corresponding to the phrase translated in the translation unit step is executed according to the emotion estimated in the emotion estimation step. 5.

Images