WO2025158547A1 - Learning device, inference device, learning method, inference method, and program - Google Patents

Abstract

This learning device comprises: an input unit for inputting learning data including speech, a first sentence regarding the speech, and a second sentence corresponding to the first sentence; a speech feature generation unit that generates information representing features of the speech for each prescribed time interval, on the basis of a speech feature extractor composed of a plurality of layers; a first integration unit that, on the basis of a first parameter, generates first integrated information for each time interval by integrating information representing features generated individually in a prescribed plurality of layers of the speech feature extractor; a second integration unit that, on the basis of a second parameter, generates second integrated information by integrating the first integrated information for each time interval in the time direction; a calculation unit that calculates the generation probability of a third sentence corresponding to the first sentence on the basis of the first sentence, the second integrated information, and a language model; and a learning unit that learns parameters to be learned, including the first parameter and the second parameter, on the basis of the generation probability of the third sentence, and the second sentence.

Description

Learning device, inference device, learning method, inference method, and program

This disclosure relates to a learning device, an inference device, a learning method, an inference method, and a program.

It is known that speech contains three types of information: linguistic information, non-linguistic information, and paralinguistic information (hereinafter, these three types of information will be collectively referred to as "speech information") (Non-Patent Document 1). Furthermore, technology for recognizing non-linguistic and paralinguistic information from speech is known (Non-Patent Document 2).

On the other hand, in the field of image processing, there is a technology known as image understanding technology that outputs various information contained in images in natural language (Non-Patent Document 3).

H. Fujisaki, “Prosody, Models, and Spontaneous Speech,” in Computing Prosody, Y. Sagisaka, N. Campbell, and N. Higuchi, Springer, pp.27-42, 1996. A. Ando, S. Kobashikawa, H. Kamiyama, R. Masumura, Y. Ijima and Y. Aono, "SOFT-TARGET TRAINING WITH AMBIGUOU S EMOTIONAL UTTERANCESFOR DNN-BASED SPEECH EMOTION CLASSIFICATION," in Proc. of ICASSP, 2018, pp. 4964-4968. D. Zhu, J. Chen, X. Shen, X. Li, M. Elhoseiny, "MINIGPT-4: ENHANCED VISION-LANGUAGE UNDERSTANDING WITH ADVANCED LARGE LANGUAGE MODELS," in arXiv preprint arXiv:2304.10592, 2023.

By using a speech encoder instead of the image encoder used in image understanding technology, it is thought possible to realize a technology that outputs speech information contained in speech in natural language (hereinafter also referred to as "speech understanding technology"). However, due to differences in the configurations of image and speech encoders and differences in the properties of images and speech, it is difficult to realize speech understanding technology simply by using a speech encoder instead of an image encoder.

This disclosure has been made in light of the above points and aims to realize speech understanding technology.

A learning device according to one aspect of the present disclosure includes an input unit that inputs learning data including speech, a first sentence related to the speech, and a second sentence corresponding to the first sentence; a speech feature generation unit that generates information representing features of the speech for each predetermined time interval based on a speech feature extractor configured with multiple layers; a first integration unit that generates first integrated information for each time interval based on first parameters, integrating the information representing the features generated by each of the predetermined multiple layers of the speech feature extractor; a second integration unit that generates second integrated information for each time interval based on second parameters, integrating the first integrated information in the time direction; a calculation unit that calculates the generation probability of a third sentence corresponding to the first sentence based on the first sentence, the second integrated information, and a language model; and a learning unit that learns learning target parameters including the first parameter and the second parameter based on the generation probability of the third sentence and the second sentence.

Speech understanding technology can be realized.

FIG. 2 is a diagram illustrating an example of a speech understanding model. FIG. 10 illustrates an example of an audio encoder and an audio encoder output integrated block. FIG. 10 is a diagram illustrating an example of a time information integration block. FIG. 2 is a diagram illustrating an example of a hardware configuration of a speech understanding device during learning. FIG. 2 is a diagram illustrating an example of a functional configuration of a speech understanding device during learning. FIG. 10 is a diagram illustrating an example of a training data set. FIG. 2 is a diagram illustrating an example of a detailed functional configuration of a model learning unit. 10 is a flowchart illustrating an example of a model construction process. 10 is a flowchart illustrating an example of a model learning process. FIG. 2 is a diagram illustrating an example of a functional configuration of a speech understanding device during inference. FIG. 2 is a diagram illustrating an example of a detailed functional configuration of an output statement generation unit. 10 is a flowchart illustrating an example of an output sentence generation process.

Below, one embodiment of the present invention will be described in detail with reference to the drawings.

<Non-verbal and para-linguistic information recognition technology>
It is known that speech contains speech information (i.e., linguistic information, non-linguistic information, and paralinguistic information) (Non-Patent Document 1). Here, linguistic information refers to information about the words spoken by a speaker. Non-linguistic information refers to information that is not linguistic but cannot be changed at will (e.g., information that represents the speaker's identity, gender, emotions, etc.). Paralinguistic information refers to information that is not linguistic but can be changed at will (e.g., information that represents intentions, attitudes, etc.).

In conventional technologies for recognizing non-verbal and paralinguistic information from speech, a finite number of states are defined in advance, and then an estimation is made of which of these states the speech most closely matches. For example, the technology described in Non-Patent Document 2 uses a statistical model based on deep learning to estimate which emotional state, such as anger, joy, or sadness, the speech most closely matches. However, such conventional technologies are unable to recognize fine-grained non-verbal and paralinguistic information. For example, they are unable to estimate emotional states such as "irritated," which are not predefined, or "angry and sad," which spans multiple emotional states. This results in a problem of reduced processing accuracy in downstream systems that use the results of non-verbal and paralinguistic information recognition (for example, the accuracy of call analysis in contact center systems, the accuracy of dialogue control and analysis in voice dialogue systems, etc.).

<Image understanding technology>
In the field of image processing, there is known a technology called image understanding, which outputs various information contained in an image in natural language (Non-Patent Document 3). Note that natural language refers to a sentence written in a natural language (e.g., a language used by humans for communication, such as Japanese, English, or Chinese). Image understanding technology is composed of a deep learning model that combines a large-scale language model that acquires relationships and co-occurrences between words using a large amount of text data with an image encoder that extracts information about objects in the image from the image. When this deep learning model is given an input image and a natural language question about the input image (e.g., a sentence such as "What do you think about the logo in this image?"), it outputs an output sentence corresponding to the question (e.g., a sentence such as "This log is a simple and symbolic logo").

Deep learning models that realize image understanding technology can infer various pieces of information contained in an image in natural language by providing a pair of an input image, a natural language question about that input image, and a correct output sentence corresponding to that question. For example, in response to the question, "What color is the logo in this image?", a sentence such as "It's pink" can be output as the output sentence.

<Speech understanding technology>
By realizing speech understanding technology that can output speech information contained in speech in natural sentences, it will be possible to recognize a variety of speech information, including detailed non-linguistic and paralinguistic information, and output that speech information in natural sentences.

A simple way to realize speech understanding technology would be to use a speech encoder that extracts information representing the characteristics of speech from speech, instead of the image encoder used in image understanding technology. However, in practice, it is difficult to realize speech understanding technology using this method. There are two reasons for this.

The first reason is the difference in the structure of image encoders and speech encoders. Existing speech encoders (e.g., wav2vec2.0 (Reference 1), WavLM (Reference 2), etc.) extract different information at each layer, making it impossible to understand diverse speech information using only the information ultimately output by the speech encoder, as is the case with image understanding technology. For example, Reference 2 suggests that information closer to physical properties, such as speaker information, is extracted in the lower layers of the speech encoder, while information closer to abstract properties, such as phonology, is extracted in the higher layers of the speech encoder. For this reason, for example, while speaking style recognition uses information extracted in the lower layers of the speech encoder, speech recognition must use information extracted in the higher layers of the speech encoder; otherwise, it is thought that accurate natural-sounding output will be difficult.

The second reason is the difference in the nature of images and audio. In other words, unlike images, audio has a variable length, so it is thought that time-domain processing is required to recognize the audio information contained in audio of any length and output that audio information in natural language.

<Speech understanding model>
Therefore, we propose a deep learning model (hereinafter referred to as a "speech understanding model") that can solve the problems caused by the above two causes. This speech understanding model makes it possible to recognize diverse speech information, including detailed non-verbal and paralinguistic information, from speech of any length and output the speech information in natural sentences. In other words, it is possible to realize a speech understanding technology that outputs diverse speech information contained in speech of any length (more specifically, speech information related to at least one of the physical and abstract properties of speech) in natural sentences. This can be expected to improve the processing accuracy of downstream systems that use the recognition results of non-verbal and paralinguistic information (e.g., the accuracy of call analysis in contact center systems, the accuracy of dialogue control and analysis in voice dialogue systems, etc.).

An example of the speech understanding model 1000 proposed in this embodiment will be described with reference to Figure 1. Figure 1 is a diagram showing an example of the speech understanding model 1000.

As shown in Figure 1, the speech understanding model 1000 is composed of a speech encoder 1100, a speech encoder output integration block 1200, a temporal information integration block 1300, a linear transformation layer 1400, and a large-scale language model 1500.

The audio encoder 1100 is any existing audio encoder (for example, wav2vec2.0, WavLM, etc.). The audio encoder 1100 receives audio (hereinafter also referred to as "input audio") as input and outputs information representing the characteristics of that audio. At this time, for each predetermined time interval, the audio encoder 1100 receives the input audio for that time interval and outputs information representing the characteristics of the audio for that time interval.

The audio encoder output merging block 1200 merges the outputs of multiple pre-specified layers from among the outputs of each layer of the audio encoder 1100 in each time interval. Hereinafter, each of the multiple pre-specified layers will be referred to as the "layer to be merged." However, it is assumed that each layer to be merged has the same number of output dimensions. The layer to be merged is specified by the user, etc., from among the layers of the audio encoder 1100 that have the same number of output dimensions.

Hereinafter, an index representing a time interval is defined as t, where t = 1, ..., T. T is an index representing the last time interval of the input speech, and its value may vary depending on the length of the input speech. Furthermore, hereinafter, the number of layers to be integrated is defined as N, and the output of the nth layer to be integrated (where n = 1, ..., N) in time interval t is defined as h _n (t). Note that for each n = 1, ..., N, h _n (t) represents some feature of the input speech (e.g., a physical property or an abstract property), and each h _n (t) can be described by a vector with a predetermined number of dimensions, so hereinafter each h _n (t) will be referred to as a "first speech feature vector."

In this case, the speech encoder output integration block 1200 receives, in each time interval t, each first speech feature vector h _n (t) in that time interval t as input, and outputs a vector obtained by integrating each first speech feature vector h _n (t). Hereinafter, the vector obtained by integrating each first speech feature vector h _n (t) will be referred to as the "first integrated vector" and represented by e(t).

For example, as shown in Figure 2, suppose the speech encoder 1100 is composed of one convolutional layer and N transformer layers, and these N transformer layers are layers to be integrated. In this case, the output of the nth transformer layer in time interval t is a first speech feature vector h _n (t), and the speech encoder output integration block 1200 integrates these first speech feature vectors h _n (t) to create a first integrated vector e(t).

The first speech feature vectors h _n (t) can be integrated by, for example, weighted sum or linear transformation sum. When weighted sum is used, the first integrated vector e(t) is created by e(t) = α ₁ h ₁ (t) + ... + α _N h _N (t). Here, α ₁ , ..., α _N are also called weighting coefficients and are training target parameters that satisfy α ₁ + ... + α _N = 1. On the other hand, when linear transformation sum is used, the first integrated vector e(t) is created by e(t) = ((W ₁ h ₁ (t) + b ₁ ) + ... + (W _N h _N (t) + b _N ))/N. Here, W ₁ , ..., W _N , b ₁ , ..., b _N are also called linear transformation coefficients and are training target parameters.

The temporal information integration block 1300 integrates the first integrated vector e(t) in the time direction. That is, the temporal information integration block 1300 receives the first integrated vector e(t) for each time interval t as input and outputs a vector obtained by integrating each first integrated vector e(t) in the time direction. In this case, it is not sufficient to simply integrate the first integrated vector e(t) for all time intervals t; it is necessary to consider which parts of the time period should be emphasized and which parts should not. For example, when recognizing a speaker's emotions or speaker information from speech, it is necessary to ignore time periods representing short pauses in the speech or time periods where breathing occurs, and to emphasize the time periods during which the speaker is speaking. For this reason, the temporal information integration block 1300 integrates each first integrated vector e(t) using a weighted sum. Hereinafter, the vector obtained by integrating each first integrated vector e(t) in the time direction will be referred to as the "second integrated vector" and represented by v.

For example, as shown in Figure 3, when first integrated vectors e(1), ..., e(T) are input, the time information integration block 1300 creates a second integrated vector v by integrating these first integrated vectors e(1), ..., e(T) in the time direction.

As a method for integrating each of the first integrated vectors e(1), ..., e(T), for example, a self-attentive pooling layer can be used. Let E = [e(1), ..., e(T)] ^τ . In this case, when a self-attention pooling layer is used, the second integrated vector v is created by v = aE. Here, a = softmax(ReLU(W ₁ 'E)W ₂ '). Furthermore, W ₁ ' and W ₂ ' are training parameters, and τ is a symbol representing transposition. As a result, the second integrated vector v is obtained by integrating each of the first integrated vectors e(1), ..., e(T) by weighted sum.

The time information integration block 1300 may integrate each of the first integrated vectors e(1), ..., e(T) in the time direction using a convolutional neural network. In this case, a matrix V = [v(1), ..., v(K)] ^τ composed of K second integrated vectors v(1), ..., v(K) is obtained by V = conv1D(E). In this case, the learnable parameters of the one-dimensional convolutional neural network are the parameters to be learned. Furthermore, K is an integer equal to or greater than 1 determined by the window size of the one-dimensional convolutional neural network and the sequence length T of the first integrated vectors e(1), ..., e(T).

The linear transformation layer 1400 linearly transforms the second integrated vector v. That is, the linear transformation layer 1400 takes the second integrated vector v as input and outputs a vector obtained by linearly transforming this second integrated vector v. Hereinafter, the vector obtained by linearly transforming the second integrated vector v will be referred to as the "second speech feature vector" and represented by w. The second speech feature vector w is created by w = Wv + b. Here, W and b are also called linear transformation coefficients and are training target parameters. Note that if the number of dimensions of the token embedding space in the large-scale language model 1500 is M, then the number of dimensions of the second speech feature vector w is also M. This means that the linear transformation layer 1400 creates a second speech feature vector sequence with a length of 1 and M dimensions.

When the matrix V = [v(1), ..., v(K)] ^τ is output from the time information integration block 1300, for example, K vectors obtained by linearly transforming v(1), ..., v(K) can be set as second speech feature vectors w(1), ..., w(K). This means that a second speech feature vector sequence with a length of K and a number of dimensions of M is created.

Large-scale language model 1500 is any existing large-scale language model (LLM). Large-scale language model 1500 takes as input an input sentence, which is a natural language question about the input speech, and a second speech feature vector w (or second speech feature vector w(1), ..., w(K)), and outputs an output sentence corresponding to the question. Such an output sentence is generated according to the posterior probability given the input sentence and the second speech feature vector w (or second speech feature vector w(1), ..., w(K)). Note that in large-scale language model 1500, the posterior probability is calculated by processing a vector sequence combining an embedding vector sequence representing the embedded representation of the tokens that make up the input sentence and the second speech feature vector w (or second speech feature vector w(1), ..., w(K)).

The example shown in Figure 1 shows a case where, when an input sentence of "Please tell me the emotional state of the person in the following speech," is given, an output sentence of "This person is male and is slightly irritated" is generated. Note that, for example, Llama 2 7B (Reference 3) or the like can be used as the large-scale language model 1500. However, a language model other than a large-scale language model may also be used as the large-scale language model 1500, as long as it is a language model that can generate an output sentence according to the posterior probability when given an input sentence and a second speech feature vector w (or second speech feature vectors w(1), ..., w(K)).

For simplicity, the following description will be given assuming that a self-attention pooling layer is used in the temporal information integration block 1300, that a second integrated vector v is obtained in the temporal information integration block 1300, and that a second audio feature vector w is created in the linear transformation layer 1400. However, if a convolutional neural network is used in the temporal information integration block 1300, the following embodiment can be similarly applied by replacing "second integrated vector v" with "second integrated vector v(1), ..., v(K)" and "second audio feature vector w" with "second audio feature vector w(1), ..., w(K)."

Note that the speech encoder 1100 may be referred to as, for example, a "speech feature extractor" or "speech coder." Furthermore, the large-scale language model 1500 may be referred to as, for example, a "language model," a "natural language model," or a "natural language processing model." Furthermore, the components that make up the speech understanding model 1000 (speech encoder 1100, speech encoder output integration block 1200, temporal information integration block 1300, linear transformation layer 1400, and large-scale language model 1500) may be referred to as, for example, a "module."

The following describes a speech understanding device 10 that realizes speech understanding technology using the speech understanding model 1000 shown in Figure 1. Here, the speech understanding device 10 undergoes a "model construction" phase during which the speech understanding model 1000 is constructed, a "learning" phase during which parameters to be learned for the speech understanding model 1000 are learned, and an "inference" phase during which output sentences are generated by the speech understanding model 1000 using the learned parameters. During learning, the speech understanding device 10 is provided with a collection of training data (hereinafter also referred to as a "training dataset") represented by pairs of input speech, input sentences which are questions in natural language related to the input speech, and correct output sentences corresponding to the questions. On the other hand, during inference, the speech understanding device 10 is provided with test data represented by pairs of input speech and input sentences which are questions in natural language related to the input speech.

Note that the speech understanding device 10 during model construction may be referred to as, for example, a "model construction device" or a "model creation device." The speech understanding device 10 during learning may be referred to as, for example, a "learning device," a "parameter estimation device," a "parameter optimization device," etc. The speech understanding device 10 during inference may be referred to as, for example, an "inference device," an "estimation device," a "natural sentence generation device," etc.

In the following, for simplicity's sake, model construction is considered to be included in learning, and a case will be described in which the speech understanding device 10 also constructs the speech understanding model 1000 during learning.

[Study]
The speech understanding device 10 during learning will be described below.

<Example of hardware configuration of the speech understanding device 10 during learning>
An example of the hardware configuration of the speech understanding device 10 during learning will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the hardware configuration of the speech understanding device 10 during learning.

As shown in FIG. 4, the speech understanding device 10 during learning has an input device 101, a display device 102, an external I/F 103, a communication I/F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other so that they can communicate with each other via a bus 109.

The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the speech understanding device 10 does not necessarily have to have at least one of the input device 101 and the display device 102, for example.

The external I/F 103 is an interface with external devices such as a recording medium 103a. Examples of recording media 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

The communication I/F 104 is an interface for connecting to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a non-volatile storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory. The processor 108 is a variety of arithmetic devices such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit).

Note that the hardware configuration shown in FIG. 4 is an example, and the hardware configuration of the speech understanding device 10 is not limited to this. For example, the speech understanding device 10 may have multiple auxiliary storage devices 107 or multiple processors 108, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

<Example of functional configuration of the speech understanding device 10 during learning>
An example of the functional configuration of the speech understanding device 10 during learning will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the functional configuration of the speech understanding device 10 during learning.

As shown in FIG. 5, the speech understanding device 10 during learning has a model construction unit 201 and a model learning unit 202. These units are realized, for example, by processing in which one or more programs installed in the speech understanding device 10 are executed by the processor 108 or the like. The speech understanding device 10 during learning also has a trained speech encoder storage unit 203, a trained large-scale language model storage unit 204, a speech understanding model storage unit 205, and a training dataset storage unit 206. Each of these storage units is realized, for example, by a storage area such as the auxiliary storage device 107. However, at least one of these storage units may also be realized by a storage area of a storage device (for example, a storage device included in a database server, etc.) that is communicatively connected to the speech understanding device 10.

The model construction unit 201 constructs the speech understanding model 1000 shown in FIG. 1 using the trained speech encoder stored in the trained speech encoder storage unit 203 and the trained large-scale language model stored in the trained large-scale language model storage unit 204. That is, the model construction unit 201 constructs the speech understanding model 1000 shown in FIG. 1 using the trained speech encoder as the speech encoder 1100 and the trained large-scale language model as the large-scale language model 1500. At this time, the model construction unit 201 initializes the training target parameters of the speech encoder output integration block 1200, the training target parameters of the time information integration block 1300, and the training target parameters of the linear transformation layer 1400. The training target parameters may be initialized using any method, such as random initialization or sampling from a predetermined distribution. Note that a trained speech encoder is a speech encoder whose parameters have been trained. Similarly, a trained large-scale language model is a large-scale language model whose parameters have been trained.

In addition, the model construction unit 201 stores the speech understanding model 1000 in the speech understanding model storage unit 205.

The model training unit 202 uses the training dataset stored in the training dataset storage unit 206 to train the speech understanding model 1000 stored in the speech understanding model storage unit 205. At this time, the model training unit 202 trains the training target parameters of the speech encoder output integration block 1200, the training target parameters of the temporal information integration block 1300, and the training target parameters of the linear transformation layer 1400, while keeping the parameters of the speech encoder 1100 and large-scale language model 1500 fixed. More specifically, the model training unit 202 uses, as a loss function, the cross entropy between the output sentence generated by the speech understanding model 1000 when given the input speech and input sentence contained in the training data and the correct output sentence contained in the training data, and trains the training target parameters using an existing optimization method so as to minimize the loss function. A detailed example of the functional configuration of the model training unit 202 will be described later.

The trained speech encoder storage unit 203 stores a trained speech encoder. The trained large-scale language model storage unit 204 stores a trained large-scale language model. The speech understanding model storage unit 205 stores the speech understanding model 1000 constructed by the model construction unit 201. The training dataset storage unit 206 stores a given training dataset.

<Learning dataset>
An example of the training data set stored in the training data set storage unit 206 will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the training data set.

As shown in Figure 6, a training dataset is made up of one or more training data, each of which includes input speech, an input sentence, and a correct output sentence. Generally, a training dataset is made up of a large number of training data.

The input speech is speech data input to the speech understanding model 1000. The input sentence is text data that represents a question in natural language related to the input speech. The correct output sentence is text data that represents a response or answer in natural language that is the correct answer to the question represented by the input sentence. Note that the input speech does not necessarily have to be audio data that records a human voice, but may also be audio data that records any sound. The correct output sentence may also be called, for example, "teaching data."

6 includes the input speech "Speech A," the input sentence "Please transcribe this speech," and the correct output sentence "Please explain what is going on." Similarly, the training data for the second line of the example shown in FIG. 6 includes the input speech "Speech A," the input sentence "Please tell me how this speech is spoken," and the correct output sentence "A woman is speaking quickly and loudly." Similarly, the training data for the third line of the example shown in FIG. 6 includes the input speech "Speech B," the input sentence "Please tell me how this speech is spoken," and the correct output sentence "A man speaks slowly and calmly." Similarly, the training data for the fourth line of the example shown in FIG. 6 includes the input speech "Speech B," the input sentence "Please tell me the gender of the speaker of this speech," and the correct output sentence "It is a man." Similarly, the training data on the fifth line of the example shown in Figure 6 includes the input speech "Speech C," the input sentence "What is the emotion of the speaker of this speech?", and the correct output sentence "This speaker seems a little irritated."

In this way, the training dataset is composed of training data represented by pairs of input speech, input sentences, and correct output sentences. Note that, as shown in Figure 6, the training dataset may contain multiple training data sets that contain different input sentences and correct output sentences for the same input speech.

<<Detailed Functional Configuration Example of Model Learning Unit 202>>
An example of a detailed functional configuration of the model learning unit 202 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a detailed functional configuration of the model learning unit 202.

As shown in FIG. 7, the model learning unit 202 includes a learning data input unit 211, an audio encoding unit 212, a first integration unit 213, a second integration unit 214, a linear transformation unit 215, a posterior probability calculation unit 216, a parameter update unit 217, and an end determination unit 218.

The learning data input unit 211 inputs one piece of learning data from the learning dataset stored in the learning dataset storage unit 206.

The speech encoding unit 212 is realized by the speech encoder 1100 included in the speech understanding model 1000. The speech encoding unit 212 receives input speech included in the training data input by the training data input unit 211, and outputs N first speech feature vectors h _n (t) (n=1, ..., N) from the N integration target layers in each time interval t (t=1, ..., T).

The first integration unit 213 is realized by the speech encoder output integration block 1200 included in the speech understanding model 1000. In each time interval t (t=1, ..., T), the first integration unit 213 receives the first speech feature vectors h _n (t) (n=1, ..., N) in that time interval t as input, and outputs a first integrated vector e(t) obtained by integrating each of these first speech feature vectors h _n (t) (n=1, ..., N).

The second integration unit 214 is realized by the time information integration block 1300 included in the speech understanding model 1000. The second integration unit 214 receives the first integrated vector e(t) for each time interval t as input, and outputs the second integrated vector v obtained by integrating each of these first integrated vectors e(t) (t = 1, ..., T) in the time direction.

The linear transformation unit 215 is realized by the linear transformation layer 1400 included in the speech understanding model 1000. The linear transformation unit 215 receives the second integrated vector v as input and outputs the second speech feature vector w obtained by linearly transforming the second integrated vector v.

The posterior probability calculation unit 216 is realized by the large-scale language model 1500 included in the speech understanding model 1000. The posterior probability calculation unit 216 receives as input an input sentence included in the training data input by the training data input unit 211 and a second speech feature vector w, and calculates the posterior probability of an output sentence when the input sentence and the second speech feature vector w are given.

More specifically, the i-th token constituting the output sentence is defined as s _i (where s ₁ is the token representing the beginning of the sentence). Furthermore, the posterior probability that token s ₁ is generated when the input sentence and the second speech feature vector w are given is defined as p(s ₁ ), and the posterior probability that token s _i is generated when the input sentence, the second speech feature vector w, and s ₁ , ..., s _i-1 are given is defined as p(s _i ) (where i≧2). In this case, the posterior probability calculation unit 216 calculates the posterior probability p(s _i ) for i=1, ..., I. I is the number of tokens included in the output sentence (i.e., the length of the output sentence), and may be, for example, the length of the correct output sentence included in the training data input by the training data input unit 211. Here, the posterior probability p(s _i ) is expressed as an M-dimensional vector in which, for example, when the number of dimensions of the token embedding space is M, the m-th element represents the probability that the m-th type of token is generated, and the sum of the values of all elements is 1.

Note that a token is the basic processing unit used when a language model, such as a large-scale language model, processes a string of characters. A typical example of a token is a word, but tokens are not limited to words and can also be, for example, characters, morphemes, subwords, or coherent strings of characters.

The parameter update unit 217 uses the posterior probabilities calculated by the posterior probability calculation unit 216 and the correct output sentences included in the learning data input by the learning data input unit 211 to learn the learning target parameters of the speech understanding model 1000 using an existing optimization method.

More specifically, the i-th token constituting the correct output sentence is denoted by s _i ' (where s ₁ ' is a token representing the beginning of the sentence, and s _I ' is a token representing the end of the sentence). Furthermore, the probability that token s _i ' is generated is denoted by p(s _i '). For example, when the number of dimensions of the token embedding space is M, the probability p(s _i ') is expressed as an M-dimensional vector in which only the element corresponding to token s _i ' has a value of 1 and the other elements have a value of 0. In this case, the parameter update unit 217 uses the sum of -p(s _i ')logp(s _i ) for i = 1, ..., I (i.e., cross entropy) as a loss function, and updates the training parameters using an existing optimization method so as to minimize the loss function. The optimization method that can be used to update the training parameters is not limited to a specific method, but an online optimization method based on stochastic gradient descent, for example, can be used.

The termination determination unit 218 determines whether to terminate the updating of the training parameters. At this time, the termination determination unit 218 determines to terminate the updating of the training parameters if a predetermined termination condition is met, and determines not to terminate the updating of the training parameters if not. As a result, the training parameters of the audio encoder output integrated block 1200, the training parameters of the temporal information integrated block 1300, and the training parameters of the linear transformation layer 1400 are repeatedly updated until the predetermined termination condition is met. Here, examples of the predetermined termination condition include the training parameters being updated a predetermined number of times or more, the number of epochs being a predetermined number of epochs or more, the value of the loss function being less than a predetermined value, the loss function converging, etc.

<Model Building Process>
The model construction process will be described below with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the model construction process.

The model construction unit 201 constructs the speech understanding model 1000 shown in FIG. 1 using the trained speech encoder stored in the trained speech encoder storage unit 203 and the trained large-scale language model stored in the trained large-scale language model storage unit 204 (step S101). At this time, the model construction unit 201 initializes the training parameters of the speech encoder output integration block 1200, the training parameters of the time information integration block 1300, and the training parameters of the linear transformation layer 1400 using any method. This constructs an untrained speech understanding model 1000.

Then, the model construction unit 201 stores the speech understanding model 1000 constructed in step S101 above in the speech understanding model storage unit 205 (step S102).

<Model learning process>
The model learning process will be described below with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the model learning process.

The learning data input unit 211 of the model learning unit 202 inputs one piece of learning data from the learning dataset stored in the learning dataset storage unit 206 (step S201). The learning data input unit 211 inputs, for example, one piece of learning data that has not yet been input for the current number of epochs from the learning data that make up the learning dataset. The epoch number starts from 0 and is incremented by 1 each time all of the learning data that make up the learning dataset is input.

The speech encoding unit 212 of the model training unit 202 takes the input speech included in the training data input in step S201 above as input, and outputs N first speech feature vectors h _n (t) (n = 1, ..., N) from the N integration target layers in each time interval t (t = 1, ..., T) (step S202).

The first integration unit 213 of the model learning unit 202 takes as input the first speech feature vector h _n (t) (n = 1, ..., N) for each time interval t (t = 1, ..., T) and outputs a first integrated vector e(t) by integrating each of these first speech feature vectors h _n (t) (n = 1, ..., N) (step S203).

The second integration unit 214 of the model learning unit 202 receives the first integrated vector e(t) for each time interval t as input, and outputs a second integrated vector v obtained by integrating each of these first integrated vectors e(t) (t = 1, ..., T) in the time direction (step S204).

The linear transformation unit 215 of the model learning unit 202 receives the second integrated vector v as input and outputs the second speech feature vector w obtained by linearly transforming the second integrated vector v (step S205).

The posterior probability calculation unit 216 of the model training unit 202 receives as input the input sentence included in the training data input in step S201 above and the second speech feature vector w output in step S205 above, and calculates the posterior probability of the output sentence when the input sentence and the second speech feature vector w are given (step S206).

The parameter update unit 217 of the model training unit 202 uses the posterior probability calculated in step S206 and the correct output sentence included in the training data input in step S201 to train the training parameters of the speech understanding model 1000 by an existing optimization method (step S207). That is, the parameter update unit 217 uses the cross entropy (specifically, the sum of -p(s i ')logp(s i ) for i = 1, ..., _I ) between the output sentence generated by the speech understanding model 1000 when the input speech and input sentence included in the training data input in step _S201 are given and the correct output sentence included in the training data as a loss function, and updates the training parameters by an existing optimization method so as to minimize the loss function.

The termination determination unit 218 of the model learning unit 202 determines whether to terminate the update of the learning parameter (step S208). That is, if a predetermined termination condition is met, the termination determination unit 218 determines to terminate the update of the learning parameter, and if not, determines not to terminate the update of the learning parameter.

If it is determined in step S208 above that the update of the learning target parameters should not be terminated, the model learning unit 202 returns to step S201 above. As a result, steps S201 to S207 above are repeatedly executed until a predetermined termination condition is met.

On the other hand, if it is determined in step S208 above that the update of the learning target parameters is to be terminated, the model learning unit 202 terminates the model learning process. As a result, the learning target parameters are learned and the trained speech understanding model 1000 is obtained.

[At the time of inference]
The following describes the speech understanding device 10 during inference. Note that the following mainly describes differences from the learning period, and omits explanations of points that may be the same as the learning period, as appropriate.

<Example of hardware configuration of the speech understanding device 10 during inference>
The hardware configuration of the speech understanding device 10 during inference may be the same as during learning, and therefore a description thereof will be omitted.

<Example of functional configuration of the speech understanding device 10 during inference>
An example of the functional configuration of the speech understanding device 10 at the time of inference will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the functional configuration of the speech understanding device 10 at the time of inference.

As shown in FIG. 10, the speech understanding device 10 during inference has an output sentence generation unit 207. The output sentence generation unit 207 is realized, for example, by processing in which one or more programs installed in the speech understanding device 10 are executed by the processor 108 or the like. The speech understanding device 10 during inference also has a trained speech understanding model storage unit 208 and a test data storage unit 209. Each of these storage units is realized, for example, by a storage area of the auxiliary storage device 107 or the like. However, at least one of these storage units may also be realized by a storage area of a storage device (for example, a storage device included in a database server or the like) that is communicatively connected to the speech understanding device 10.

The output sentence generation unit 207 uses the test data stored in the test data storage unit 209 and the trained speech understanding model 1000 stored in the trained speech understanding model storage unit 208 to generate and output an output sentence corresponding to the question expressed by the input sentence included in the test data (i.e., text data representing a natural language response or answer to the question). Here, test data refers to data represented by a pair of input speech and an input sentence that is a natural language question related to the input speech. Furthermore, the trained speech understanding model 1000 refers to a speech understanding model 1000 whose learning target parameters have been trained. A detailed example of the functional configuration of the output sentence generation unit 207 will be described later.

The trained speech understanding model storage unit 208 stores the trained speech understanding model 1000. The test data storage unit 209 stores the given test data.

<<Detailed Functional Configuration Example of Output Sentence Generation Unit 207>>
An example of a detailed functional configuration of the output sentence generation unit 207 will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of a detailed functional configuration of the output sentence generation unit 207.

As shown in FIG. 11, the output sentence generation unit 207 includes a test data input unit 221, a speech encoding unit 222, a first integration unit 223, a second integration unit 224, a linear conversion unit 225, a generation unit 226, and an output unit 227.

The test data input unit 221 inputs one piece of test data stored in the test data storage unit 209.

The speech encoding unit 222 is realized by the speech encoder 1100 included in the trained speech understanding model 1000. The speech encoding unit 222 receives input speech included in the test data input by the test data input unit 221, and outputs N first speech feature vectors h _n (t) (n=1, ..., N) from the N integration target layers in each time interval t (t=1, ..., T).

The first integration unit 223 is realized by the speech encoder output integration block 1200 included in the trained speech understanding model 1000. In each time interval t (t=1, ..., T), the first integration unit 223 receives the first speech feature vectors h _n (t) (n=1, ..., N) in that time interval t as input, and outputs a first integrated vector e(t) obtained by integrating each of these first speech feature vectors h _n (t) (n=1, ..., N).

The second integration unit 224 is realized by the time information integration block 1300 included in the trained speech understanding model 1000. The second integration unit 224 receives the first integrated vector e(t) for each time interval t as input, and outputs the second integrated vector v obtained by integrating each of these first integrated vectors e(t) (t = 1, ..., T) in the time direction.

The linear transformation unit 225 is realized by the linear transformation layer 1400 included in the trained speech understanding model 1000. The linear transformation unit 225 receives the second integrated vector v as input and outputs the second speech feature vector w obtained by linearly transforming the second integrated vector v.

The generation unit 226 is realized by the large-scale language model 1500 included in the trained speech understanding model 1000. The generation unit 226 receives as input an input sentence included in the test data input by the test data input unit 221 and a second speech feature vector w, and generates an output sentence when the input sentence and the second speech feature vector w are given.

More specifically, the i-th token constituting the output sentence is denoted by s _i . Furthermore, the posterior probability that token s _i will be generated when the input sentence and the second speech feature vector w are given is denoted by p(s _i ), and the posterior probability that token s _i will be generated when the input sentence, the second speech feature vector w, and s ₁ , ..., s _i-1 are given is denoted by p(s _i ) (where i≧2). In this case, the generation unit 226 generates the output sentence by generating token s _i in accordance with the posterior probability p(s _i ) until, for example, a token representing the end of the sentence is generated.

The output unit 227 outputs the output sentence generated by the generation unit 226 to a predetermined output destination. Here, examples of the predetermined output destination include a storage area such as the auxiliary storage device 107, a display device 102 such as a display, other devices or equipment that are communicatively connected, etc.

<Output sentence generation process>
The output sentence generation process will be described below with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the output sentence generation process.

The test data input unit 221 of the output statement generation unit 207 inputs one piece of test data stored in the test data storage unit 209 (step S301).

The speech encoding unit 222 of the output sentence generation unit 207 uses the input speech contained in the test data input in step S301 above as input, and outputs N first speech feature vectors h _n (t) (n = 1, ..., N) from the N integration target layers in each time interval t (t = 1, ..., T) (step S302).

The first integration unit 223 of the output sentence generation unit 207 takes the first speech feature vector h _n (t) (n = 1, ..., N) for each time interval t (t = 1, ..., T) as input, and outputs a first integrated vector e(t) by integrating each of these first speech feature vectors h _n (t) (n = 1, ..., N) (step S303).

The second integration unit 224 of the output sentence generation unit 207 receives the first integrated vector e(t) for each time interval t as input, and outputs a second integrated vector v obtained by integrating each of these first integrated vectors e(t) (t = 1, ..., T) in the time direction (step S304).

The linear transformation unit 225 of the output sentence generation unit 207 receives the second integrated vector v as input and outputs the second speech feature vector w obtained by linearly transforming the second integrated vector v (step S305).

The generation unit 226 of the output sentence generation unit 207 receives as input the input sentence included in the test data input in step S301 above and the second speech feature vector w output in step S305 above, and generates an output sentence when the input sentence and the second speech feature vector w are given (step S306).

The output unit 227 of the output sentence generation unit 207 outputs the output sentence generated in step S306 above to a predetermined output destination (step S307). This results in an output sentence that is a response or answer to the question related to the input voice.

<Summary>
As described above, the speech understanding device 10 according to this embodiment can realize speech understanding technology using the speech understanding model 1000 in which the speech encoder output integration block 1200 and the time information integration block 1300 are present between the speech encoder 1100 and the large-scale language model 1500. For this reason, by using the speech understanding device 10 according to this embodiment, it is possible to expect an improvement in the processing accuracy of a downstream system that uses the recognition results of non-linguistic information and paralinguistic information, for example.

The following additional notes are further disclosed regarding the above-described embodiments.
(Appendix 1)
Memory and
at least one processor coupled to said memory;
Including,
The processor:
inputting training data including a speech, a first sentence related to the speech, and a second sentence corresponding to the first sentence;
generating information representing the features of the speech for each predetermined time interval based on a speech feature extractor configured with multiple layers;
generating first integrated information for each time interval based on a first parameter, the first integrated information being obtained by integrating information representing the features generated in a plurality of predetermined layers of the audio feature extractor;
generating second integrated information by integrating the first integrated information for each time section in a time direction based on a second parameter;
calculating a generation probability of a third sentence corresponding to the first sentence based on the first sentence, the second integrated information, and a language model;
A learning device that learns learning target parameters including the first parameter and the second parameter based on the generation probability of the third sentence and the second sentence.
(Appendix 2)
at least one processor coupled to said memory;
Including,
The processor:
inputting test data including a speech and a first sentence related to the speech;
generating information representing the features of the speech for each predetermined time interval based on a speech feature extractor configured with multiple layers;
generating first integrated information for each time interval by integrating information representing the features generated in a plurality of predetermined layers of the speech feature extractor based on the trained first parameters;
generating second integrated information by integrating the first integrated information for each time interval in a time direction based on the learned second parameter;
a reasoning device that generates a second sentence corresponding to the first sentence based on the first sentence, the second integrated information, and a language model;
(Appendix 3)
The processor:
linearly transforming the second integrated information based on a third parameter;
2. The learning device according to claim 1, wherein the learning device calculates the probability of generating the third sentence based on the first sentence, the second integrated information after the linear transformation, and the language model.
(Appendix 4)
the first parameter is a weight used in a weighted sum or a linear transform coefficient used in a linear transform sum,
The processor:
The learning device according to claim 1 or 3, wherein first integrated information is generated by integrating the features using the weighted sum or the linear transformation sum.
(Appendix 5)
The second parameter is a weight of a self-attention pooling layer or a parameter of a one-dimensional convolutional neural network;
The processor:
The learning device according to claim 1 or 3, wherein second integrated information is generated by integrating the first integrated information in a time direction using the self-attention pooling layer or the one-dimensional convolutional neural network.
(Appendix 6)
A non-transitory storage medium storing a program executable by a computer to perform a learning process,
The learning process includes:
inputting training data including a speech, a first sentence related to the speech, and a second sentence corresponding to the first sentence;
generating information representing the features of the speech for each predetermined time interval based on a speech feature extractor configured with multiple layers;
generating first integrated information for each time interval based on a first parameter, the first integrated information being obtained by integrating information representing the features generated in a plurality of predetermined layers of the audio feature extractor;
generating second integrated information by integrating the first integrated information for each time section in a time direction based on a second parameter;
calculating a generation probability of a third sentence corresponding to the first sentence based on the first sentence, the second integrated information, and a language model;
A non-transitory storage medium for learning learning target parameters including the first parameter and the second parameter based on the generation probability of the third sentence and the second sentence.
(Appendix 7)
A non-transitory storage medium storing a program executable by a computer to perform an inference process,
The inference process includes:
inputting test data including a speech and a first sentence related to the speech;
generating information representing the features of the speech for each predetermined time interval based on a speech feature extractor configured with multiple layers;
generating first integrated information for each time interval by integrating information representing the features generated in a plurality of predetermined layers of the speech feature extractor based on the trained first parameters;
generating second integrated information by integrating the first integrated information for each time interval in a time direction based on the learned second parameter;
a non-transitory storage medium for generating a second sentence corresponding to the first sentence based on the first sentence, the second integrated information, and a language model;

The present invention is not limited to the specifically disclosed embodiments above, and various modifications, alterations, and combinations with known technologies are possible without departing from the scope of the claims.

[References]
Reference 1: Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli, "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations," arXiv preprint arXiv:2006.11477, 2020.
Reference 2: S. Chen et al., "WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing," in IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505-1518, Oct. 2022, doi: 10.1109/JSTSP.2022.3188113.
Reference 3: H. Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models," arXiv preprint arXiv:2307.09288, 2023.

10 Speech understanding device 101 Input device 102 Display device 103 External I/F
103a Recording medium 104 Communication I/F
105 RAM
106 ROM
107 Auxiliary storage device 108 Processor 109 Bus 201 Model construction unit 202 Model learning unit 203 Trained speech encoder storage unit 204 Trained large-scale language model storage unit 205 Speech understanding model storage unit 206 Training dataset storage unit 207 Output sentence generation unit 208 Trained speech understanding model storage unit 209 Test data storage unit 211 Training data input unit 212 Speech encoding unit 213 First integration unit 214 Second integration unit 215 Linear transformation unit 216 Posterior probability calculation unit 217 Parameter update unit 218 End determination unit 221 Test data input unit 222 Speech encoding unit 223 First integration unit 224 Second integration unit 225 Linear transformation unit 226 Generation unit 227 Output unit

Claims (8)

an input unit for inputting training data including a speech, a first sentence related to the speech, and a second sentence corresponding to the first sentence;
a speech feature generation unit that generates information representing the speech features for each predetermined time interval based on a speech feature extractor configured with multiple layers;
a first integration unit that generates, for each time interval, first integrated information by integrating information representing the features generated in a plurality of predetermined layers of the speech feature extractor based on a first parameter;
a second integration unit that generates second integrated information by integrating the first integrated information for each time section in a time direction based on a second parameter;
a calculation unit that calculates a generation probability of a third sentence corresponding to the first sentence based on the first sentence, the second integrated information, and a language model;
a learning unit that learns learning target parameters including the first parameter and the second parameter based on the generation probability of the third sentence and the second sentence;
A learning device having the above configuration. an input unit for inputting test data including a speech and a first sentence related to the speech;
a speech feature generation unit that generates information representing the speech features for each predetermined time interval based on a speech feature extractor configured with multiple layers;
a first integration unit that generates first integrated information by integrating information representing the features generated in a plurality of predetermined layers of the speech feature extractor for each time interval based on the trained first parameters;
a second integration unit that generates second integrated information by integrating the first integrated information for each time interval in a time direction based on a learned second parameter;
a sentence generation unit that generates a second sentence corresponding to the first sentence based on the first sentence, the second integrated information, and a language model;
An inference device having: a linear transformation unit that linearly transforms the second integrated information based on a third parameter;
The calculation unit
The learning device according to claim 1 , wherein the generation probability of the third sentence is calculated based on the first sentence, the second integrated information after the linear transformation, and the language model. the first parameter is a weight used in a weighted sum or a linear transform coefficient used in a linear transform sum,
The first integration unit
The learning device according to claim 1 or 3, wherein first integrated information is generated by integrating the features using the weighted sum or the linear transformation sum. The second parameter is a weight of a self-attention pooling layer or a parameter of a one-dimensional convolutional neural network;
The second integration unit
The learning device according to claim 1 or 3, wherein the first integrated information is integrated in a time direction by the self-attention pooling layer or the one-dimensional convolutional neural network to generate second integrated information. an input step of inputting training data including a speech, a first sentence related to the speech, and a second sentence corresponding to the first sentence;
a speech feature generation step for generating information representing the speech features for each predetermined time interval based on a speech feature extractor configured with multiple layers;
a first integration step for generating, for each time interval, first integrated information by integrating information representing the features generated in a plurality of predetermined layers of the speech feature extractor based on a first parameter;
a second integration step of generating second integrated information by integrating the first integrated information for each time section in a time direction based on a second parameter;
a calculation step of calculating a generation probability of a third sentence corresponding to the first sentence based on the first sentence, the second integrated information, and a language model;
a learning procedure for learning learning target parameters including the first parameter and the second parameter based on the generation probability of the third sentence and the second sentence;
A learning method that computers perform. an input step of inputting test data including a speech and a first sentence related to the speech;
a speech feature generation step for generating information representing the speech features for each predetermined time interval based on a speech feature extractor configured with multiple layers;
a first integration step for generating, for each time interval, first integrated information by integrating information representing the features generated in a plurality of predetermined layers of the speech feature extractor based on the trained first parameters;
a second integration step of generating second integrated information by integrating the first integrated information for each time interval in a time direction based on a learned second parameter;
a sentence generation step of generating a second sentence corresponding to the first sentence based on the first sentence, the second integrated information, and a language model;
A method of inference performed by a computer. A program that causes a computer to function as the learning device described in claim 1 or the inference device described in claim 2.