WO2025099939A1 - Learning device, generation device, learning method, generation method, and program - Google Patents

Abstract

A learning device according to one aspect includes: an input unit for inputting learning data including voice data and training data representing a natural language description of paralinguistic information or non-language information included in the voice represented by the voice data; a calculation unit for calculating, by a machine learning model that takes a voice sequence, obtained by converting the voice data into a predetermined format for each predetermined unit, as input and calculates the generation probability of a natural language description of paralinguistic information or non-language information included in the voice, the generation probability of the natural language description of paralinguistic information or non-language information included in the voice represented by the voice data of the conversion source of the input voice sequence; and an update unit for updating learnable parameters of the machine learning model on the basis of the error between the natural language description generated according to the generation probability and the training data included in the learning data.

Description

Learning device, generation device, learning method, generation method, and program

This disclosure relates to a learning device, a generating device, a learning method, a generating method, and a program.

In recent years, in the field of speech processing, many technologies have been proposed for recognizing or detecting paralinguistic and non-linguistic information contained in speech (for example, Non-Patent Document 1). Here, paralinguistic and non-linguistic information both refer to information contained in speech that is not linguistic information. Hereinafter, paralinguistic and non-linguistic information will be collectively referred to as "paralinguistic and non-linguistic information."

M. B. Akcay and K. Oguz, “Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers,” Speech Communication, vol. 116, pp. 56-76, 2020.

However, conventional technologies that recognize or detect paralinguistic and non-linguistic information contained in speech often only output the results of the recognition or detection, and are not highly interpretable.

The present disclosure has been made in consideration of the above points, and provides technology that can output paralinguistic and non-linguistic information contained in speech in a highly interpretable format.

A learning device according to one aspect of the present disclosure includes an input unit that inputs learning data including voice data and teacher data representing a natural language description of paralinguistic information or non-linguistic information contained in the voice represented by the voice data, a calculation unit that uses as input a voice sequence obtained by converting the voice data into a predetermined format for each predetermined unit, and calculates the generation probability of a natural language description of paralinguistic information or non-linguistic information contained in the voice represented by the voice data from which the input voice sequence is converted using a machine learning model that calculates the generation probability of a natural language description of paralinguistic information or non-linguistic information contained in the voice, and an update unit that updates the learnable parameters of the machine learning model based on the error between the natural language description generated according to the generation probability and the teacher data included in the learning data.

Technology is provided that can output paralinguistic and non-linguistic information contained in speech in a highly interpretable format.

1 is a diagram illustrating an example of a hardware configuration of a caption generation device according to a first embodiment. A figure showing an example of the functional configuration of the caption generation device according to the first embodiment during learning. FIG. 4 is a diagram illustrating an example of learning data stored in a learning data storage unit. 2 is a diagram illustrating an example of a detailed functional configuration of a caption generation unit according to the first embodiment. FIG. 5 is a flowchart illustrating an example of a learning process according to the first embodiment. 11 is a flowchart illustrating an example of a generation process of an output token random sequence according to the first embodiment. 2 is a diagram illustrating an example of a functional configuration at the time of inference of the caption generation device according to the first embodiment. FIG. 11 is a flowchart showing an example of a caption generation process according to the first embodiment. 11 is a flowchart illustrating an example of a generation process of an output token sequence according to the first embodiment. A figure showing an example of the functional configuration of a caption generation device according to a second embodiment during learning. FIG. 11 is a diagram illustrating an example of a detailed functional configuration of a caption generation unit according to a second embodiment. 13 is a flowchart illustrating an example of a learning process according to the second embodiment. 13 is a flowchart illustrating an example of a generation process of an output token random sequence according to the second embodiment. A figure showing an example of a functional configuration at the time of inference of a caption generation device according to a second embodiment. 13 is a flowchart showing an example of a caption generation process according to the second embodiment. 13 is a flowchart illustrating an example of a generation process of an output token sequence according to the second embodiment.

Each embodiment of the present invention will be described below with reference to the drawings. In each of the following embodiments, as an example, a caption generation device 10 will be described that can output paralinguistic and non-linguistic information contained in audio in the form of a caption written in natural language. A caption is generally a term that refers to an explanatory text about the contents of, for example, a video or a photo, but in the following embodiments, the term is used to refer to an explanatory text that describes the paralinguistic and non-linguistic information contained in audio in natural language. This makes it possible to output the paralinguistic and non-linguistic information contained in audio in the highly interpretable form of a caption. Note that instead of the term caption, terms such as "natural language sentence", "explanatory text", "sentence", and "character string" may be used.

However, outputting paralinguistic and non-linguistic information contained in audio in the form of captions is just one example, and the highly interpretable format for expressing paralinguistic and non-linguistic information is not limited to captions. Each of the embodiments described below can be similarly applied to cases where paralinguistic and non-linguistic information is expressed in a highly interpretable format other than captions. Examples of highly interpretable formats other than captions include figures, tables, images, etc. that include explanatory text that describes the paralinguistic and non-linguistic information in natural language.

Note that both paralinguistic and non-linguistic information are non-linguistic information contained in speech, but generally, paralinguistic information refers to information that can be changed at will, while non-linguistic information refers to information that cannot be changed at will. Examples of paralinguistic information include information that indicates the speaker's intentions and attitudes. On the other hand, examples of non-linguistic information include information that indicates the speaker's gender and emotions, the presence or absence of a certain illness, and information that identifies the speaker.

[First embodiment]
A first embodiment will be described below. Here, the caption generation device 10 according to the first embodiment has a "learning time" during which a machine learning model that generates captions (hereinafter referred to as a "captioning model") is trained, and an "inference time" during which a caption is generated from a voice using the trained captioning model. In the following, it is assumed that the caption generation device 10 is realized by the same device during learning and inference, but for example, the caption generation device 10 may be realized by different devices during learning and inference.

Note that the caption generation device 10 during learning may be called, for example, a "learning device." Furthermore, the caption generation device 10 during inference may be called, for example, simply a "generation device" or an "inference device."

<Hardware configuration example of the caption generation device 10 according to the first embodiment>
An example of the hardware configuration of the caption generation device 10 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the hardware configuration of the caption generation device 10 according to the first embodiment.

As shown in FIG. 1, the caption generating device 10 according to the first embodiment has an input device 101, a display device 102, an external I/F 103, a communication I/F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other so as to be able to communicate with each other via a bus 109.

The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the caption generation device 10 does not have to have at least one of the input device 101 and the display device 102, for example.

The external I/F 103 is an interface with external devices such as a recording medium 103a. Examples of recording media 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

The communication I/F 104 is an interface for connecting to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a non-volatile storage device such as a HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory. The processor 108 is various types of arithmetic devices such as a CPU (Central Processing Unit) or GPU (Graphic Processing Unit).

Note that the hardware configuration shown in FIG. 1 is an example, and the hardware configuration of the caption generation device 10 is not limited to this. For example, the caption generation device 10 may have multiple auxiliary storage devices 107 and multiple processors 108, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

- Learning Time Hereinafter, the learning time of the caption generation device 10 according to the first embodiment will be described.

<Example of functional configuration during learning of the caption generation device 10 according to the first embodiment>
An example of a functional configuration of the caption generation device 10 according to the first embodiment during learning will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of a functional configuration of the caption generation device 10 according to the first embodiment during learning.

As shown in FIG. 2, during learning, the caption generation device 10 according to the first embodiment has an input unit 201, a voice conversion unit 202, a caption generation unit 203, a correct caption conversion unit 204, and a parameter update unit 205. Each of these units is realized, for example, by a process in which one or more programs installed in the caption generation device 10 are executed by the processor 108. Furthermore, during learning, the caption generation device 10 according to the first embodiment has a learning data storage unit 206 and a model parameter storage unit 207. Each of these storage units is realized, for example, by a storage area such as the auxiliary storage device 107. However, for example, at least one of the storage units of the learning data storage unit 206 and the model parameter storage unit 207 may be realized by a storage area of a storage device or the like that is communicably connected to the caption generation device 10.

The input unit 201 inputs the learning data stored in the learning data storage unit 206. Here, the learning data is data for training a captioning model, and is represented as a pair of audio data representing the voice of a certain speaker and a correct answer caption that describes the paralinguistic and non-linguistic information contained in the voice in natural language.

The voice conversion unit 202 converts the voice data included in the training data input by the input unit 201 into an input voice sequence. The input voice sequence is the voice sequence input to the captioning model. The voice sequence is also a sequence of data obtained by converting voice data into a format that can be handled on a frame-by-frame basis by signal processing or the like. Specific examples of voice sequences include sequence of data converted into formats such as spectrum, mel frequency cepstrum (MFCC), and mel spectrogram.

However, depending on the model that realizes the audio encoding unit 301 described below, the audio conversion unit 202 may use data obtained by dividing the audio waveform of the audio data into frames as the input audio sequence.

The caption generation unit 203 is realized by a captioning model, and generates an output token probability sequence from the input speech sequence converted by the speech conversion unit 202. The output token probability sequence is a sequence of generation probabilities of the output token that constitutes the caption. A token is a sequence of one or more characters that represents a certain unit, and a typical example is a word. The generation probability of the output token is expressed as an N-dimensional vector in which the sum of the elements of each dimension is 1, and the value of each element of the dimension is the probability that the token corresponding to that element will be generated, when the number of types of tokens that can be generated is N. An example of a detailed functional configuration of the caption generation unit 203 will be described later.

The correct caption conversion unit 204 converts the correct caption contained in the learning data input by the input unit 201 into a correct token probability sequence. A correct token probability sequence is a sequence of probabilities corresponding to the correct tokens that make up the correct caption. For example, when the number of types of tokens to be generated is N, the probability corresponding to the correct token is expressed as an N-dimensional vector in which only the value of the element corresponding to the correct token is 1 and the values of the other elements are 0.

The parameter update unit 205 updates the learnable parameters of the captioning model (hereinafter referred to as "model parameters") using the error between the output token probability sequence generated by the caption generation unit 203 and the correct token probability sequence converted by the correct caption conversion unit 204. Note that, for example, cross-entropy error can be used as the error.

The learning data storage unit 206 stores learning data that has been created in advance. Specific examples of learning data will be described later.

The model parameter storage unit 207 stores the model parameters of the captioning model.

<Example of learning data stored in learning data storage unit 206>
An example of the learning data stored in the learning data storage unit 206 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the learning data stored in the learning data storage unit 206.

As shown in FIG. 3, the learning data storage unit 206 stores one or more learning data. Each learning data includes audio data representing a voice of a certain speaker speaking a certain sentence, and a correct answer caption representing an explanatory text in natural language describing the paralinguistic and non-linguistic information received by a person who hears the voice. The correct answer caption may be called, for example, teacher data.

For example, in the example shown in FIG. 3, the learning data in the first row contains audio data 1 and the correct caption "An elderly man speaking slowly." Similarly, the learning data in the second row contains audio data 2 and the correct caption "The voice of a young woman who seems to be suffering from a cold." Similarly, the learning data in the third row contains audio data 3 and the correct caption "A little girl playing happily with her friends." Similarly, the learning data in the fourth row contains audio data 4 and the correct caption "A male university student whispering to the person next to him."

In this way, the training data storage unit 206 stores multiple training data, each represented by a pair of audio data and its correct caption. The number of training data stored in the training data storage unit 206 is preferably several hundred to several thousand or more. It is also preferable that the number of speakers of the audio data included in the training data is several hundred or more, and the number of audio documents represented by the audio data is several to several tens of sentences or more.

<<Detailed Functional Configuration Example of the Caption Generation Unit 203 According to the First Embodiment>>
A detailed functional configuration example of the caption generation unit 203 according to the first embodiment will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of a detailed functional configuration of the caption generation unit 203 according to the first embodiment. In the following, as an example, a case where a captioning model is configured with three models, a voice encoder, which is a machine learning model including a neural network, a conversion network, and text data, will be described. However, this is only an example, and the captioning model does not necessarily have to be configured with three machine learning models, a voice encoder, a conversion network, and text data, and may have other configurations. For example, the captioning model may be configured with one machine learning model including a neural network.

As shown in FIG. 4, the caption generation unit 203 according to the first embodiment is composed of an audio encoding unit 301 realized by an audio encoder, a vector conversion unit 302 realized by a conversion network, and a text decoding unit 303 realized by a text decoder.

The speech encoding unit 301 takes an input speech sequence as input and generates a fixed-length vector (hereinafter referred to as an "audio fixed-length vector") that expresses the features of the input speech sequence (features that express paralinguistic and non-linguistic information). As a speech encoder that realizes the speech encoding unit 301, for example, a Global Style Token (Reference 1) or the like can be used. In addition, as a speech encoder that realizes the speech encoding unit 301, for example, a speech representation model based on self-supervised learning such as HuBERT (Reference 2) or WavLM (Reference 3) and a Global Style Token or the like may be used.

When an audio SSL model is used in the audio encoder that realizes the audio encoding unit 301, the above-mentioned audio conversion unit 202 takes data obtained by dividing the audio waveform of the audio data by frame as the input audio sequence. In this case, the input audio sequence is used as input, and a sequence of frame-by-frame features output from the audio SSL model is input to a Global Style Token, etc., and an audio fixed-length vector is generated.

The vector conversion unit 302 receives the fixed-length audio vector generated by the audio encoding unit 301 as input and generates a sequence of P text information vectors (hereinafter referred to as a "text information vector sequence") that can be handled by the text decoding unit 303. In other words, the vector conversion unit 302 converts the input fixed-length audio vector into a text information vector sequence. Here, P is a predetermined fixed value. The text information vector sequence is a vector sequence that represents a word embedding expression that reflects the features expressed by the input fixed-length audio vector for the P vectors included in the model parameters. The P vectors included in the model parameters are also one of the learnable parameters.

As a conversion network that realizes the vector conversion unit 302, for example, a normal MLP (Multilayer Perceptron) can be used, similar to the Mapping Network described in Reference 4. In addition to this, as a conversion network that realizes the vector conversion unit 302, for example, a neural network that can take into account sequence information before and after a transformer-encoder, or a neural network that combines an MLP and a transformer-encoder, etc. can also be used.

The text decoding unit 303 receives the text information vector sequence converted by the vector conversion unit 302 as input and generates an output token probability sequence.

As a text decoder for realizing the text decoding unit 303, for example, a decoder-type pre-trained model (Reference 5) trained with large-scale text data can be used. Examples of such pre-trained models include large-scale language models (LLMs) such as the Generative Pre-trained Transformer (GPT).

The text decoding unit 303 can also generate an output token sequence that constitutes a caption, for example, by sampling the output tokens according to each generation probability included in the output token probability sequence.

<Learning Process According to the First Embodiment>
The learning process according to the first embodiment will be described below with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the learning process according to the first embodiment. In the following, online learning is assumed as an example, and steps S101 to S105 in Fig. 5 are repeatedly executed for each piece of learning data. However, this is only an example, and the learning process may be executed by mini-batch learning, batch learning, or the like.

First, the input unit 201 inputs one piece of learning data stored in the learning data storage unit 206 (step S101).

Next, the voice conversion unit 202 converts the voice data contained in the training data input in step S101 above into an input voice sequence (step S102).

Next, the caption generation unit 203 generates an output token probability sequence from the input speech sequence converted in step S102 (step S103). Details of the processing in this step will be described later.

Next, the correct caption conversion unit 204 converts the correct caption contained in the learning data input in step S101 above into a correct token probability sequence (step S104).

Then, the parameter update unit 205 updates the model parameters using the error between the output token probability sequence generated in step S103 and the correct token probability sequence converted in step S104 (step S105). That is, the parameter update unit 205 updates the model parameters by a known optimization method so as to minimize the error. However, at this time, the parameter update unit 205 may exclude the model parameters of the text decoder that realizes the text decoding unit 303 from the update target. Also, if a voice SSL model is used for the voice encoder that realizes the voice encoding unit 301, the parameter update unit 205 may exclude the model parameters of the voice SSL model from the update target. Note that, when updating the model parameters of the text decoder that realizes the text decoding unit 303 or the model parameters of the voice SSL model, the parameter update unit 205 may update the entire model parameters, or may use an adapter such as LoRA (Reference 6).

For example, let i = 1, ..., I, the i-th output token be t(i), and the generation probability (posterior probability) of the i-th output token when output tokens t(1) to t(i-1) are obtained be p(t(i)). On the other hand, let the i-th correct token be s(i), and the probability corresponding to the i-th correct token be p(s(i)). In this case, when using the cross-entropy error, the parameter update unit 205 updates the model parameters so as to minimize the sum of -p(s(i))logp(t(i)) for i = 1, ..., I.

By repeatedly executing steps S101 to S105 above, a captioning model in which the trained model parameters are set (i.e., a trained captioning model) is obtained.

<<Generation process of output token probability sequence>>
Hereinafter, details of the process of generating an output token probability sequence in step S103 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the process of generating an output token probability sequence according to the first embodiment.

First, the audio encoding unit 301 of the caption generation unit 203 generates a fixed-length audio vector from the input audio sequence (step S201).

Next, the vector conversion unit 302 of the caption generation unit 203 converts the fixed-length audio vector generated in step S201 above into a text information vector sequence (step S202).

Then, the text decoding unit 303 of the caption generation unit 203 generates an output token probability sequence from the text information vector sequence converted in step S202 above (step S203).

Inference Time Inference time of the caption generation device 10 according to the first embodiment will be described below. In the following, it is assumed that the model parameters of the captioning model have already been learned at the time of inference. Note that, at the time of inference, differences from the time of learning will be mainly described, and descriptions of components similar to those at the time of learning will be omitted.

<Example of functional configuration at the time of inference of the caption generation device 10 according to the first embodiment>
An example of a functional configuration at the time of inference of the caption generation device 10 according to the first embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of a functional configuration at the time of inference of the caption generation device 10 according to the first embodiment.

As shown in FIG. 7, during inference, the caption generation device 10 according to the first embodiment has an input unit 201, a voice conversion unit 202, a caption generation unit 203, and an output unit 208. The output unit 208 is realized, for example, by a process in which one or more programs installed in the caption generation device 10 are executed by the processor 108. Furthermore, during inference, the caption generation device 10 according to the first embodiment has a model parameter storage unit 207.

When audio data for which a caption is to be generated is given, the input unit 201 inputs this audio data.

The voice conversion unit 202 converts the voice data input by the input unit 201 into an input voice sequence.

The caption generation unit 203 is realized by a trained captioning model, and generates an output token sequence from the input speech sequence converted by the speech conversion unit 202.

The output unit 208 outputs a caption composed of an output token sequence generated by the caption generation unit 203 to a predetermined output destination. Note that the output destination is not limited to a specific output destination, and any output destination can be targeted. For example, the output destination can be a storage area of the auxiliary storage device 107, a display device 102 such as a display, other devices connected in a communicable manner, etc.

The model parameter storage unit 207 stores the learned model parameters of the captioning model.

<Caption Generation Process According to the First Embodiment>
The caption generation process according to the first embodiment will be described below with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the caption generation process according to the first embodiment. Note that, in the following, it is assumed that audio data for which a caption is to be generated is provided to the caption generation device 10.

First, the input unit 201 inputs the given voice data (step S301).

Next, the voice conversion unit 202 converts the voice data input in step S301 above into an input voice sequence (step S302).

Next, the caption generation unit 203 generates an output token sequence from the input speech sequence converted in step S302 (step S303). Details of the processing in this step will be described later.

Then, the output unit 208 outputs the caption composed of the output token sequence generated in step S303 above to a predetermined output destination (step S304). This results in a caption that describes in natural language the paralinguistic and non-linguistic information contained in the speech represented by the given audio data.

<<Generation process of output token sequence>>
Hereinafter, details of the process of generating an output token sequence in step S303 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the process of generating an output token sequence according to the first embodiment.

First, the audio encoding unit 301 of the caption generation unit 203 generates an audio fixed-length vector from the input audio sequence (step S401).

Next, the vector conversion unit 302 of the caption generation unit 203 converts the fixed-length audio vector generated in step S401 above into a text information vector sequence (step S402).

Then, the text decoding unit 303 of the caption generating unit 203 generates an output token sequence from the text information vector sequence converted in step S402 (step S403). Note that the output token sequence can be generated, for example, by sampling the output tokens according to each generation probability included in the output token probability sequence.

[Second embodiment]
A second embodiment will be described below. For example, when considering a case where a discrimination model for discriminating paralinguistic and non-linguistic information contained in a voice is applied to the medical field to discriminate illnesses and their symptoms from a voice, it is considered that the discrimination result of the discrimination model requires a justification. In addition, it is considered that the natural language description expressing the justification changes depending on the discrimination result.

In the second embodiment, we therefore explain the case where model parameter learning and caption generation are carried out while taking into account the results of a discrimination model that discriminates paralinguistic and non-linguistic information contained in speech.

In addition, in the second embodiment, the differences from the first embodiment will be mainly described, and the description of the components that are the same as those in the first embodiment will be omitted.

- Learning Time Hereinafter, the learning time of the caption generation device 10 according to the second embodiment will be described.

<Example of functional configuration during learning of the caption generation device 10 according to the second embodiment>
An example of a functional configuration of the caption generation device 10 according to the second embodiment during learning will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of a functional configuration of the caption generation device 10 according to the second embodiment during learning.

As shown in FIG. 10, during learning, the caption generation device 10 according to the second embodiment has, in addition to the various units described in the first embodiment, a recognition unit 209 that is realized by a recognition model that recognizes paralinguistic and non-linguistic information contained in speech. The recognition unit 209 is realized, for example, by a process in which one or more programs installed in the caption generation device 10 are executed by the processor 108.

The identification unit 209 generates an identification result that is a result of identifying paralinguistic and non-linguistic information from the input speech sequence converted by the speech conversion unit 202. The identification model that realizes the identification unit 209 is not limited to a specific identification model, and any identification model that can identify any paralinguistic and non-linguistic information such as illness, gender, emotion, etc. from the input speech sequence can be used. Furthermore, the identification unit 209 may be realized by one identification model or by multiple identification models. When the identification unit 209 is realized by multiple identification models, the identification unit 209 generates an identification result that is a result of identifying each of the multiple pieces of paralinguistic and non-linguistic information from the input speech sequence.

Note that one or more discrimination models that realize the discrimination unit 209 may be pre-trained, or may be trained together with a captioning model. When one or more discrimination models that realize the discrimination unit 209 are trained together with a captioning model, the training data includes, in addition to the audio data and the correct caption, correct paralinguistic and non-linguistic information that indicates the correct discrimination result.

The caption generation unit 203 generates an output token probability sequence from the input speech sequence converted by the speech conversion unit 202 and the classification result generated by the classification unit 209. An example of a detailed functional configuration of the caption generation unit 203 will be described later.

<<Detailed Functional Configuration Example of the Caption Generation Unit 203 According to the Second Embodiment>>
A detailed functional configuration example of the caption generation unit 203 according to the second embodiment will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of a detailed functional configuration of the caption generation unit 203 according to the second embodiment. As in the first embodiment, the following describes, as an example, a case in which a captioning model is composed of three machine learning models, a voice encoder, a conversion network, and text data.

As shown in FIG. 11, the caption generation unit 203 according to the second embodiment is composed of an audio encoding unit 301 realized by an audio encoder, a vector conversion unit 302 realized by a conversion network, and a text decoding unit 303 realized by a text decoder.

The vector conversion unit 302 generates a text information vector sequence using the fixed-length audio vector generated by the audio encoding unit 301 and the classification result generated by the classification unit 209 as input. In other words, the vector conversion unit 302 converts the input fixed-length audio vector and the input classification result into a text information vector sequence. Specifically, the vector conversion unit 302 combines the fixed-length audio vector generated by the audio encoding unit 301 and the classification result generated by the classification unit 209 to create a new fixed-length audio vector, and then generates a text information vector sequence from this new fixed-length audio vector, similar to the vector conversion unit 302 according to the first embodiment. This makes it possible to generate a text information vector sequence that takes into account not only the fixed-length audio vector but also the classification result of the classification model, and as a result, it becomes possible to generate a caption that is composed of an output token sequence that takes these into account.

Note that a vector combining a fixed-length audio vector and a classification result is, for example, an M+L-dimensional vector whose elements are each element of the fixed-length audio vector and the L classification results, in the case where the fixed-length audio vector is an M-dimensional vector, there are L classification results, and all classification results are scalar values that take the two values 0 or 1.

<Learning Process According to the Second Embodiment>
The learning process according to the first embodiment will be described below with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the learning process according to the second embodiment. In the following, online learning is assumed as an example, and steps S501 to S506 in Fig. 12 are repeatedly executed for each piece of learning data.

Steps S501 and S502 in FIG. 12 are similar to steps S101 and S102 in FIG. 5, respectively, and therefore will not be described.

Following step S502, the identification unit 209 generates an identification result that identifies paralinguistic and non-linguistic information from the input speech sequence converted in step S502 (step S503).

Next, the caption generation unit 203 generates an output token probability sequence from the input speech sequence converted in step S502 and the classification result generated in step S503 (step S504). Details of the processing in this step will be described later.

Next, the correct caption conversion unit 204 converts the correct caption included in the learning data input in step S501 into a correct token probability sequence (step S505), similar to step S104 in FIG. 5.

Then, the parameter update unit 205 updates the model parameters using the error between the output token probability sequence generated in the above step S504 and the correct token probability sequence converted in the above step S505, similar to step S105 in FIG. 5 (step S506). However, at this time, if one or more discrimination models realizing the identification unit 209 are also trained, the parameter update unit 205 also uses the error between the correct paralinguistic and non-linguistic information included in the training data input in step S501 and the paralinguistic and non-linguistic information represented by the recognition result generated in the above step S503 to update the learnable parameters of the one or more discrimination models realizing the identification unit 209 in addition to the model parameters.

By repeatedly executing steps S501 to S506 above, a captioning model in which the trained model parameters are set (i.e., a trained captioning model) is obtained.

<<Generation process of output token probability sequence>>
Hereinafter, details of the process of generating an output token probability sequence in step S504 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the process of generating an output token probability sequence according to the second embodiment.

First, the audio encoding unit 301 of the caption generation unit 203 generates an audio fixed-length vector from the input audio sequence (step S601), similar to step S201 in FIG. 6.

Next, the vector conversion unit 302 of the caption generation unit 203 converts the fixed-length audio vector generated in step S601 above and the classification result generated in step S503 of FIG. 14 into a text information vector sequence (step S602).

Then, the text decoding unit 303 of the caption generation unit 203 generates an output token probability sequence from the text information vector sequence converted in step S602 above (step S603), similar to step S203 in FIG. 6.

Inference Time Inference time of the caption generation device 10 according to the second embodiment will be described below. In the following, it is assumed that the model parameters of the captioning model and the learnable parameters of the discriminative model have both been learned at the time of inference. Note that, at the time of inference, differences from the time of learning will be mainly described, and descriptions of components similar to those at the time of learning will be omitted.

<Example of functional configuration at the time of inference of the caption generation device 10 according to the second embodiment>
An example of a functional configuration at the time of inference of the caption generation device 10 according to the second embodiment will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of a functional configuration at the time of inference of the caption generation device 10 according to the second embodiment.

As shown in FIG. 14, during inference, the caption generation device 10 according to the second embodiment has an identification unit 209 in addition to the units described in the first embodiment.

The identification unit 209 generates an identification result by identifying paralinguistic and non-linguistic information from the input speech sequence converted by the speech conversion unit 202.

The caption generation unit 203 is realized by a trained captioning model, and generates an output token sequence from the input speech sequence converted by the speech conversion unit 202 and the classification result generated by the classification unit 209.

<Caption Generation Process According to the Second Embodiment>
The caption generation process according to the second embodiment will be described below with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the caption generation process according to the second embodiment. Note that, in the following, it is assumed that audio data for which a caption is to be generated is provided to the caption generation device 10.

Steps S701 and S702 in FIG. 15 are similar to steps S301 and S302 in FIG. 8, respectively, and therefore will not be described.

Following step S702, the identification unit 209 generates an identification result that identifies paralinguistic and non-linguistic information from the input speech sequence converted in step S702 (step S703).

Next, the caption generation unit 203 generates an output token sequence from the input speech sequence converted in step S702 above and the identification result generated in step S703 above (step S704).

Then, the output unit 208 outputs the caption composed of the output token sequence generated in step S704 above to a predetermined output destination (step S705), similar to step S304 in FIG. 8. This allows for the acquisition of a caption that describes in natural language the paralinguistic and non-linguistic information contained in the speech represented by the given speech data, taking into account the identification results obtained from the speech data using the identification model.

<<Generation process of output token sequence>>
Details of the output token sequence generation process in step S704 above will be described below with reference to Fig. 16. Fig. 16 is a flowchart showing an example of the output token sequence generation process according to the second embodiment.

First, the audio encoding unit 301 of the caption generation unit 203 generates an audio fixed-length vector from the input audio sequence (step S801), similar to step S401 in FIG. 9.

Next, the vector conversion unit 302 of the caption generation unit 203 converts the fixed-length audio vector generated in step S801 above and the classification result generated in step S703 of FIG. 15 into a text information vector sequence (step S802).

Then, the text decoding unit 303 of the caption generation unit 203 generates an output token sequence from the text information vector sequence converted in step S802 above (step S803), similar to step S403 in FIG. 9.

[summary]
As described above, the caption generation device 10 according to the first embodiment learns a captioning model from learning data including voice data and correct captions that describe in natural language the paralinguistic and non-linguistic information contained in the voice represented by the voice data. The caption generation device 10 according to the first embodiment can then generate captions that describe in natural language the paralinguistic and non-linguistic information contained in the voice represented by the given voice data, using the learned captioning model. This makes it possible to output the paralinguistic and non-linguistic information contained in the voice in the form of a caption that has high interpretability.

The caption generation device 10 according to the second embodiment also learns a captioning model using the results of a discrimination model that discriminates paralinguistic and non-linguistic information from the speech represented by the audio data. The caption generation device 10 according to the second embodiment can then use the learned captioning model and discrimination model to generate a caption for given audio data and generate paralinguistic and non-linguistic information as discrimination results. This makes it possible to obtain the paralinguistic and non-linguistic information contained in the speech and its caption.

The present invention is not limited to the specifically disclosed embodiments above, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[References]
Reference 1: Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor, et al., "Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis," in ICML, 2018.
Reference 2: Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed, "HuBERT: Self-supervised speech representation learning by masked prediction of hidden units," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451-3460, 2021.
Reference 3: Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al., "WavLM: Large-scale self-supervised pre-training for full stack speech processing," IEEE J-STSP, vol. 16, no. 6, pp. 1505-1518, 2022.
Reference 4: R. Mokady, A. Hertz, and A. H. Bermano, "ClipCap: CLIP prefix for image captioning," arXiv preprint arXiv: 2111.09734, 2021.
Reference 5: T. Brown, B. Mann, N. Ryder, et al., "Language models are few-shot learners," in Proc. NeurIPS, 2020.
Reference 6: E. J Hu, Y. Shen, P. Wallis, et al., "Language models are few-shot learners," in Proc. ICLR, 2022.

10 Caption generating device 101 Input device 102 Display device 103 External I/F
103a Recording medium 104 Communication I/F
105 RAM
106 ROM
107 Auxiliary storage device 108 Processor 109 Bus 201 Input unit 202 Voice conversion unit 203 Caption generation unit 204 Correct caption conversion unit 205 Parameter update unit 206 Learning data storage unit 207 Model parameter storage unit 208 Output unit 209 Classification unit

Claims (8)

an input unit for inputting learning data including speech data and training data representing a natural language description of paralinguistic information or non-linguistic information included in the speech represented by the speech data;
a calculation unit that uses an input of a speech sequence obtained by converting the speech data into a predetermined format for each predetermined unit, and calculates the generation probability of a natural language description of paralinguistic information or non-linguistic information contained in the speech, which is represented by the speech data that is the source of the input speech sequence, using a machine learning model that calculates the generation probability of a natural language description of paralinguistic information or non-linguistic information contained in the speech;
an update unit that updates a learnable parameter of the machine learning model based on an error between a natural language description generated according to the generation probability and teacher data included in the training data;
A learning device having the above configuration. The calculation unit is
converting an input speech sequence into fixed-length first information representative of a feature of the speech sequence;
Converting the first information into a string of second information representing word embedding expressions that reflect characteristics of the first information;
The learning device according to claim 1 , further comprising: a generation probability of a character sequence for each predetermined unit included in the natural language description, the generation probability being calculated from the second information string. a discrimination unit that uses the speech sequence as an input and discriminates paralinguistic information or non-linguistic information contained in speech represented by the speech data that is the source of conversion of the input speech sequence using a discrimination model that discriminates paralinguistic information or non-linguistic information contained in speech,
The calculation unit is
The learning device according to claim 2, wherein third information consisting of the first information and information representing paralinguistic information or non-linguistic information identified by the identification unit is converted into a string of second information representing word embedding expressions reflecting characteristics of the third information. a generation unit that receives as input a speech sequence obtained by converting given speech data into a predetermined format for each predetermined unit, and generates a natural language description of paralinguistic information or non-linguistic information contained in the speech represented by the speech data from which the input speech sequence is converted, using a trained machine learning model that generates a natural language description of paralinguistic information or non-linguistic information contained in the speech;
A generating device having the following: a discrimination unit that uses the speech sequence as an input and discriminates paralinguistic information or non-linguistic information contained in speech represented by the speech data that is the source of conversion of the input speech sequence using a discrimination model that discriminates paralinguistic information or non-linguistic information contained in speech,
The generation unit is
The generating device according to claim 4 , further comprising: a generating section configured to generate a natural language description of the paralinguistic information or the non-linguistic information based on the paralinguistic information or the non-linguistic information identified by the identifying section. an input step of inputting learning data including speech data and training data representing a natural language description of paralinguistic information or non-linguistic information contained in the speech represented by the speech data;
a calculation step of calculating the probability of generating a natural language description of paralinguistic information or non-linguistic information contained in speech represented by the speech data from which the input speech sequence is converted, using a machine learning model that calculates the probability of generating a natural language description of paralinguistic information or non-linguistic information contained in speech, the natural language description being a source of conversion of the input speech sequence, using a speech sequence obtained by converting the speech data into a predetermined format for each predetermined unit as an input;
an update step of updating a learnable parameter of the machine learning model based on an error between the natural language description generated according to the generation probability and teacher data included in the training data;
The computer executes the learning method. a generation step of using as input a speech sequence obtained by converting given speech data into a predetermined format for each predetermined unit, and generating a natural language description of the paralinguistic or non-linguistic information contained in the speech, which is represented by the speech data from which the input speech sequence is converted, using a trained machine learning model that generates a natural language description of the paralinguistic or non-linguistic information contained in the speech;
A computer executed generation method. A program that causes a computer to function as a learning device according to any one of claims 1 to 3, or as a generating device according to claim 4 or 5.

Images