WO2024261877A1 - Learning device, training method, and training program - Google Patents

Abstract

A learning device (10) according to an embodiment has a loss function calculation unit (152) and an update unit (153). The loss function calculation unit (152) calculates, for a model that outputs an estimated value on the basis of attention between a plurality of pieces of time-series data having different modalities, a loss function relating to the magnitude of attention at different times between the pieces of time-series data. The update unit (153) updates parameters of the model using the loss function so that attention at different times between the pieces of time-series data decreases.

Description

Learning device, learning method, and learning program

The present invention relates to a learning device, a learning method, and a learning program.

　Conventionally, a technology using a Transformer Encoder is known as an estimator that estimates corresponding output information from a multimodal time series (see, for example, Non-Patent Document 1). Non-Patent Document 1 describes a method in which multiple input feature time series are combined in the time axis direction and input to a model including a Transformer Encoder.

When training a model in the technology described in Non-Patent Document 1, training data consisting of a multimodal time series and corresponding correct output information is used. The model is trained based on the criterion of minimizing the error function between the estimation result by the model and the correct answer. Meanwhile, during inference, the multimodal time series is input into the trained model to obtain an estimation result.

Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid, “VideoBERT: A Joint Model for Video and Language “Representation Learning,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 7464-7473.

However, conventional techniques have the problem that it can be difficult to train a model accurately when the amount of training data is small.

In models that handle multimodal time series, the relationship between feature time series corresponding to each modality at the same time is important. For example, to estimate communication skills from video, relationships such as gestures during speech and synchronization of facial expressions between the speaker and listener are important. In this case, "gestures during speech," "speaker's facial expression," and "listener's facial expression" correspond to each modality.

In conventional technology, the attention matrix of the Transformer Encoder is used to model the relationships between sequential data. However, when the amount of training data is limited, it is difficult to train a model using conventional technology. This is because conventional technology requires learning a high-dimensional attention matrix from the training data alone.

In order to solve the above-mentioned problems and achieve the objective, the learning device is characterized by having a loss function calculation unit that calculates a loss function related to the magnitude of attention at different time steps between multiple time series data of different modalities for a model that outputs an estimated value based on attention between the time series data, and an update unit that uses the loss function to update the parameters of the model so that the attention at different time steps between the time series data is reduced.

According to the present invention, it is possible to accurately train a model even when the amount of training data is small.

FIG. 1 is a diagram illustrating an example of the configuration of a learning device according to the first embodiment. FIG. 2 is a diagram illustrating the structure of the model. FIG. 3 is a diagram for explaining time steps corresponding to the same time between modalities. FIG. 4 is a diagram illustrating the attention penalty matrix. FIG. 5 is a flowchart showing the flow of processing by the learning device. FIG. 6 is a diagram illustrating an example of a computer that executes a program.

Below, embodiments of the learning device, learning method, and learning program according to the present application are described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments described below.

[Configuration of the first embodiment]
The configuration of the learning device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the learning device according to the first embodiment.

The learning device 10 learns a machine learning model (hereinafter, simply called the model). The model outputs an estimation result based on the features of the input data. The model is, for example, Transformer (Reference 1: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.).

In this embodiment, multiple time series data of different modalities are input to the model. The model outputs an estimated value based on attention between the multiple time series data. For example, the estimated value is a binary label. Note that the estimated value may be a discrete value other than binary, or may be a continuous quantity. The model may also be referred to as an estimator.

Traditionally, in model learning, the model parameters are updated to reduce the error between the estimated value output by the model and the correct estimated value. In this embodiment, further, based on the prior knowledge that "the relationship between corresponding data at the same time in multiple feature time series is important," a loss function (attention penalty loss) is introduced that induces an attention matrix to associate data that correspond to the same time.

As a result, according to this embodiment, by utilizing prior knowledge, it is possible to accurately train a model using a limited amount of training data, i.e., to improve the accuracy of a trained model.

The model of this embodiment will be explained using Figure 2. Figure 2 is a diagram explaining the structure of the model. Note that the model shown in Figure 2 is an example, and this embodiment can also be applied to other models.

The model is used to estimate the merits of a salesperson's speech based on video and audio recorded during a conversation between two speakers, a salesperson (seller) and a customer (buyer). The model is input with data X shown in equation (1).

X _ss is voice data of a salesperson (seller's speech), X _vs is video data of a salesperson (seller's video), and X _vb is video data of a customer (buyer's video). Each data is represented as a scalar or a vector.

Note that each piece of video data and audio data is assumed to be data extracted at a time corresponding to a certain section of speech in which the salesperson speaks. Note that, since the customer is often silent while the salesperson is speaking, the customer's audio data is not used here.

The model outputs a binary label l. Label l being "0" means that the salesperson's speech does not show sufficient empathy. Label l being "1" means that the salesperson's speech shows sufficient empathy.

In other words, if label l is "1", the corresponding salesperson's utterance is worthy of praise. In that case, it is presumed that there was something good about that utterance.

The audio encoder extracts a time-series feature quantity Z _ss from data X _ss . The audio encoder extracts a time-series feature quantity Z _vs from data X _vs. The video encoder extracts a time-series feature quantity Z _vb from data X _vb . The audio encoder and the video encoder can extract the features using, for example, a pre-trained neural network.

The model further transforms the features using a Transformer Encoder layer, and outputs the label l through a Pooling layer and a Softmax layer. The Transformer Encoder layer includes N hidden layers (e.g., 6). The Transformer Encoder layer may also have H heads (e.g., 16). Each hidden layer and each header is provided with a corresponding attention mechanism. Here, N and H are positive integers.

Salesperson voice data _Xss , salesperson video data _Xvs , and customer video data _Xvb are data with different modalities. Similarly, feature _Zss , feature Zvs, and feature Zvb are feature amounts with different modalities. Using the subscript of X of each data, the modality is expressed as mε{ss, _vs , vb}. In this case, ZmεR ^D×Tm . That is, feature _Zm is a matrix of D× _Tm , where _D is the dimension of the feature. Also, _Tm is the length in the time direction of the input _{data Xm} .

Returning to FIG. 1, the learning device 10 has a communication unit 11, an input unit 12, an output unit 13, a memory unit 14, and a control unit 15.

The communication unit 11 communicates data with other devices via a network. For example, the communication unit 11 is a NIC (Network Interface Card).

The input unit 12 accepts data input. For example, the input unit 12 is an interface that is connected to input devices such as a mouse and a keyboard.

The output unit 13 outputs data. The output unit 13 is an interface that is connected to an output device such as a display and a speaker.

The memory unit 14 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or an optical disk. The memory unit 14 may also be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM), flash memory, or non-volatile static random access memory (NVSRAM). The memory unit 14 stores the operating system (OS) and various programs executed by the learning device 10.

The storage unit 14 stores model information 141. The model information 141 is parameters for constructing a model. For example, the model information 141 is parameters such as the weight matrix of each layer included in the Transformer.

The control unit 15 controls the entire learning device 10. The control unit 15 is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

The control unit 15 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory. The control unit 15 also functions as various processing units as the various programs run. For example, the control unit 15 functions as an estimation unit 151, a loss function calculation unit 152, and an update unit 153.

The estimation unit 151 uses the model to calculate an estimate for the input data. For example, the estimation unit 151 estimates a label l from the input data X _ss , data X _vs , and data X _vb .

The loss function calculation unit 152 calculates a loss function. The loss function calculation unit 152 calculates a loss function that can optimize both the estimation error and the attention penalty error of the model.

The estimation error is represented by the error between the estimated value of the label and the correct answer. The attention penalty error is an error related to the magnitude of attention at different time steps in the time series data. In particular, the attention penalty error will be explained in detail later. The method of calculating the loss function by the loss function calculation unit 152 will be explained below.

The loss function calculation unit 152 calculates the loss function using the attention matrix that the model defines in the process of calculating the estimated value. First, we will explain the attention matrix.

The model configures an input vector Z by linking the feature amount Z _ss , the feature amount Z _vs , and the feature amount Z _vb in the time axis direction. However, each element of the vector Z may be a matrix.

Next, the Transformer Encoder models the association between the time steps of the input vector using the attention matrix A. The attention matrix A is the scaled dot product of the query matrix Q and the key matrix K. The query matrix Q and the key matrix K are calculated by multiplying the vector of each time step of the input vector by a weight matrix. Therefore, the query matrix Q and the key matrix K can be considered to be composed of submatrices corresponding to the data of each modality, as shown in equations (3) and (4).

The weighting matrices for calculating the query matrix Q and the key matrix K are parameters to be learned in each hidden layer of the Transformer Encoder. In other words, the model information 141 includes the weighting matrices. The update unit 153 updates the weighting matrices.

Each hidden layer of the Transformer Encoder receives the vector output from the previous hidden layer. Then, in each hidden layer, the Transformer Encoder multiplies the input vector by a weight matrix to calculate the query matrix Q and key matrix K.

Here, for data m of each modality, _Qm , _Km ∈ ^Rd×Tm . That is, _Qm , _Km are matrices of d× _Tm , where d is the dimension of the query matrix and the key matrix, and _Tm is the length of the input data _Xm in the time direction.

The attention matrix A, defined by the scaled dot product of the query matrix Q and the key matrix K, is constructed from a block matrix as shown in equation (5).

where softmax(·) is the softmax function. The block matrix A _m1,m2 ∈ ^{R Tm1×Tm2} can be regarded as the attention from modality _m1 to modality _m2 .

Attention between modalities is expressed by a non-diagonal block matrix A _m1,m2 (m ₁ ≠m ₂ ). As shown in Fig. 3, the diagonal elements (diagonal straight lines parallel to the diagonal of the rectangle) of the non-diagonal block matrix correspond to time steps at the same time between different modalities. Fig. 3 is a diagram for explaining time steps corresponding to the same time between modalities.

The off-diagonal block matrix A _vs,ss corresponds to the region in which the vertical direction (Q) is VS and the horizontal direction (K) is SS among the nine rectangular regions constituting the rectangle in Fig. 3. A straight line is drawn on the diagonal line of this region. This is because the diagonal elements of the block matrix A _vs,ss correspond to the same time step between different modalities.

On the other hand, the non-diagonal block matrix A _ss,ss corresponds to the region in which the vertical direction (Q) is SS and the horizontal direction (K) is SS among the nine rectangular regions constituting the rectangle in Fig. 3. No straight lines are drawn on the diagonal lines of this region. This is because the diagonal elements of the block matrix A _ss,ss correspond to the same time step between the same modality.

From this, if the diagonal components of the non-diagonal block matrix A _m1,m2 (m ₁ ≠ m ₂ ) become dominant, attention at different times between time series data of different modalities will be small. Since the magnitude of attention is relative, in other words, if the diagonal components of the non-diagonal block matrix A _m1,m2 (m ₁ ≠ m ₂ ) become dominant, attention at the same time between time series data of different modalities will be large.

In other words, the model will tend to follow the prior knowledge that "the relationship between corresponding data at the same time in multiple feature time series is important." Note that attention can also be rephrased as the strength of the relationship between data.

Therefore, the loss function calculation unit 152 calculates a loss function that becomes smaller as the diagonal components of the non-diagonal block matrix A _m1,m2 (m ₁ ≠m ₂ ) become more dominant.

First, the loss function calculation unit 152 defines the attention penalty matrix shown in equations (6) and (7).

Here, W ^m1,m2 _i,j is the (i,j)-th element of the block matrix W ^m1,m2 ∈ R ^Tm1×Tm2 . σ is a hyperparameter. Also, according to equation (7), the off-diagonal elements take positive values.

Here, T _m1 and T _m2 are the numbers of time slots for modality m ₁ and modality m ₂ , respectively. In this way, the number of time slots may differ for each modality. However, at least within the range in which the attention penalty matrix is defined, the total length of time of the time slots is common between the modalities.

For example, consider a case where salesperson voice data X _ss (modality ss) and salesperson video data X _vs (modality vs) are both acquired for 12 seconds (time t=0 to time t=12). The number of time slots T _ss for modality ss is 12. Meanwhile, the number of time slots T _vs for modality vs is 4.

In other words, the time slot of modality ss includes 12 intervals, from time t = 0 to 1 second, 1 to 2 seconds, ..., 11 to 12 seconds. On the other hand, the time slot of modality vs includes 4 intervals, from time t = 0 to 3 seconds, 3 to 6 seconds, 6 to 9 seconds, and 9 to 12 seconds.

According to equation (7), when (i,j)=(12,4), (i/T _m1 -j/T _m2 ) ² in the numerator in the parentheses of exp becomes (12/12 - 4/4) ² =0, and W ^ss,vs _12,4 becomes 0, the smallest value that W ^m1,m2 _i,j can take. Note that the components corresponding to (i,j)=(12,4) are diagonal components.

According to equation (7), when (i,j)=(11,4), the numerator in the parentheses of exp, (i/T _m1 -j/T _m2 ) ² , is a small value, (11/12 - 4/4) ² = 1/144, and W ^ss,vs _11,4 is the smallest value among the possible values of W ^m1,m2 _i,j . Note that the components corresponding to (i,j)=(11,4) are off-diagonal components close to the diagonal components.

According to equation (7), when (i,j)=(3,4), the numerator in the parentheses of exp, (i/T _m1 -j/T _m2 ) ² , is a large value of (3/12 - 4/4) ² = 81/144, and W ^ss,vs _3,4 is the larger of the possible values of W ^m1,m2 _i,j . Note that the components corresponding to (i,j)=(3,4) are off-diagonal components far from the diagonal components.

In this way, the values of the diagonal components of the off-diagonal block matrix of the attention penalty matrix are 0, and the values increase as they move away from the diagonal components. For this reason, when the attention penalty matrix is represented as a heat map, a gradation appears as shown in Figure 4. Figure 4 is a diagram explaining the attention penalty matrix.

The loss function calculation unit 152 sums up the losses of each head hε{1, . . . , H} of each layer nε{1, . . . , N} of the Transformer, and calculates the loss function L _sg as shown in equation (8).

Furthermore, the loss function calculation unit 152 calculates the loss function L as shown in equation (9) together with the loss function L _label representing the estimation error, that is, the error between the estimation result and the correct label.

λ _sg is an L _sg regularization parameter for L _label , and is set in advance as a hyperparameter. In addition, L _label is, for example, a cross-entropy function.

In this way, the loss function calculation unit 152 calculates a loss function related to the magnitude of attention at different times in the time series data for a model that outputs an estimate based on attention between multiple time series data of different modalities.

The update unit 153 uses a loss function to update the model parameters so that attention at different times in the time series data is reduced.

FIG. 5 is a flowchart showing the processing flow of the learning device. As shown in FIG. 5, first, the learning device 10 initializes parameters (step S101). The parameters here are, for example, a weight matrix of Transformer, and are included in the model information 141.

Then, the learning device 10 repeats the process of step S2 and updates the parameters until a condition is met. The condition is that the number of iterations exceeds a certain number, the amount of parameter update falls below a threshold, etc.

The learning device 10 performs forward calculation (step S102). That is, the learning device 10 uses a model to calculate estimates for multiple input time-series data with different modalities.

Next, the learning device 10 calculates an estimation error based on the estimation result (step S103). The estimation error is, for example, L _label in equation (9).

The learning device 10 also calculates an attention penalty loss based on the attention matrix (step S104). The attention penalty error is, for example, L _sg in equation (9).

Then, the learning device 10 performs back propagation (step S105) so that the loss function combining the estimation error and the attention penalty error (e.g., L in equation (9)) is optimized, and updates the model parameters (step S106).

[Effects of the First Embodiment]
As described above, the loss function calculation unit 152 calculates a loss function relating to the magnitude of attention at different times in the time series data for a model that outputs an estimated value based on attention between multiple time series data of different modalities. The update unit 153 uses the loss function to update the parameters of the model so that the attention at different times in the time series data is reduced.

In this way, the learning device 10 can generate in the model a tendency for the relationship between corresponding data at the same time to become stronger in the feature time series by performing learning using the attention penalty error. As a result, according to this embodiment, the model can be trained with high accuracy even when the amount of training data is small.

The loss function calculation unit 152 also calculates an attention matrix, each component of which corresponds to a combination of two modalities and a combination of two times, based on multiple time series data, and assigns weights to each component of the attention matrix so that the weight increases the further apart the corresponding two times are, and calculates a loss function including the weighted attention matrix. The update unit 153 also updates the model parameters so that each component of the weighted attention matrix becomes smaller.

In particular, the loss function calculation unit 152 calculates the scaled dot product of the query matrix and key matrix input to one or more layers included in the Transformer model as an attention matrix, multiplies each element of the off-diagonal block matrix that constitutes the attention matrix by an attention penalty matrix that weights elements more the further apart the corresponding two times are, and calculates a loss function that includes the norm of the product of the attention matrix and the attention penalty matrix.

Here, as described in formula (5), each block matrix of the attention matrix A corresponds to a combination of two modalities. Also, each component of the block matrix corresponds to a combination of time represented by a time slot. For example, the block matrix A _vs,ss corresponds to a combination of modality vs and modality ss. Also, each component of the block matrix A _vs,ss corresponds to a combination of two time slots assigned to rows and columns, respectively.

Also, as explained in equations (6), (7), and (8), the attention matrix A is weighted by the attention penalty matrix W so that the greater the distance between two times (e.g., the i-th time slot and the j-th time slot) the greater the weight.

As a result, this embodiment allows for highly accurate training, particularly for the Transformer.

[System configuration, etc.]
In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of them can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[program]
In one embodiment, the learning device 10 can be implemented by installing a program that executes the above learning process as package software or online software on a desired computer. For example, the above program can be executed by an information processing device, causing the information processing device to function as the learning device 10. The information processing device here includes desktop or notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).

The learning device 10 can also be implemented as a server device that provides services related to the above-mentioned processing to a client, with the terminal device used by the user as the client. For example, the server device is implemented as a server device that provides a service that takes the parameters of the model before the update as input and outputs the parameters of the model after the update. In this case, the server device may be implemented as a web server, or may be implemented as a cloud that provides services related to the above-mentioned processing by outsourcing.

FIG. 6 is a diagram showing an example of a computer that executes a program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these components is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the learning device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to the functional configuration of the learning device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

Furthermore, the setting data used in the processing of the above-mentioned embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary, and executes the processing of the above-mentioned embodiment.

The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a LAN (Local Area Network), WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

REFERENCE SIGNS LIST 10 Learning device 11 Communication unit 12 Input unit 13 Output unit 14 Storage unit 15 Control unit 141 Model information 151 Estimation unit 152 Loss function calculation unit 153 Update unit

Claims (5)

a loss function calculation unit that calculates a loss function relating to the magnitude of attention at different times between a plurality of time-series data of different modalities for a model that outputs an estimated value based on attention between the time-series data;
an update unit that updates parameters of the model by using the loss function so that attention at different times in the time series data is reduced;
A learning device comprising: the loss function calculation unit calculates an attention matrix, each component of which corresponds to a combination of two modalities and a combination of two times, based on the plurality of time series data, weights each component of the attention matrix so that the weight increases as the distance between the two corresponding times increases, and calculates the loss function including the weighted attention matrix;
The learning device according to claim 1 , wherein the update unit updates the parameters of the model so that each component of the weighted attention matrix becomes smaller. The learning device described in claim 2, characterized in that the loss function calculation unit calculates the attention matrix as a scaled dot product of a query matrix and a key matrix input to one or more layers included in the model which is a Transformer, multiplies each element of a non-diagonal block matrix that constitutes the attention matrix by an attention penalty matrix that assigns a weighting that is larger the greater the distance between two corresponding times, and calculates the loss function including the norm of the product of the attention matrix and the attention penalty matrix. A learning method performed by a learning device, comprising:
a loss function calculation step of calculating a loss function relating to the magnitude of attention at different times in a plurality of time series data of different modalities for a model that outputs an estimated value based on attention between the time series data;
an updating step of updating parameters of the model using the loss function so that attention at different times in the time series data is reduced;
A learning method comprising: a loss function calculation step of calculating a loss function relating to the magnitude of attention at different times in a plurality of time series data of different modalities for a model that outputs an estimated value based on attention between the time series data;
an updating step of updating parameters of the model using the loss function so that attention at different times in the time series data is reduced;
A learning program characterized by causing a computer to execute the above.

Images