WO2021095213A1 - Learning method, learning program, and learning device - Google Patents

Abstract

This learning device (100) extracts a feature quantity from first modal information using an extraction model (111). The learning device (100) converts the extracted feature quantity using a conversion model (112) on the basis of a plurality of parameters and thereby acquires a new feature quantity. The learning device (100) inputs the acquired new feature quantity to a first processing model (121) and thereby acquires a first output value. The learning device (100) inputs other feature quantities extracted from second modal information to a second processing model (122) and thereby acquires a second output value. The learning device (100) inputs the acquired first and second output values to a third processing model (123) and thereby acquires a third output value. The learning device (100) updates the plurality of parameters on the basis of the acquired third output value.

Description

Learning methods, learning programs, and learning devices

The present invention relates to a learning method, a learning program, and a learning device.

Conventionally, there is a technology to solve problems such as document translation, question and answer, object detection, and situation judgment using predetermined modal information. For example, there is a model called BERT (Bidirectional Encoder Representations from Transformers) for solving a problem using modal information about a document. There is also a model that extends BERT to solve problems using modal information about images. Here, the modal is a concept indicating the style and type of information, and specific examples thereof include images, documents (texts), and sounds. Machine learning using multiple modals is called multimodal learning. Further, among multimodal learning, learning the co-occurrence relationship between a plurality of modals is sometimes called cross-modal learning.

As the prior art, for example, there is a method in which a gesture feature is input and a model for classifying whether or not the input gesture feature corresponds to a word is generated. Further, for example, a word co-occurrence vector whose element is the frequency of occurrence of word co-occurrence words appearing near a word and an image co-occurrence vector whose element is the frequency of appearance of image co-occurrence words appearing near an image are used. There is a technique for determining the similarity between words and images. Further, for example, there is a technique of performing recognition processing of sign language elements, which are individual components of sign language movements, from time-series data of hand movements, and performing sign language word recognition processing from the recognized sign language elements.

JP-A-2018-163400 Japanese Unexamined Patent Publication No. 2002-132823 Japanese Unexamined Patent Publication No. 9-34863

However, with the conventional technology, it is difficult to obtain a useful model for extracting features from modal information when solving a problem using modal information. For example, when dealing with modal information about images, the model does not have an expression space that expresses features based on the relationship between modal information about images and modal information about languages, and is not useful in solving problems. May not be available.

In one aspect, the present invention aims to learn a useful model for extracting features from modal information.

According to one embodiment, a feature amount is extracted from the information of the first modal, and the extracted feature amount is converted based on a parameter to acquire a new feature amount, and the acquired new feature amount is obtained. By inputting the quantity into the first processing model for the first modal, the first output value is obtained and other features extracted from the information of the second modal different from the first modal. By inputting the amount into the second processing model relating to the second modal, a second output value is acquired, and the acquired first output value and the second output value are combined with the first output value. A learning method in which a third output value is acquired by inputting into a third processing model relating to the modal of the above and the second modal, and the parameter is updated based on the acquired third output value. A learning program and a learning device are proposed.

According to one aspect, it becomes possible to learn a useful model for extracting features from modal information.

FIG. 1 is an explanatory diagram showing an embodiment of a learning method according to an embodiment. FIG. 2 is an explanatory diagram showing an example of the information processing system 200. FIG. 3 is a block diagram showing a hardware configuration example of the learning device 100. FIG. 4 is a block diagram showing a functional configuration example of the learning device 100. FIG. 5 is a block diagram showing a specific functional configuration example of the learning device 100. FIG. 6 is an explanatory diagram showing an example of learning the integrated model 600. FIG. 7 is an explanatory diagram showing an example of converting the image feature quantity sequence F. FIG. 8 is an explanatory diagram showing a specific example of the calculation for converting the image feature quantity sequence F. FIG. 9 is an explanatory diagram showing another example of converting the image feature quantity sequence. FIG. 10 is an explanatory diagram showing an example in which the learning device 100 uses the integrated model 1000. FIG. 11 is a flowchart showing an example of the learning processing procedure.

Hereinafter, embodiments of the learning method, learning program, and learning device according to the present invention will be described in detail with reference to the drawings.

(An example of a learning method according to an embodiment)
FIG. 1 is an explanatory diagram showing an embodiment of a learning method according to an embodiment. The learning device 100 is a computer for learning a useful model for extracting a feature amount from a predetermined modal information, which can be used when solving a problem using a predetermined modal information.

Conventionally, for example, there is a pre-learning model (pre-train model) called BERT for solving a problem. Specifically, the BERT is formed by stacking the Encoder portions of the Transformer. For BERT, for example, the following Non-Patent Document 1 can be referred to.

Non-Patent Document 1: Devlin, Jacob et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." NAACL-HLT (2019).

Here, BERT is supposed to be applied to a situation where a problem is solved using modal information about a language, and cannot be applied to a situation where a problem is solved using a plurality of modal information. ..

On the other hand, for example, there is an extended model called VideoBERT that extends BERT so that it can be applied to situations where problems are solved using modal information related to languages as well as modal information related to images. In addition, there is an extended model called CBT (Contrastive Biraditional Transformer for Temporal Representation Learning), which has improved performance over VideoBERT.

Specifically, CBT integrates the output of a language processing model that learns the co-occurrence relationship of language features, an image processing model that learns the co-occurrence relationship of image features, and a language processing model and an image processing model. It is formed by a cross-modal processing model that learns the co-occurrence relationship between language and images. For CBT, for example, the following Non-Patent Document 2 can be referred to.

Non-Patent Document 2: Sun, Chen, et al. "Contrastive Biraditional Transition Former for Temporal Representation Learning." ArXiv preprint arXiv: 1906.05743 (2019).

However, the various models mentioned above may not be useful models for solving problems. For example, the various models described above may not be useful models for extracting features from modal information when solving a problem using modal information. For example, when dealing with modal information about images, CBT does not have an expression space that expresses features based on the relationship between modal information about images and modal information about languages, which is useful in solving problems. It may not be a model.

Specifically, the image feature amount is an expression space that reflects modal information about the image, and has the property of not having an expression space that reflects modal information about the language. Therefore, even if CBT is pre-learned, the image processing model included in CBT does not become a model that can effectively utilize modal information about the language, and does not become a useful model for solving the problem. Further, the image feature amount has a property that even if the same object is captured, the image feature amount can be different for images of different appearances. For this reason, when reflecting the characteristics of modal information related to the language with respect to the image feature amount, it is necessary to update not one image feature amount but various image feature amount expressions, which makes effective update difficult or difficult. , It is possible that it will have an adverse effect on solving the problem.

Therefore, in the present embodiment, by setting the parameters used when converting the modal information and providing the expression space by the parameters, it is possible to learn a useful model for extracting the feature amount from the modal information. The method will be described.

In FIG. 1, the learning device 100 has, for example, a model 101. The model 101 has an extraction model 111, a conversion model 112, a first processing model 121, a second processing model 122, and a third processing model 123. The extraction model 111, the conversion model 112, and the first processing model 121 relate to the first modal. The second processing model 122 relates to a second modal. The third processing model 123 relates to a first modal and a second modal.

The learning device 100 acquires the information of the first modal and the information of the second modal. Modal means a form of information. The first modal and the second modal are different modals. The first modal is, for example, an image modal. If the first modal is about an image, the information in the first modal is, for example, an image. The second modal is, for example, a modal related to language. If the second modal is about language, the information in the second modal is, for example, a document.

(1-1) The learning device 100 uses the extraction model 111 to extract the feature amount from the first modal information. The learning device 100 extracts, for example, an image feature amount from an image. The image feature amount is represented by, for example, a vector indicating an array.

(1-2) The learning device 100 acquires a new feature amount by converting the extracted feature amount based on the parameter using the conversion model 112. There are a plurality of parameters, for example. The learning device 100 calculates, for example, a correction amount for correcting the extracted image feature amount based on the extracted image feature amount and a plurality of parameters, and adds the correction amount to the extracted image feature amount to obtain a new image feature. Get the quantity.

(1-3) The learning device 100 acquires the first output value by inputting the acquired new feature amount into the first processing model 121. The first processing model 121 is, for example, an image processing model. The learning device 100 acquires the first output value by inputting a new image feature amount into the image processing model, for example.

(1-4) The learning device 100 acquires a second output value by inputting another feature amount extracted from the second modal information into the second processing model 122. The second processing model 122 is, for example, a language processing model. The learning device 100 acquires a second output value by inputting, for example, a language feature amount extracted from a document into a language processing model. Language features are represented, for example, by vectors representing arrays.

(1-5) The learning device 100 acquires the third output value by inputting the acquired first output value and the second output value into the third processing model 123. The third processing model 123 is, for example, a cross-modal processing model. The cross-modal processing model integrates information from multiple modals and learns co-occurrence of information from multiple modals. The learning device 100 acquires the third output value by inputting the first output value and the second output value into the cross-modal processing model, for example.

(1-6) The learning device 100 updates a plurality of parameters based on the acquired third output value. The learning device 100 updates a plurality of parameters by the error back propagation method, for example, based on the third output value. The learning device 100 may output a plurality of updated parameters. As a result, the learning device 100 can obtain a plurality of parameters useful from the viewpoint of handling the first modal information and the second modal information in solving the problem.

The learning device 100 is, for example, explicitly prepared with a plurality of parameters capable of reflecting the features of the information of the second modal, and when acquiring a new feature amount from the information of the first modal, the second It is possible to effectively utilize the characteristics of modal information. Further, the learning device 100 does not directly reflect the features of the information of the second modal in the feature amount itself extracted from the information of the first modal, and can reduce the adverse effect when solving the problem.

Here, the case where the learning device 100 updates a plurality of parameters based on the third output value has been described, but the present invention is not limited to this. For example, the learning device 100 may further update the first processing model 121 based on the third output value. As a result, the learning device 100 can obtain a first processing model 121 that is useful from the viewpoint of handling the first modal information in solving the problem. Then, the learning device 100 can use the model 101 when solving a problem by using the information of the first modal and the information of the second modal, and can improve the accuracy of the obtained solution.

Here, the learning device 100 may separate the extraction model 111, the conversion model 112, and the first processing model 121 from the model 101. According to this, the learning device 100 can obtain a useful combination model of the extraction model 111, the conversion model 112, and the first processing model 121. Then, the learning device 100 uses the separated extraction model 111, the conversion model 112, and the first processing model 121 when solving the problem using the first modal information, and the accuracy of the obtained solution is obtained. You may try to improve.

Further, for example, the learning device 100 may further update the second processing model 122 and the third processing model 123. As a result, the learning device 100 can obtain a second processing model 122 that is useful from the viewpoint of handling the second modal information in solving the problem. In addition, the learning device 100 can obtain a third processing model 123 that is useful from the viewpoint of integrating the information of the first modal and the information of the second modal in solving the problem. According to this, the learning device 100 is a useful model in which the extraction model 111, the conversion model 112, the first processing model 121, the second processing model 122, and the third processing model 123 are combined. 101 can be obtained. Then, the learning device 100 can use the model 101 when solving a problem by using the information of the first modal and the information of the second modal, and can improve the accuracy of the obtained solution.

Further, for example, the learning device 100 may separate the conversion model 112 from the model 101. According to this, the learning device 100 can obtain a useful conversion model 112. Then, the learning device 100 may use the separated conversion model 112 when solving the problem using the information of the first modal to improve the accuracy of the obtained solution.

Further, for example, the learning device 100 may separate the extraction model 111, the conversion model 112, the first processing model 121, and the second processing model 122 from the model 101. According to this, the learning device 100 can obtain a useful combination model of the extraction model 111, the conversion model 112, the first processing model 121, and the second processing model 122. Then, the learning device 100 uses the separated extraction model 111, the conversion model 112, the first processing model 121, and the second processing model 122 as the first modal information and the second modal information. It may be used when solving a problem by using and to improve the accuracy of the obtained solution.

As described above, the learning device 100 can obtain a useful model. The useful model is, for example, one of the updated extraction model 111, the conversion model 112, the first processing model 121, the second processing model 122, and the third processing model 123. Is. Further, the useful model is, for example, two of the updated extraction model 111, the conversion model 112, the first processing model 121, the second processing model 122, and the third processing model 123. The above combination may be used.

(Example of information processing system 200)
Next, an example of the information processing system 200 to which the learning device 100 shown in FIG. 1 is applied will be described with reference to FIG.

FIG. 2 is an explanatory diagram showing an example of the information processing system 200. In FIG. 2, the information processing system 200 includes a learning device 100, a client device 201, and a terminal device 202.

In the information processing system 200, the learning device 100 and the client device 201 are connected via a wired or wireless network 210. The network 210 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or the like. Further, in the information processing system 200, the learning device 100 and the terminal device 202 are connected via a wired or wireless network 210.

The learning device 100 stores an integrated model that accepts input of the first modal information and the second modal information. The stored integrated model corresponds to, for example, the model 101 shown in FIG. The learning device 100 updates the integrated model based on the teacher data.

The teacher data is, for example, correspondence information in which the sample first modal information, the sample second modal information, and the correct answer data are associated with each other. The teacher data is input to the learning device 100 by the user of the learning device 100, for example. The correct answer data indicates, for example, the correct answer for the output value of the integrated model. The correct answer data may indicate the correct answer for the solution obtained by solving the problem based on the output value of the integrated model. If the first modal is about an image, the information in the first modal is an image. If the second modal is about language, the information in the second modal is a document. The update of the integrated model is realized by, for example, the error back propagation method. The update of the integrated model may be realized by, for example, a learning method other than error back propagation.

Further, the learning device 100 acquires the information of the first modal and the information of the second modal when solving the problem. The learning device 100 acquires, for example, first modal information input to the learning device 100 by the user of the learning device 100. Further, the learning device 100 may acquire the first modal information by receiving the information from the client device 201 or the terminal device 202. The learning device 100 acquires, for example, second modal information input to the learning device 100 by the user of the learning device 100. Further, the learning device 100 may acquire the second modal information by receiving the information from the client device 201 or the terminal device 202.

Then, the learning device 100 solves the problem based on the acquired first modal information and the second modal information by using the updated integrated model, and transfers the obtained solution to the client device 201. Send. The learning device 100 may be used for solving the problem after further fine-tuning the updated integrated model. The learning device 100 is, for example, a server, a PC (Personal Computer), or the like.

The client device 201 is a computer capable of communicating with the learning device 100. The client device 201 may, for example, transmit the first modal information to the learning device 100. Further, the client device 201 may transmit, for example, second modal information to the learning device 100. The client device 201 receives and outputs the solution obtained by the learning device 100 solving the problem. The output format is, for example, display on a display, print output to a printer, transmission to another computer, or storage in a storage area. The client device 201 is, for example, a PC, a tablet terminal, a smartphone, or the like.

The terminal device 202 is a computer capable of communicating with the learning device 100. The terminal device 202 may, for example, transmit the first modal information to the learning device 100. The terminal device 202 may transmit, for example, second modal information to the learning device 100. The terminal device 202 is, for example, a PC, a tablet terminal, a smartphone, an electronic device, an IoT (Internet of Things) device, a sensor device, or the like. Specifically, the terminal device 202 may be a surveillance camera.

Here, the case where the learning device 100 updates the integrated model and solves the problem using the integrated model has been described, but the present invention is not limited to this. For example, another computer may update the integrated model, and the learning device 100 may solve the problem using the integrated model received from the other computer. Further, for example, the learning device 100 may update the integrated model, provide it to another computer, and solve the problem on the other computer by using the integrated model.

Here, the case where the learning device 100 is a device different from the client device 201 and the terminal device 202 has been described, but the present invention is not limited to this. For example, the learning device 100 may be integrated with the client device 201. Further, for example, the learning device 100 may be integrated with the terminal device 202.

Here, the case where the learning device 100 realizes the integrated model in terms of software has been described, but the present invention is not limited to this. For example, the learning device 100 may realize the integrated model electronically.

(Application example of information processing system 200)
For example, the terminal device 202 is a surveillance camera, and transmits an image of the target to the learning device 100. The object is specifically the appearance of the fitting room. In addition, the learning device 100 stores a document that serves as an explanatory text about the target. The description specifically states that the dressing room curtains tend to be closed while humans are using the dressing room, and that shoes are placed in front of the dressing room while humans are using the dressing room. It is a document that describes things that tend to be done. Then, the learning device 100 solves the problem of determining the degree of risk based on the image and the document by using the model. The degree of risk is, for example, an index value indicating the high possibility that a person who has not completed evacuation remains in the fitting room. The degree of risk may be, for example, two values as to whether or not there are any humans who have not completed evacuation in the fitting room.

(Example of hardware configuration of learning device 100)
Next, a hardware configuration example of the learning device 100 will be described with reference to FIG.

FIG. 3 is a block diagram showing a hardware configuration example of the learning device 100. In FIG. 3, the learning device 100 includes a CPU (Central Processing Unit) 301, a memory 302, a network I / F (Interface) 303, a recording medium I / F 304, and a recording medium 305. Further, each component is connected by a bus 300.

Here, the CPU 301 controls the entire learning device 100. The memory 302 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash ROM, and the like. Specifically, for example, a flash ROM or ROM stores various programs, and RAM is used as a work area of CPU 301. The program stored in the memory 302 is loaded into the CPU 301 to cause the CPU 301 to execute the coded process.

The network I / F 303 is connected to the network 210 through a communication line, and is connected to another computer via the network 210. Then, the network I / F 303 controls the internal interface with the network 210 and controls the input / output of data from another computer. The network I / F 303 is, for example, a modem or a LAN adapter.

The recording medium I / F 304 controls data read / write to the recording medium 305 according to the control of the CPU 301. The recording medium I / F 304 is, for example, a disk drive, an SSD (Solid State Drive), a USB (Universal Serial Bus) port, or the like. The recording medium 305 is a non-volatile memory that stores data written under the control of the recording medium I / F 304. The recording medium 305 is, for example, a disk, a semiconductor memory, a USB memory, or the like. The recording medium 305 may be detachable from the learning device 100.

The learning device 100 may include, for example, a keyboard, a mouse, a display, a printer, a scanner, a microphone, a speaker, and the like, in addition to the above-described components. Further, the learning device 100 may have a plurality of recording media I / F 304 and recording media 305. Further, the learning device 100 does not have to have the recording medium I / F 304 or the recording medium 305.

(Hardware configuration example of client device 201)
The hardware configuration example of the client device 201 is specifically the same as the hardware configuration example of the learning device 100 shown in FIG. 3, so the description thereof will be omitted.

(Hardware configuration example of terminal device 202)
Since the hardware configuration example of the terminal device 202 is specifically the same as the hardware configuration example of the learning device 100 shown in FIG. 3, the description thereof will be omitted.

(Example of functional configuration of learning device 100)
Next, an example of a functional configuration of the learning device 100 will be described with reference to FIG.

FIG. 4 is a block diagram showing a functional configuration example of the learning device 100. The learning device 100 includes a storage unit 400, an acquisition unit 401, a first extraction unit 402, a conversion unit 403, a first processing unit 404, a second extraction unit 405, and a second processing unit 406. A third processing unit 407, an updating unit 408, a utilization unit 409, and an output unit 410 are included.

The storage unit 400 is realized by, for example, a storage area such as the memory 302 or the recording medium 305 shown in FIG. Hereinafter, the case where the storage unit 400 is included in the learning device 100 will be described, but the present invention is not limited to this. For example, the storage unit 400 may be included in a device different from the learning device 100, and the stored contents of the storage unit 400 may be referred to by the learning device 100.

The acquisition unit 401 to the output unit 410 function as an example of the control unit. Specifically, the acquisition unit 401 to the output unit 410 may cause the CPU 301 to execute a program stored in a storage area such as the memory 302 or the recording medium 305 shown in FIG. 3, or the network I / F 303. To realize the function. The processing result of each functional unit is stored in a storage area such as the memory 302 or the recording medium 305 shown in FIG. 3, for example.

The storage unit 400 stores various information referred to or updated in the processing of each functional unit. The storage unit 400 stores an integrated model that accepts input of the first modal information and the second modal information. The integrated model to be stored has, for example, a first extraction model, a transformation model, a first processing model, a second extraction model, a second processing model, and a third processing model. The first extraction model, the transformation model, and the first processing model relate to the first modal. The second extraction model and the second processing model relate to a second modal. The third processing model relates to a first modal and a second modal.

The second modal is different from the first modal. For example, the first modal is an image modal and the second modal is a language modal. For example, the first modal is an image modal and the second modal is an audio modal. For example, the first modal is a modal for a first language and the second modal is a modal for a second language.

The acquisition unit 401 acquires various information used for processing of each functional unit. The acquisition unit 401 stores various acquired information in the storage unit 400 or outputs it to each function unit. Further, the acquisition unit 401 may output various information stored in the storage unit 400 to each function unit. The acquisition unit 401 acquires various information based on, for example, a user's operation input. The acquisition unit 401 may receive various information from a device different from the learning device 100, for example.

The acquisition unit 401 acquires the information of the first modal and the information of the second modal. The acquisition unit 401 acquires the first modal information and the second modal information by, for example, accepting the input of the first modal information and the second modal information by the user. .. The acquisition unit 401 may acquire, for example, the information of the first modal and the information of the second modal by receiving from the client device 201 or the terminal device 202. Even if the acquisition unit 401 acquires the information of the first modal and the information of the second modal by acquiring the teacher data including the information of the first modal and the information of the second modal. Good.

The acquisition unit 401 may accept a start trigger to start processing of any of the functional units. The start trigger is, for example, that there is a predetermined operation input by the user. The start trigger may be, for example, the receipt of predetermined information from another computer. The start trigger may be, for example, that any functional unit outputs predetermined information. The acquisition unit 401 receives, for example, the acquisition of the first modal information and the second modal information as a start trigger for starting the processing of each functional unit.

The first extraction unit 402 extracts the feature amount from the first modal information. The first extraction unit 402 extracts, for example, an image feature amount from an image. The extracted image feature amount is, for example, an image feature amount indicating an object appearing in the image. As a result, the first extraction unit 402 can change the information of the first modal into a format that can be input to the conversion model. In addition, the first extraction unit 402 can extract useful information for solving the problem from the first modal information.

The conversion unit 403 acquires a new feature amount by converting the extracted feature amount based on a plurality of parameters using the conversion model. Here, the plurality of parameters include, for example, a first feature amount and a second feature amount. For example, the conversion unit 403 calculates the degree of coincidence between the extracted feature amount and the first feature amount, and based on the index value obtained by weighting the second feature amount based on the calculated degree of coincidence. , A new feature amount is acquired by converting the extracted feature amount.

Specific examples of acquiring a new feature amount when the plurality of parameters are the first feature amount and the second feature amount will be described later with reference to FIGS. 5 to 8. As a result, the conversion unit 403 prepares a plurality of parameters capable of reflecting the features of the information of the second modal, and the feature amount extracted from the information of the first modal via the plurality of parameters is subjected to the second. It is possible to reflect the characteristics of modal information.

Further, the plurality of parameters may be, for example, parameters of a neural network in which the number of nodes in the input layer, the number of nodes in the output layer, and the number of nodes in the intermediate layer are larger. Neural networks correspond to transformation models. The conversion unit 403 acquires a new feature amount by inputting the extracted feature amount into the neural network, for example.

A specific example of acquiring a new feature amount when a plurality of parameters are neural network parameters will be described later with reference to FIG. As a result, the conversion unit 403 prepares a plurality of parameters capable of reflecting the features of the information of the second modal, and the feature amount extracted from the information of the first modal via the plurality of parameters is subjected to the second. It is possible to reflect the characteristics of modal information.

The first processing unit 404 acquires the first output value by inputting the acquired new feature amount into the first processing model. The first processing model is, for example, an image processing model. The first processing unit 404 acquires the first output value, for example, by inputting a new image feature amount into the image processing model.

An example of acquiring the first output value will be specifically described later with reference to FIGS. 5 to 8. As a result, the first processing unit 404 can change the acquired new feature quantity into a format that can be input to the third processing model. In addition, the first processing unit 404 can extract useful information for solving the problem from the acquired new features.

The second extraction unit 405 extracts other feature quantities from the second modal information. The second extraction unit 405 extracts, for example, a language feature from a document. The extracted linguistic feature is, for example, a linguistic feature indicating a word contained in a document. As a result, the second extraction unit 405 can change the information of the second modal into a format that can be input to the second processing model. In addition, the second extraction unit 405 can extract useful information for solving the problem from the second modal information.

The second processing unit 406 acquires the second output value by inputting the extracted other feature quantities into the second processing model. The second processing model is, for example, a language processing model. The second processing unit 406 acquires the second output value by inputting the language feature amount into the language processing model, for example.

An example of acquiring the second output value will be specifically described later with reference to FIGS. 5 to 8. As a result, the second processing unit 406 can change the extracted other features into a format that can be input to the third processing model. In addition, the second processing unit 406 can extract useful information for solving the problem from the extracted other features.

The third processing unit 407 acquires the third output value by inputting the acquired first output value and the second output value into the third processing model. The third processing model is, for example, a cross-modal processing model. The third processing unit 407 acquires the third output value by inputting the first output value and the second output value into the cross-modal processing model, for example. An example of acquiring the third output value will be specifically described later with reference to FIGS. 5 to 8. As a result, the third processing unit 407 can obtain a third output value that integrates the features of the language and the image.

The update unit 408 updates a plurality of parameters based on the acquired third output value. The update unit 408 updates a plurality of parameters by the error back propagation method, for example, based on the third output value. The update unit 408 calculates the loss value based on the third output value by, for example, the loss function, and updates a plurality of parameters based on the loss value.

As a result, the update unit 408 can obtain a plurality of parameters useful from the viewpoint of handling the first modal information and the second modal information in solving the problem. For example, the update unit 408 optimizes a plurality of explicitly prepared parameters so that the features of the second modal information can be effectively utilized when acquiring a new feature amount from the first modal information. Can be planned.

The update unit 408 updates the first processing model based on the third output value. The update unit 408 updates the first processing model by the error back propagation method, for example, based on the third output value. The update unit 408 calculates the loss value based on the third output value by, for example, the loss function, and updates the first processing model based on the loss value. As a result, the update unit 408 can obtain a first processing model that is useful from the viewpoint of handling the first modal information in solving the problem.

The update unit 408 updates the second processing model and the third processing model based on the third output value. The update unit 408 updates the second processing model and the third processing model by the error back propagation method, for example, based on the third output value. For example, the update unit 408 calculates the loss value based on the third output value by the loss function, and updates the second processing model and the third processing model based on the loss value.

As a result, the update unit 408 can obtain a second processing model that is useful from the viewpoint of handling the second modal information in solving the problem. In addition, the learning device 100 can obtain a third processing model that is useful from the viewpoint of integrating the information of the first modal and the information of the second modal in solving the problem.

The user unit 409 solves a predetermined problem. The utilization unit 409 solves a predetermined problem in response to input of other information of the first modal by using, for example, a plurality of updated parameters and an unupdated first processing model. As a result, the utilization unit 409 can use the updated plurality of parameters when solving the problem based on at least other information of the first modal, and can improve the accuracy of the obtained solution.

The utilization unit 409 solves a predetermined problem in response to input of other information of the first modal by using, for example, a plurality of parameters after the update and the first processing model after the update. As a result, the utilization unit 409 uses the updated parameters and the updated first processing model when solving the problem based on at least other information of the first modal, and obtains a solution. It is possible to improve the accuracy of.

The utilization unit 409 uses, for example, a plurality of parameters after the update, the first processing model after the update, the second processing model after the update, and the third processing model after the update. Based on the other information of the modal and the other information of the second modal, the predetermined problem is solved. As a result, the utilization unit 409 solves the problem of the plurality of parameters after the update, the first processing model after the update, the second processing model after the update, and the third processing model after the update. It can be used in some cases to improve the accuracy of the obtained solution.

The output unit 410 outputs the processing result of any of the functional units. The output format is, for example, display on a display, print output to a printer, transmission to an external device by the network I / F 303, or storage in a storage area such as a memory 302 or a recording medium 305. As a result, the output unit 410 can notify the user of the processing result of each functional unit, and can improve the convenience of the learning device 100.

The output unit 410 outputs, for example, a plurality of updated parameters. As a result, the output unit 410 can refer to a plurality of updated parameters that are useful from the viewpoint of handling the first modal information and the second modal information in solving the problem. The output unit 410 can make the updated plurality of parameters available to other computers, for example. Therefore, the output unit 410 can improve the accuracy of the solution obtained by solving the problem by using at least a plurality of parameters.

The output unit 410 may output, for example, the first processing model. As a result, the output unit 410 can refer to the updated first processing model, which is useful from the viewpoint of handling the first modal information in solving the problem. The output unit 410 can make the updated first processing model available to other computers, for example. Therefore, the output unit 410 can improve the accuracy of the solution obtained by solving the problem based on the information of the first modal by using at least the first processing model.

The output unit 410 may output, for example, a second processing model and a third processing model. As a result, the output unit 410 can refer to the updated second processing model. The output unit 410 can make the updated second processing model available to other computers, for example. Therefore, the output unit 410 solves the problem based on the information of the first modal and the information of the second modal by using at least the second processing model and the third processing model. It is possible to improve the accuracy of the obtained solution.

(Specific functional configuration example of the learning device 100)
Next, a specific functional configuration example of the learning device 100 will be described with reference to FIG. 5 in the case where the first modal is related to an image and the second modal is related to a language.

FIG. 5 is a block diagram showing a specific functional configuration example of the learning device 100. In FIG. 5, the learning device 100 has an integrated model 500. The integrated model 500 includes an image feature amount generation unit 501, a query generation unit 502, a table search unit 503, an image processing unit 504, a language processing unit 505, a crossmodal processing unit 506, and a loss function calculation unit 507. including.

Further, the learning device 100 has a data table 510. The data table 510 is a table that stores the searched key sequence and the cross-modal feature quantity column. The data table 510 is a table in which linguistic information is reflected by pre-learning.

The data table 510 is, for example, a table in which a cross-modal feature amount reflecting linguistic information is associated with a searched key that quantizes an image feature amount on a one-to-one basis. Based on the search query generated from the image feature amount, the data table 510 weights the linguistic information reflected as the cross-modal feature amount in the data table 510 according to the degree of relevance to the image feature amount. It is a table for extracting.

The image feature amount generation unit 501 generates an image feature amount sequence from the input image and outputs it. The image feature amount generation unit 501 detects, for example, an object appearing in the input image, and generates and outputs an image feature amount sequence including an image feature amount indicating each of the detected objects. The image feature sequence is represented by, for example, a vector. The image feature amount indicates visual feature information of an object including, for example, a color and a shape.

The query generation unit 502 generates and outputs a search query sequence based on the input image feature sequence. The query generation unit 502 generates a search query sequence by, for example, multiplying the image feature sequence by the transformation matrix W _q. The search query column is represented by, for example, a vector.

The table search unit 503 calculates and outputs an index value string by weighting the cross-modal feature amount from the data table 510 based on the input search query column. The table search unit 503 calculates a new weighted average of the cross-modal features of the data table 510 based on the inner product of the input search query column and the searched key column of the data table 510, for example. Generates and outputs a cross-modal feature sequence.

The image processing unit 504 converts and outputs the image feature quantity sequence based on the input new cross-modal feature quantity sequence. For example, the image processing unit 504 adds a new cross-modal feature sequence to the image feature sequence, converts the added image feature sequence using an image processing model, and converts the converted image feature. Output a quantity sequence.

The language processing unit 505 generates and outputs a word embedding string based on the input word string. The language processing unit 505 generates and outputs a word embedding string based on the input word string, for example, using a language processing model.

The cross-modal processing unit 506 integrates the input word embedding sequence and the input image feature amount sequence, generates a new word embedding sequence, and outputs a new image feature amount sequence. The cross-modal processing unit 506 integrates the input word embedding sequence and the input image feature amount sequence by using, for example, a cross-modal processing model, and forms a new word embedding column and a new image feature amount sequence. And output.

The loss function calculation unit 507 calculates the loss value using the loss function based on the input word embedding sequence and the input image feature amount sequence. Then, the learning device 100 updates the data table 510, the image processing model, the language processing model, and the cross-modal processing model by pre-learning the integrated model 500 based on the loss value.

As a result, the learning device 100 can reflect the language information in the data table 510. The learning device 100 can reflect, for example, linguistic information in the cross-modal features of the data table 510. Specifically, the learning device 100 can reflect the loss value based on the word string in the cross-modal feature amount of the data table 510.

In this way, the learning device 100 explicitly prepares the data table 510 that reflects the linguistic information, and when the linguistic information is reflected, the cross-modal feature amount is effectively based on the quantization by the searched key. Can be updated. As a result, the learning device 100 can prepare an expression space in consideration of linguistic information when handling an image, and the image processing unit 504 can generate a useful image feature quantity sequence.

(An example in which the learning device 100 learns the integrated model 600)
Next, an example in which the learning device 100 learns the integrated model 600 will be described with reference to FIG.

FIG. 6 is an explanatory diagram showing an example of learning the integrated model 600. In FIG. 6, the learning device 100 has an integrated model 600 that embodies the integrated model 500. The integrated model 600 includes an add unit 601, an image Transferr 602, a language Transferr 603, and a cross-modal Transferr 604.

Further, the learning device 100 has a language information data table 610 that embodies the data table 510. The language information data table 610 includes an array Queue 611 for setting a search query sequence, an array Key 621 for setting a search key sequence, and an array Value 622 for setting a cross-modal feature quantity sequence.

The learning device 100 performs pre-learning of the integrated model 600 by masking a part of the input and predicting the masked part. Learning apparatus 100 is, for example, as an input, the image feature amount sequence _{F = f 1, f 2,} ···, f i, ···, and f _I, language embedding column _{E = e 1, e 2,} ··・, E _j , ..., e _L and so on. Then, the learning apparatus 100, the image feature amount sequence _{F = f 1, f 2,} ···, f i, ···, of f _I, image features f _i and Mask, language embedding column E = Of e ₁ , e ₂ , ..., e _j , ..., e _L , the language embeddinge _j is masked.

Next, the learning device 100 converts the masked image feature amount sequence F by the Add unit 601 and outputs the converted image feature amount sequence F'. The learning device 100 further converts the corrected image feature amount sequence F'by the image Transferr 602, and outputs the converted image feature amount sequence F ”. Further, the learning device 100 uses the language Transfermer 603 to create a masked language. The embedded column E is converted and the converted language embedding column E'is output. Then, the learning device 100 uses the crossmodal Transferformer 604 to convert the converted image feature amount column F'and the converted language embedding column E'. To generate a new image feature sequence F ^ and a new language embedding sequence E ^.

Learning apparatus 100, the image feature amount corresponding to Mask portion of the image feature quantity column F ^ and "f _i ^", the language embedding sequence E ^ Mask language corresponding to the portion embedding of the "e _j ^" Based on, the Masked part is predicted. Then, the learning device 100 performs pre-learning of the integrated model 600 based on the predicted result. Next, moving to the description of FIG. 7, an example in which the learning device 100 converts the image feature quantity sequence F by the Add unit 601 will be described.

FIG. 7 is an explanatory diagram showing an example of converting the image feature quantity sequence F. In FIG. 7, the learning apparatus 100 sets the initial value of the searched key sequence obtained by multiplying the N-dimensional identity matrix I by the transformation matrix W _k in advance in the array Key621, and converts it into the N-dimensional identity matrix I. The _{crossmodal feature matrix obtained by multiplying W v} is set in the array Value622.

Here, the learning device 100 multiplies the masked image feature sequence F by the transformation matrix W _q , calculates the search query sequences Q = q ₁ , ..., Q _I , and sets them in the array Queue 611. The learning device 100 calculates the inner product of the sequence Queen 611 and the sequence Key 621, and calculates the softmax of the calculated inner product. The learning device 100 calculates the inner product of the calculated softmax and the array Value622 as correction information. The learning device 100 converts the image feature quantity sequence F by adding the correction information to the image feature quantity sequence F. Next, the description shifts to FIG. 8, and a specific example of the calculation for converting the image feature quantity sequence F will be described.

FIG. 8 is an explanatory diagram showing a specific example of the calculation for converting the image feature quantity sequence F. As shown in FIG. 8, (8-1) Learning unit 100, when converting the image feature amount column F, the respective image feature amount _{f 1, f 2, ···,} f i, ···, f to _I, each of the transform vector W _q1 of the transformation matrix _{W q, ···, W qh,} ···, multiplying W _qH. Learning apparatus 100 is, for example, the image feature amount f _i, converting the vector _{W q1, ···, W qh,} ···, W qH multiplies, H-number of the partial feature query q _i, 1, · · · , Q _{i, h} , ···, q _{i, H} are acquired.

Here, the learning apparatus 100, based on the image feature amount f _i, the partial feature query _{q i, 1, ···, q} i, h, ···, q i, has been described to obtain the _H It is carried out similar calculations for other image characteristic amount other than the image feature amount f _i. In the following explanation, the calculation to be performed thereafter will be described by taking the _{partial feature queries q i and h} as an example, but the same calculation is performed for other partial feature queries other than the _{partial feature queries q i and h.} To.

(8-2) The learning device 100 has a partial feature query q _{i, h and N partial feature keys k 1, h} , ..., k _{n, h} , ..., K set in the array Key 621. _{The degree of coincidence with N and H} is calculated, and the softmax of the degree of coincidence is calculated. The learning device 100 calculates, for example, the inner product of the partial feature queries q _{i, h} and N partial feature keys k _{1, h} , ···, k _{n, h} , ···, k _{N, H.} , The softmax of the inner product is calculated. Specifically, the learning device 100 calculates the softmax of the inner product by the following formula (1).

exp (q _{i, h} · k _{1, h} ) / Σ _x exp (q _{i, h} · k _{x, h} ), ···, exp (q _{i, h} · k _{n, h} ) / Σ _x exp (q) _{i, h} · k _{x, h} ), ···, exp (q _{i, h} · k _{N, h} ) / Σ _x exp (q _{i, h} · k _{x, h} ) ··· (1)

(8-3) The learning device 100 is associated with N partial feature keys k _{1, h} , ..., k _{n, h} , ..., K _{N, H} based on softmax, N. Calculate the weighted average df _i of the cross-modal features v _{1, h} , ···, v _{n, h} , ···, v _{N, H.} Thus, the learning device 100, I-number of image feature amounts _{f 1, ···, f i,} ···, corresponding to f _I, a weighted average df ₁ of number I, · · ·, df _i, ..., Df _I is calculated as a new cross-modal feature quantity. Then, the learning apparatus 100, I-number of image feature amounts _{f 1, ···, f i,} ···, the f _I, a weighted average df ₁ of number I, · · ·, df _i, · · ·, the df _I respectively added, I pieces of image feature amounts f _1, converts ···, f _i, ···, an f _I.

As a result, the learning device 100 can reflect the language information in the language information data table 610. For example, the learning device 100 can reflect the language information in the transformation matrix W _q , the transformation matrix W _k, and the transformation matrix W _v. As a result, the learning device 100 can prepare an expression space in consideration of linguistic information when handling an image, can generate a useful image feature sequence, and learns a useful integrated model 600. be able to. Further, the learning device 100 can efficiently perform fine tuning of the integrated model 600 after learning. After that, the learning device 100 can improve the accuracy of the solution obtained by solving the problem by using the integrated model 600 in solving the problem.

Further, the learning device 100 can, for example, make it possible to appropriately complement the image feature amount with the crossmodal feature amount obtained from the linguistic information, and to generate a useful image feature amount sequence. Therefore, the learning device 100 can separate the image Transferr 602 and efficiently perform fine tuning of the image Transferr 602 alone. After that, the learning device 100 can improve the accuracy of the solution obtained by solving the problem even if the image Transformer 602 is used alone in solving the problem.

Here, conventionally, an object captured in an image has a property of having a different meaning depending on the relationship with a surrounding object, but an image feature quantity indicating an object captured in an image is quantized unlike a language feature quantity. It is expressed as a continuous value, and it tends to be difficult to give an appropriate meaning. Specifically, it is desirable to give different meanings to the image features showing various chairs in consideration of the arrangement relationship in the room, but it is difficult to give an appropriate meaning. Further, specifically, it is desirable to give different meanings to the image features showing various red lights in consideration of the positional relationship with the stopping vehicle, but it is difficult to give an appropriate meaning. .. On the other hand, as described above, the learning device 100 makes it easy to give an appropriate meaning to the image feature amount by the cross-modal feature amount obtained from the linguistic information, and makes it easy to obtain a useful integrated model 600. Can be done.

(Another example in which the learning device 100 converts an image feature sequence)
Next, another example in which the learning device 100 converts the image feature quantity sequence will be described with reference to FIG. For example, in the examples of FIGS. 5 to 8, the case where the learning device 100 uses the data table 510 including the searched key sequence and the cross-modal feature quantity sequence has been described. On the other hand, in the example of FIG. 9, the case where the learning device 100 uses the neural network 900 instead of the data table 510 will be described.

FIG. 9 is an explanatory diagram showing another example of converting the image feature quantity sequence. In FIG. 9, the neural network 900 is a two-layer fully connected network, which has an input layer 901, an intermediate layer 902, and an output layer 903. The dimension of the input layer 901 and the dimension of the output layer 903 are formed to be the same. The dimension of the intermediate layer 902 is formed to be larger than the dimension of the input layer 901. The parameter of the connection between the input layer 901 and the intermediate layer 902 is the transformation matrix W _q → _k . The parameter of the connection between the intermediate layer 902 and the output layer 903 is the transformation matrix W _k → _v .

The learning device 100 generates N correspondences from the i-th image feature amount f _i by the transformation matrix W _q → _k , and based on the generated correspondences, the crossmodal feature amount is calculated by the _{transformation matrix W k} → _v. Will be generated. The learning device 100 can realize the same function as the data table 510 by the neural network 900 by updating the _{transformation matrix W q} → _k and the transformation matrix W _k → _{v by pre-learning.}

As a result, the learning device 100 can reflect the linguistic information on the neural network 900. For example, the learning device 100 can reflect the language information in the transformation matrix W _q → _k and the transformation matrix W _k → _v. As a result, the learning device 100 can prepare an expression space in consideration of linguistic information when handling an image, can generate a useful image feature sequence, and can learn a useful integrated model. can do. In addition, the learning device 100 can efficiently perform fine tuning of the integrated model after learning. After that, the learning device 100 can improve the accuracy of the solution obtained by solving the problem by using the integrated model in solving the problem.

(An example in which the learning device 100 uses the integrated model 1000)
Next, an example in which the learning device 100 uses the integrated model 1000 will be described with reference to FIG.

FIG. 10 is an explanatory diagram showing an example in which the learning device 100 uses the integrated model 1000. The learning device 100 uses, for example, the integrated model 1000 to solve the problem of video monitoring in view of the surrounding situation. The integrated model 1000 includes an image Transferr 1010, a language Transferr 1020, and a crossmodal Transferr 1030. The integrated model 1000 includes a linguistic information data table 610, a neural network 900, and the like.

Specifically, the learning device 100 solves the problem of detecting whether or not there is a person who has not yet evacuated in the room when a disaster occurs. At this time, the learning device 100 acquires the image information 1001 of the surveillance camera that captures the appearance of the fitting room. The learning device 100 acquires an image feature amount sequence from the image information 1001, converts it using the language information data table 610 or the neural network 900, and then inputs it to the image Transferr 1010.

Further, the learning device 100 acquires linguistic information 1002 in which "the fitting room is used by taking off shoes" and "the fitting room is used by closing the curtain". The learning device 100 acquires a language feature sequence from the language information 1002 and inputs it to the language Transferformer 1020. Then, the learning device 100 inputs the output value of the image Transferr 1010 and the output value of the language Transferr 1020 into the cross-modal Transferr 1030.

The learning device 100 determines whether or not there is a person who has not yet evacuated to the fitting room based on the output value of the cross-modal Transformer 1030. As a result, the learning device 100 can solve the problem by appropriately matching the situation presented by the linguistic information with the situation presented by the image information, and improves the accuracy of the solution obtained by solving the problem. Can be done. In the learning device 100, for example, since the curtain of the fitting room is closed and a person is not directly photographed, the learning device 100 is presented with linguistic information even when it is determined that there is no person by the image information alone. It is possible to correctly determine that there is a person in the fitting room by appropriately matching the situation that "a person is using the fitting room" with the arrangement relationship of the objects in the image.

(Learning process procedure)
Next, an example of the learning processing procedure executed by the learning device 100 will be described with reference to FIG. The learning process is realized, for example, by the CPU 301 shown in FIG. 3, a storage area such as a memory 302 or a recording medium 305, and a network I / F 303.

FIG. 11 is a flowchart showing an example of the learning processing procedure. In FIG. 11, the learning device 100 initializes the searched key value and the feature value correction value value in the table with random values (step S1101).

Next, the learning device 100 generates a set of language feature values and a set of image feature values from each of the interrelated documents and images (step S1102). Then, the learning device 100 masks the image feature values by replacing the image feature values with irrelevant values at a constant ratio in the set of image feature values (step S1103).

Next, the learning device 100 calculates the degree of relationship between each image feature value in the set of image feature values and the searched key value in the table (step S1104). Then, the learning device 100 corrects each image feature value based on the feature value correction value value in the table corresponding to the degree of relationship (step S1105).

Next, the learning device 100 uses a language processing model, an image processing model, and a cross-modal processing model to create a masked image based on a set of language feature values and a set of corrected image feature values. The feature value is restored and the predicted value is acquired (step S1106). Then, the learning device 100 updates the parameter value including the searched key value and the feature value correction value value in the table in the direction of reducing the loss value of the predicted value (step S1107).

Next, the learning device 100 determines whether or not the end condition is satisfied (step S1108). The end condition is, for example, that the loop of steps S1102 to S1108 is repeated a certain number of times or more. The end condition is, for example, that the difference between the previous loss value and the current loss value is less than a certain value.

Here, if the end condition is not satisfied (step S1108: No), the learning device 100 returns to the process of step S1102. On the other hand, when the end condition is satisfied (step S1108: Yes), the learning device 100 ends the learning process. As a result, the information processing apparatus can obtain a useful model including parameter values that reflect the characteristics of the linguistic information.

As described above, according to the information processing device, the feature amount can be extracted from the first modal information. According to the information processing apparatus, a new feature amount can be acquired by converting the extracted feature amount based on the parameter. According to the information processing apparatus, the first output value can be acquired by inputting the acquired new feature quantity into the first processing model related to the first modal. According to the information processing apparatus, the second output is performed by inputting other features extracted from the information of the second modal different from the first modal into the second processing model related to the second modal. You can get the value. According to the information processing apparatus, the acquired first output value and the second output value are input to the third processing model relating to the first modal and the second modal, so that the third output value is obtained. Can be obtained. According to the information processing device, the parameters can be updated based on the acquired third output value. This allows the information processing device to obtain a useful model that includes parameters that reflect the characteristics of the second modal information.

According to the information processing device, the first feature amount and the second feature amount can be adopted as parameters. According to the information processing device, the degree of coincidence between the extracted feature amount and the first feature amount is calculated, and the second feature amount is weighted based on the calculated degree of coincidence based on the obtained index value. , A new feature amount can be obtained by converting the extracted feature amount. As a result, the information processing apparatus reflects the features of the information of the second modal to the first feature amount and the second feature amount, and realizes the conversion of the feature amount extracted from the information of the first modal. can do.

According to the information processing device, it is possible to adopt a neural network parameter having a large number of nodes in the input layer, a number of nodes in the output layer, and a larger number of nodes in the intermediate layer as parameters. According to the information processing device, a new feature amount can be acquired by inputting the extracted feature amount into the neural network. As a result, the information processing apparatus can reflect the characteristics of the information of the second modal in the neural network and realize the conversion of the feature amount extracted from the information of the first modal.

According to the information processing apparatus, the first processing model is updated based on the third output value, and the updated parameters and the updated first processing model are used in addition to the first modal. A predetermined problem can be solved according to the input of the information of. As a result, the information processing apparatus can obtain a useful first processing model, and can improve the accuracy of the solution obtained by solving a predetermined problem using the first processing model.

According to the information processing device, the first processing model, the second processing model, and the third processing model can be updated based on the third output value. According to the information processing apparatus, a predetermined problem is solved by using the updated parameters, the updated first processing model, the updated second processing model, and the updated third processing model. Can be solved. As a result, the information processing apparatus can obtain a useful first processing model, a second processing model, and a third processing model, and improves the accuracy of the solution obtained by solving a predetermined problem. be able to.

According to the information processing device, a modal related to an image can be adopted as the first modal. According to the information processing apparatus, a modal related to language can be adopted as the second modal. As a result, the information processing apparatus can obtain a model useful for solving the problem based on the image information and the linguistic information.

According to the information processing device, a modal related to an image can be adopted as the first modal. According to the information processing device, a modal related to voice can be adopted as the second modal. As a result, the information processing apparatus can obtain a model useful for solving the problem based on the image information and the audio information.

According to the information processing device, a modal related to the first language can be adopted as the first modal. According to the information processing apparatus, a modal related to the second language can be adopted as the second modal. This allows the information processing device to obtain a useful model for solving a problem based on linguistic information in different languages.

According to the information processing device, the parameters can be updated by the error back propagation method based on the third output value. As a result, the information processing apparatus can update the parameters with high accuracy.

The learning method described in this embodiment can be realized by executing a program prepared in advance on a computer such as a PC or a workstation. The learning program described in this embodiment is executed by being recorded on a computer-readable recording medium and being read from the recording medium by the computer. The recording medium is a hard disk, a flexible disk, a CD (Compact Disc) -ROM, an MO, a DVD (Digital Versailles Disc), or the like. Further, the learning program described in this embodiment may be distributed via a network such as the Internet.

100 Learning device 101 Model 111 Extraction model 112 Conversion model 121 First processing model 122 Second processing model 123 Third processing model 200 Information processing system 201 Client device 202 Terminal device 210 Network 300 Bus 301 CPU
302 Memory 303 Network I / F
304 Recording medium I / F
305 Recording medium 400 Storage unit 401 Acquisition unit 402 First extraction unit 403 Conversion unit 404 First processing unit 405 Second extraction unit 406 Second processing unit 407 Third processing unit 408 Update unit 409 Utilization unit 410 Output Part 500, 600, 1000 Integrated model 501 Image feature amount generation part 502 Query generation part 503 Table search part 504 Image processing part 505 Language processing part 506 Cross modal processing part 507 Loss function calculation part 510 Data table 601 Add part 602,1010 Transformer
603, 1020 Language Transformer
604 Cross Modal Transformer
610 Language Information Data Table 611 Array Query
621 Array Key
622 Array Value
900 Neural network 901 Input layer 902 Intermediate layer 903 Output layer 1001 Image information 1002 Language information 1030 Crossmodal Transformer

Claims (9)

Features are extracted from the information of the first modal,
By converting the extracted feature quantity based on the parameter, a new feature quantity is obtained.
By inputting the acquired new feature quantity into the first processing model related to the first modal, the first output value is acquired.
A second output value is obtained by inputting another feature amount extracted from the information of the second modal different from the first modal into the second processing model related to the second modal.
By inputting the acquired first output value and the second output value into the third processing model relating to the first modal and the second modal, the third output value is acquired. ,
The parameter is updated based on the acquired third output value.
A learning method characterized by a computer performing processing. The parameters include a first feature amount and a second feature amount.
The process of acquiring the new feature amount is
The degree of coincidence between the extracted feature amount and the first feature amount was calculated, and based on the calculated degree of coincidence, the second feature amount was weighted and extracted based on the index value obtained. The learning method according to claim 1, wherein the new feature amount is acquired by converting the feature amount. The parameters are parameters of a neural network in which the number of nodes in the input layer, the number of nodes in the output layer, and the number of nodes in the intermediate layer are larger.
The process of acquiring the new feature amount is
The learning method according to claim 1, wherein the new feature amount is acquired by inputting the extracted feature amount into the neural network. The first processing model is updated based on the third output value.
Using the updated parameters and the updated first processing model, a predetermined problem is solved in response to input of other information of the first modal.
The learning method according to any one of claims 1 to 3, wherein the processing is executed by the computer. Based on the third output value, the first processing model, the second processing model, and the third processing model are updated.
In addition to the first modal, the updated parameters, the updated first processing model, the updated second processing model, and the updated third processing model are used. In response to the input of the information of the above and the other information of the second modal, the predetermined problem is solved.
The learning method according to any one of claims 1 to 3, wherein the processing is executed by the computer. The first modal and the second modal are the image modal and the language modal, the image modal and the audio modal, the first language modal and the second language. The learning method according to any one of claims 1 to 5, wherein the learning method is one of a set with a modal. The process of updating the parameters is
The learning method according to any one of claims 1 to 6, wherein the parameter is updated by an error back propagation method based on the third output value. Features are extracted from the information of the first modal,
By converting the extracted feature quantity based on the parameter, a new feature quantity is obtained.
By inputting the acquired new feature quantity into the first processing model related to the first modal, the first output value is acquired.
A second output value is obtained by inputting another feature amount extracted from the information of the second modal different from the first modal into the second processing model related to the second modal.
By inputting the acquired first output value and the second output value into the third processing model relating to the first modal and the second modal, the third output value is acquired. ,
The parameter is updated based on the acquired third output value.
A learning program characterized by having a computer perform processing. Features are extracted from the information of the first modal,
By converting the extracted feature quantity based on the parameter, a new feature quantity is obtained.
By inputting the acquired new feature quantity into the first processing model related to the first modal, the first output value is acquired.
A second output value is obtained by inputting another feature amount extracted from the information of the second modal different from the first modal into the second processing model related to the second modal.
By inputting the acquired first output value and the second output value into the third processing model relating to the first modal and the second modal, the third output value is acquired. ,
The parameter is updated based on the acquired third output value.
A learning device having a control unit.

Images