WO2016163556A1 - Method for estimating perceptual semantic content by analysis of brain activity - Google Patents

Abstract

Provided is an estimation method for measuring and analyzing brain activity to estimate a perceptual semantic content thereof. This method comprises: (A) inputting, to a data processing means, an output when a cranial nerve activity detection means detects an annotation of a perceptual content and brain activity induced in a subject by a training stimulation; (B) associating a sematic space representation of the training stimulation and the output of the cranial nerve activity detection means in a stored semantic space and storing the association in a training result information storage means; (C) inputting, to the data processing means, an output when the cranial nerve activity detection means detects brain activity induced by a novel stimulation, and obtaining a probability distribution in the semantic space which represents perceptual semantic contents for the output of the novel stimulation-induced brain activity by the cranial nerve activity detection means on the basis of the association; and (D) estimating a highly probable perceptual semantic content on the basis of the probability distribution. The association process may be performed for each subject. In the probability estimation process, the likelihood calculated on the basis of the coordinate of a given word in the semantic space and the probability distribution is used as an indicator.

Description

Estimation method of perceptual meaning by analysis of brain activity

The present invention measures the brain activity of a subject under natural perception such as when watching a video, and analyzes the measured information to estimate the perceptual meaning content perceived by the subject. The present invention relates to a content estimation method.

A technology (brain information decoding technology) has been developed for estimating the perceived content and predicting the behavior by analyzing the brain activity of the subject. These technologies are expected as elemental technologies for inter-brain machine interfaces, and as means for conducting preliminary evaluations, purchase predictions, etc. of videos and various products.
The semantic perception estimation technology based on the above brain activity at the present time estimates predetermined perceptual semantic content for a limited perceptual object such as a simple line drawing or still image containing a single or a small number of perceptual semantic content. Limited to things.
The procedure for decoding perceptual meaning content from brain activity using the prior art is as follows. First, model training (calibration) is performed to interpret an individual's brain activity. At this stage, a stimulus set composed of images and the like is presented to the subject, and brain activity induced by those stimuli is recorded. By this stimulus-brain activity pair (training data sample), a correspondence between perceptual content and brain activity is obtained. Next, by recording the new brain activity that is the subject of perceptual meaning content estimation and judging which brain activity is similar to the brain activity obtained in the training data sample, perceptual meaning content estimation is performed. Do.
Patent Document 1 discloses the decoding and reproduction of subjective perceptual or cognitive experience. In this disclosure, an initial set of brain activity data generated from a subject in response to an initial sensory stimulus is obtained using a brain imaging device and converted into a corresponding set of predetermined response values. The second set of brain activity data generated in response to the second sensory stimulus is obtained from the subject using the decoding distribution, and is the second set of brain activity data in accordance with the predetermined response value. Probability is determined. A second set of brain activity stimuli is decoded based on the probability of correspondence between the second set of brain activity data and the predicted response value.
Non-Patent Document 1 describes encoding (encoding) and decoding (decoding) by fMRI (Functional Magnetic Resonance Imaging). This document shows that both encoding and decoding operations are used to explore how common issues of information are represented in the brain, but focus on the encoding model. Doing has two important advantages over decoding. The reason for this is that, firstly, only a partial description can be obtained with the decoding model, but in principle the encoding model gives a complete functional description of the area of interest. Second, obtaining an optimal decoding model from an encoding model is straightforward, but obtaining an encoding model from a decoding model is much more difficult. Therefore, Non-Patent Document 1 proposes a systematic model approach from estimation of a voxel encoding model by fMRI scanning, and performs decoding using the obtained encoding model.
In addition, it has already been reported that a brain image can be acquired while a certain scene is viewed, and the brain image can be reconstructed from the brain image. Non-Patent Document 2 further shows that text corresponding to the mental content reflected in the brain image can also be generated. This starts with a brain image collected when a subject is reading the name of a specific item (eg, “apartment”) while looking at a line drawing of the name of an item. A spiritual meaning display model of a specific concept is constructed from text data, and such a display aspect is mapped to an activation pattern in a corresponding brain image. It has been reported that from this mapping, a collection of semantically related words (eg, “door”, “window” for “apartment”) could be generated.

US Patent Application Publication No. 2013-0184558

Thomas Naselaris, Kendrick N. Kay, Shinji Nishimoto, Jack L. Gallant, “Encoding and decoding in fMRI”, NeuroImage 2011, 56 (2): 400-410 Francisco Pereira, Greg Detre, Matthew Botvinick "Generating text from functional brain images", Frontiers in Human Neuroscience 2012, 5:72

The technique targeted by the present invention is capable of estimating any perceptual meaning content of a subject under natural perception such as watching a movie. In this regard, the prior art has a limitation in at least one of the following points, and the above-mentioned purpose cannot be satisfied. (1) The prior art is intended for simple line drawings and still images, and cannot be applied to situations where many things such as natural moving images, impressions, etc. occur dynamically. (2) In the prior art, the perceptual meaning contents that can be estimated are limited to those included in the training data sample, and any other perceptual meaning contents cannot be estimated.

The technique to which the present invention is directed includes estimating the perceptual meaning content perceived by the subject by analyzing the measured information as described above. By associating contents in an intermediate representation space (semantic space), estimation of arbitrary perceived contents is realized. Details are shown below.
The method of estimating perceptual meaning content by analyzing brain activity according to the present invention includes information presenting means for presenting information serving as a stimulus to a subject, cranial nerve activity detecting means for detecting a brain nerve activity signal of the subject by the stimulus, and stimulus Data processing means for inputting a language description relating to the contents of the contents and the output of the cranial nerve activity detecting means, semantic space information storage means readable from the data processing means, and training result information storage means readable and writable from the data processing means And analyzing the brain activity of the subject by using a brain activity analysis device comprising: and estimating the perceptual meaning content perceived by the subject.
(1) Training information is presented to the subject and a training stimulus is given to the subject, the language description of the perceptual content that the training stimulus induces to the subject, and the brain that the training stimulus induces to the subject The output of the cranial nerve activity detecting means that has detected the activity is input to the data processing means.
Here, the training information is an image, a moving image, or the like, and this information becomes a stimulus to the subject, and the stimulus induces some perceptual content to the subject. A language description (annotation) of this perceptual content is acquired and input to the data processing means. Further, the output of the cranial nerve activity detecting means that detects the brain activity as an electroencephalogram or an fMRI signal is also input to the data processing means.
(2) Applying a semantic space stored in the semantic space information storage means, associating the semantic space representation of the training stimulus with the output of the cranial nerve activity detecting means on the semantic space, and as a result of the association Is stored in the training result information storage means.
The above semantic space is constructed using a large-scale database such as a corpus and describes the semantic relationship of words appearing in a language description.
Further, the above relationship is performed on the coordinate axis of the above semantic space, and here, the relationship between the semantic space representation induced by the stimulus by the training information and the brain activity by the stimulus is referred to.
(3) Presenting new information to the subject, giving a new stimulus to the subject, and inputting the output of the cranial nerve activity detecting means, which detects the brain activity induced by the new stimulus to the subject, to the data processing means For the output of the cranial nerve activity detecting means for the brain activity based on the new information, a probability distribution in the semantic space representing the perceptual meaning content is obtained from the association obtained in (2).
For brain activity induced by a new stimulus, the output of the cranial nerve activity detecting means such as an electroencephalogram and fMRI signal is the output of the cranial nerve activity detecting means induced by the training stimulus, and the signal and firing pattern extracted from it, For example, by decomposing as a linear composition, the perceptual meaning content for the new stimulus can be obtained as a linear composition of a language description (annotation) corresponding to the training information. A probability distribution in the semantic space representing the perceptual meaning content of the new stimulus can be obtained from the linear synthesis coefficient and the association obtained in (2) above.
(4) A highly probable perceptual meaning content is estimated from the probability distribution obtained in (3) above.
In this estimation, dispersion of the estimation results can be suppressed by providing a threshold for the probability used for estimation in the probability distribution, or by providing a threshold for the number of perceptual meaning contents having a high probability.
In the case where there are a plurality of subjects, the association using the meaning space representation of stimulation and brain activity training data in (2) above is performed for each subject, and all or part of the training data is performed for each subject. It is good also as what shifts the correlation with the position in the said semantic space uniformly according to this projection function for every test subject.
In the above (4), in estimating a highly probable perceptual meaning content, a coordinate in the semantic space is found for a given word, and the probability distribution obtained in (3) above is obtained. And the likelihood value can be used as an indicator of the probability.

The present invention makes it possible to estimate arbitrary perceptual meaning content from brain activity under natural perception such as moving images.

FIG. 1 is a conceptual diagram of a brain activity semantic space model and perceptual meaning content estimation. It shows that perceptual meaning content is estimated from brain activity under any new condition by learning the correspondence between brain activity and corpus-derived semantic space as a quantitative model.
FIG. 2 is a diagram illustrating an example of perceptual meaning content estimation from brain activity during TV commercial (CM) video viewing. (Left) An example of a CM clip presented to the subject, (Right) Perceptual meaning content estimated from brain activity at the time of viewing the clip. Each line next to the clip indicates a word that has a high probability of perceiving the part of speech of the noun, verb, or adjective.
FIG. 3 is a diagram showing a specific impression time-series quantitative evaluation example from brain activity. The degree of recognition of a specific impression (in this case “cute”) was estimated from the brain activity during the brain activity while watching three 30-second CMs.
FIG. 4 is a diagram showing an example of a device configuration for applying the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 4 shows an apparatus configuration example for applying the present invention. Presentation of training stimuli (images, moving images, etc.) to the subject 2 is performed by the display device 1, and the brain activity signal of the subject 2 can detect, for example, an EEG (electroencephalogram) or an fMRI signal. Detect with. As a brain activity signal, a brain neuron firing pattern and a signal of activity change in one or more specific regions are detected. The detected brain activity signal is processed by the data processing device 4. The natural language annotation from the subject 2 is input to the data processing device 4. The semantic space used for data processing is obtained by analyzing the corpus data from the storage 5 by the analysis device 6 and stored in the storage 7.
As for training stimuli, natural language annotation data from the subject 2 or a third party is analyzed as a vector in the semantic space by the data processing device 4, and the analysis result as a training result together with the brain activity signal of the subject 2. It is stored in the storage 8.
When a new stimulus is presented to the subject 2 through the display device 1, a brain activity signal is detected by the cranial nerve activity detector 3, and the data processor 4 uses the semantic space from the storage 7 and the training result from the storage 8 as a basis. And the analysis result is output from the data processing device 4.
Here, the storage 5, the storage 7, and the storage 8 may be obtained by dividing one storage area, and the data processing device 4 and the analysis device 6 may be used by switching one computer.
In the estimation method of perceptual meaning content by analyzing brain activity according to the present invention, brain information decoding is performed through a corpus-derived semantic space. In this way, arbitrary perceptual meaning content is decoded from the brain activity. A more specific procedure is as follows and will be described with reference to FIG. FIG. 1 is a conceptual diagram of a brain activity semantic space model and perceptual meaning content estimation. By learning the correspondence between brain activity and corpus-derived semantic space as a quantitative model, an outline of the procedure for estimating perceptual meaning content from brain activity under any new condition is shown.
(A) With respect to the stimulus 11 for training (image, moving image, etc.), a language description (annotation) 13 of the perceptual content that the stimulus induces in the subject is acquired.
More specifically, some still image or moving image (training data) is presented to the subject 12 as a training stimulus, and a list of language descriptions recalled by the subject who has received the presentation is created.
(A) Using a large-scale database such as the corpus 16, a semantic space describing the semantic relationship of words appearing in the language description is constructed. As a means for constructing a semantic space from a corpus, it is already well known to use natural language processing techniques such as a latent semantic analysis method and a word2vec method.
As the corpus, newspaper and magazine articles, encyclopedias, stories and the like can be used. Here, the corpus-derived semantic space is a space for projecting elements such as words to a fixed-length vector space based on statistical properties inherent in the corpus, as is well known. Of course, if this semantic space has already been obtained, it can be used.
The latent semantic analysis method performs the singular value decomposition on the co-occurrence matrix indicating what kind of words are included in the sentence object to be analyzed, and performs the dimensional reduction to reduce the main meaning of the target text. It is a well-known technique that is a principal component analysis technique for grasping the structure.
The Word2Vec method is a quantification method for expressing words as vectors. Word2Vec learns a fixed-length vector space expression of a word through optimizing a word appearance prediction model in a sentence.
(C) In the semantic space obtained in (i) above, the training data is used to project 15 onto the semantic space of the stimulus 11, and the representation on the semantic space and the brain activity output 14 are related.
Training data (images, moving images, etc.) is presented to the subject, and the brain activity signals of the subject, such as EEG (electroencephalogram) and fMRI signals, are detected. The detected brain activity signal is associated with the position in the semantic space. This association associates EEG (electroencephalogram) and fMRI signal waveforms with the representation in the semantic space.
This association is preferably performed for each subject, but in this case, not all of the above training data is performed, but a part thereof is performed to obtain a projection function in the semantic space for each subject, and the projection function Accordingly, the association with the position in the semantic space may be shifted uniformly.
(D) For a new brain activity, a probability distribution in the semantic space representing the perceptual meaning content is obtained from the association obtained in (c) above.
New data (image, video, etc.) is presented to the subject, and the brain activity signal is detected using the same brain activity signal acquisition means as in the case of the training data. The detected brain activity signal is compared with the brain activity signal obtained from the training data to determine which training data is close to. Alternatively, it is determined what kind of hybrid is the closest in the case of training data. This comparison can be performed using, for example, the peak value of the cross-correlation between the brain activity signal for the new data and the brain activity signal for the previous training data as an index. By this determination, a probability distribution corresponding to the brain activity signal detected by the presentation of new data can be obtained in the semantic space.
(E) From the probability distribution obtained in (d) above, a highly probable perceptual meaning content is estimated.
In (a) above, the language description of the perceptual content corresponding to the training data is obtained, and in (a) above, each word is expressed as a vector in the semantic space. For this reason, in the semantic space, the language description of the perceptual content corresponding to the brain activity signal can be obtained with a probabilistic weight from the probability distribution corresponding to the brain activity signal. A probable language description is estimated using this probabilistic weight.
Here, for the list in (a) above, the perceptual content induced by the subject who has been stimulated using this list is expressed as a language description, so all of the above-mentioned corpus-derived semantic spaces or selected It is desirable to cover a predetermined part.
As described above, the present invention provides a technique for estimating arbitrary perceptual meaning content perceived by a subject from brain activity under relatively dynamic and complex video / audio perception such as television commercial (CM). . According to the present invention, it is possible to estimate an arbitrary perceptual meaning content from a brain activity under natural perception such as a moving image. For example, it is possible to quantitatively evaluate from a brain activity whether a moving picture such as the above-mentioned TV commercial exhibits an intended expression effect.

Regarding the handling of annotations in the first embodiment, a topic model in LDA (Lent Dirichlet Allocation) can be applied. This makes it easy to express a sentence in which the perceived meaning content is estimated from the estimated brain activity. An example of the procedure for this is shown below.
(A) With respect to the stimulus 11 for training (image, video, etc.), a language description (annotation) 13 of the perceptual content that the stimulus induces to the subject is acquired.
More specifically, some still image or moving image (training data) is presented to the subject 12 as a training stimulus, and a list of language descriptions recalled by the subject who has received the presentation is created.
(B) Using a large-scale database such as the corpus 16, a topic model that describes the semantic relationship of words that appear in the language description is constructed. This topic model can be prepared by a technique well known in the LDA method.
As is well known, the topic model is a statistical model, and the probability of appearance of each word can be obtained.
(C) In the topic model obtained in (B) above, the training data is replaced with the label of the topic to which the morpheme of the training data belongs, and the label and the brain activity output 14 are related.
In other words, training data (images, moving images, etc.) is presented to the subject, and the brain activity signals of the subject, such as EEG (electroencephalogram) and fMRI signals, are detected. The detected brain activity signal is associated with the label of the topic to which the morpheme of the training data belongs. For this association, for example, a linear combination of labels may correspond to a brain activity signal based on one training data, or conversely, a linear combination of the brain activity signals may correspond to one label.
This association is preferably performed for each subject, but at that time, it is possible to omit a part of the association work by performing a part of the training data instead of the whole of the training data.
(D) For a new brain activity, a probability distribution of a language description represented by a topic model label representing perceptual meaning content is obtained from the association obtained in (C) above.
New data (image, video, etc.) is presented to the subject, and the brain activity signal is detected using the same brain activity signal acquisition means as in the case of the training data. The detected brain activity signal is compared with the brain activity signal obtained from the training data to determine which training data is close to. Alternatively, it is determined what kind of hybrid is the closest in the case of training data. This comparison can be performed using, for example, the peak value of the cross-correlation between the brain activity signal for the new data and the brain activity signal for the previous training data as an index. By this determination, it is possible to obtain the probability distribution of the language description corresponding to the brain activity signal detected in the presentation of new data.
(E) From the probability distribution obtained in (D) above, a highly probable perceptual meaning content is estimated.
In this embodiment, since the probability distribution of the language description is obtained in (D) above, the sentence can be estimated by the same method as LDA.
Here, with respect to the lists in (A) and (A) above, the perceptual content induced by the subject who has been stimulated using this list is expressed as a language description, so all the meaning spaces derived from the corpus Or it is desirable to cover the selected predetermined part.
The present invention provides a technique for estimating arbitrary perceptual meaning content perceived by a subject from brain activity under relatively dynamic and complex video and audio perception such as television commercial (CM). According to the present invention, it is possible to estimate an arbitrary perceptual meaning content from a brain activity under natural perception such as a moving image. For example, it is possible to quantitatively evaluate from a brain activity whether a moving picture such as the above-mentioned TV commercial exhibits an intended expression effect.

The example shown in FIG. 2 is an example of estimation of perceptual meaning content from brain activity during CM video viewing. Specifically, for example, the purpose is to rationally answer the question of how the perception of “intimacy” can be induced in the viewer. This shows the perceptual meaning content estimated from the brain activity by the procedures (a) to (e) above for the presented CM video in FIG. 2, and the left column shows an example of a CM clip presented to the subject, The right column shows the perceptual meaning content estimated from the brain activity at the time of viewing the clip. Each row next to the clip indicates a word having a high probability that the subject perceives the part of speech of each of the noun, verb, and adjective in order from the top.
Figure 2 (a): A scene in which a daughter holds a mobile phone over her mother (noun) Male Female Single Neighborhood Parents Older relatives Visit older mothers (verbs) Quit Dating Get acquainted Join together Visit Die (adjective) Intimate Poor Young Young Figure 2 (b): Dog and owner sitting on a bench with a radio tower and looking at the scenery (noun) Female Male Older Blonde Friend Her Mother Single (verb) Dating Wear Speak Ask Ask Ask Sit Sit (adjective) ) Intimate Friendly Young Young Cute Figure 2 (c): A scene where a dog appears explosively by breaking the center of the scene in (b) (Noun) Face Mouth Glasses Facial expression Personal appearance Mouth Serious (verb) Speaking Dating Angry Wearing Wearing Sit and swing around (adjectives) intimate cute kind young want scary Figure 2 (d): (c) dog campaign for a product (noun) ) Letter Font Logo Gothic Alphabet Display (verb) Replace Write Append It becomes possible to objectively judge whether these perceptual meaning contents representing the viewer's brain activity are in line with the intention of the CM creator.
In addition, the sentence can be estimated by the procedures (A) to (E).

The example shown in FIG. 3 is an example of quantitative evaluation from the brain activity of a specific impression time series. This is for the purpose of providing a quantitative index such as which of the two images A and B gives a specific impression to the viewer more strongly. By confirming whether the specific impression (in this case “cute”) is a highly probable language description from the time series of brain activity during the brain activity while watching three 30-second CMs, This is an estimate of the degree of recognition. CM-1: A scene where a high school girl is talking with a relative, CM-2: A scene where a board meeting is held, CM-3: A scene where an idol is practicing dance, It can be seen that a relatively strong reaction is obtained.

The present invention can be widely used as a basis of a brain machine interface through prior evaluation of audiovisual materials (video, music, teaching materials, etc.) and perception / behavioral intention reading.

DESCRIPTION OF SYMBOLS 1 Display apparatus 2 Test subject 3 Cranial nerve activity detector 4 Data processing apparatus 5 Storage 6 Corpus data analysis apparatus 7, 8 Storage 11 Stimulus 12 Subject 13 Language description 14 Brain activity output 15 Projection to meaning space 16 Corpus

Claims (3)

Information presenting means for presenting information to be a stimulus to the subject, cranial nerve activity detecting means for detecting the cranial nerve activity signal of the subject by the stimulation, data processing for inputting a language description relating to the stimulus content and the output of the cranial nerve activity detecting means Analyzing the brain activity of the subject using a brain activity analyzer comprising: means, semantic space information storage means readable from the data processing means, and training result information storage means readable and writable from the data processing means A method for estimating the perceptual meaning content perceived by the subject,
(1) Training information is presented to the subject and a training stimulus is given to the subject, the language description of the perceptual content that the training stimulus induces to the subject, and the brain that the training stimulus induces to the subject The output of the cranial nerve activity detection means that detected the activity is input to the data processing means,
(2) Applying a semantic space stored in the semantic space information storage means, associating the semantic space representation of the training stimulus with the output of the cranial nerve activity detecting means on the semantic space, and as a result of the association Is stored in the training result information storage means,
(3) Presenting new information to the subject, giving a new stimulus to the subject, and inputting the output of the cranial nerve activity detecting means, which detects the brain activity induced by the new stimulus to the subject, to the data processing means For the output of the cranial nerve activity detecting means of the brain activity based on the new information, a probability distribution on the semantic space representing the perceptual meaning content is obtained from the association obtained in (2) above,
(4) Estimating highly probable perceptual meaning content from the probability distribution obtained in (3) above,
Perceptual meaning content estimation method by analyzing brain activity. For the plurality of subjects, the association using the training information between the meaning space representation of the stimulus and the brain activity in (2) above is performed for all or part of the training information for each subject, and the subject The brain function according to claim 1, wherein a projection function on each semantic space is obtained, and the association with the position on the semantic space is uniformly shifted for each subject according to the projection function. An estimation method of perceptual meaning content by analysis. In the above (4), in estimating a highly probable perceptual meaning content, a coordinate in the semantic space is found for a given word, and an inner product of the coordinate and the probability distribution obtained in (d) above is taken. 3. The method of estimating perceptual meaning content by analyzing brain activity according to claim 1, wherein the inner product value is used as an index of the probability.

Images