FR3070591A1 - METHOD AND MODULE ADAPTED TO IDENTIFY AND QUANTIFY EMOTIONS RELATED BY AT LEAST ONE INDIVIDUAL - Google Patents

Abstract

The present invention relates to means for identifying and quantifying the emotions felt by at least one individual involved in an interactive, immersive and / or sensory experience, by: - collecting, during the experiment, measurements relating to a set physiological characteristics of the individual; extracting a set of characteristic quantities relating to the physiological characteristics, as a function of the measurements relating to the physiological characteristics; determining, for each of the emotional classes of an emotion classification, an intensity perceived by the individual obtained by calculating the image of the set of characteristic quantities of a filter function specific to said at least one individual and to said emotional class.

Description

The present invention relates to the field of means for identifying and representing emotions felt by a person. The invention more particularly finds application in tools for measuring and graphical visualization of the emotions felt, during a sensory experience, by individuals, in particular when the latter interact with an immersive environment generated by a virtual reality computer program. , or attend as spectators to an audiovisual projection.

The evaluation of emotions is of particular importance for designers, screenwriters, and programmers of interactive, immersive and / or sensory experiences. In fact, it is particularly desirable, during the design phase, to check whether the intensity of the emotions felt by individuals at a given time corresponds to that which the experience is supposed to arouse.

Also, for this purpose, portable devices such as bracelets, smart watches or even virtual reality headsets can be used to measure the physiological characteristics relating to the cardiac, dermal, and / or cerebral electrophysiological activity of individuals, part of the experience. From the measures collected, it is then possible to distinguish the emotional state of an individual according to a pre-established classification, for example by determining a probability for each state of the classification according to the measures collected. Visual sensing and processing devices can also be used to identify an individual's emotional state.

Thus, a photographic device can be used to capture the expression of a face, then be interpreted by a processing device to identify an emotional state among those of the classification. However, none of these devices makes it possible to quantify an intensity associated with this emotional state.

Furthermore, these devices are mainly intended to discriminate the dominant primary emotional state, and are not adapted to discern or represent the equally relevant secondary emotional states. In addition, the higher the number of emotional states in the classification, the more difficult it is to identify, discriminate and represent the relevant associated secondary emotional states.

It is also known to represent the emotional state of an individual using a multidimensional model, such as the Plutchik model comprising primary emotions from which secondary emotions are derived. However, such a representation is obtained from subjective data collected from each individual, during a long and tedious step of manual annotation of the emotions felt, during the experience.

This is why there is still a need for automated means of identification and representation of the different emotional states and their respective intensity, relating to an individual involved in an interactive, immersive and / or sensory experience.

One of the objects of the invention is to make it possible to identify and quantify the emotions felt by at least one individual involved in an interactive, immersive and / or sensory experience.

An object of the invention is to provide means suitable for the annotation of content, fast and efficient, using means for collecting objective physiological characteristics, such as portable devices of the bracelets, smart watches or even virtual reality headsets type. .

An object of the invention is to provide means adapted to characterize an experience as a function of physiological data.

Another object of the invention is to provide reliable means for identifying the elements of content capable of causing emotions to be felt and of quantifying their intensity.

Another object of the invention is to provide means of visualization optimized in terms of clarity and fidelity to display a set of emotions aroused by an interactive, immersive and / or sensory experience.

Another object of the invention is to provide additional means to improve the search for content, in particular to allow the organization of the results of a request according to the intensity of the emotions they aroused on a group of 'people.

Another object of the invention is to allow the modulation of the emotional impact of an experience, according to certain characteristics of an individual, such as his age.

One or more of these objects are fulfilled by the method and the device according to the invention.

More particularly, according to a first aspect, the invention relates to a suitable method for identifying and quantifying the emotions felt by at least one individual involved in an interactive, immersive and / or sensory experience. The process includes the following steps:

- a step of collecting, during the experiment, measurements relating to a set of physiological characteristics of the individual;

a step of extracting a set of characteristic quantities relating to the physiological characteristics, as a function of the measurements;

a step of determining, for each of the emotional classes of a classification of emotions, an intensity felt by the individual obtained by calculating the image of the set of quantities characteristic of a filter function specific to said audit at least an individual and said emotional class.

The method can also include a step of producing and displaying a representation of the intensity felt for each of the emotional classes of the classification of emotions.

The representation of the intensity felt for each of the emotional classes of the classification of emotions can be a two-dimensional representation, adapted to be viewed in particular by means of a two-dimensional display device, such as a 2D screen. Thus, the two-dimensional representation can be a radar diagram comprising a radial axis for each of the emotional classes of the classification of emotions, the intensity felt by the user for each of the emotional classes then being transferred to the corresponding axis. The two-dimensional representation can also be a bar diagram, where the statement describing each of the emotional classes is visibly annotated along a horizontal axis, and where the vertical axis is used to represent the intensity felt for the corresponding emotional class, the lowest intensity level being typically located at the horizontal axis.

The representation of the intensity felt for each of the emotional classes of the classification of emotions can also be a three-dimensional representation, adapted to be visualized in a three-dimensional representation system, such as a display device for virtual environment or even of a three-dimensional hologram projection device.

Advantageously, the filter functions, associated with each of the emotional classes of the classification of emotions, each include a set of parameters obtained and / or updated during a calibration phase.

More particularly, the set of parameters obtained and / or updated during a calibration phase for each of the filter functions comprises:

- a coefficient for each characteristic quantity of the set of characteristic quantities; and,

- bias.

As a non-limiting example, the filter functions, associated with the emotional classes of the classification of emotions, are linear transformations.

According to one embodiment, during the calibration phase:

- We collect, for said at least one individual, during a calibration experiment, measurements relating to the physiological characteristics of the individual as well as temporal information relating to said measurements, and at least one recording of a representation of said calibration experience;

- subjective information is collected relating, for each of the emotional classes of the classification of emotions, to the intensity of the corresponding emotion felt by said at least individual, during at least one temporal period of the experience calibration, the subjective information being produced by said at least individual and / or by an observer following viewing of the recording of the representation of the calibration experience;

- on the basis of the subjective information collected and of the measures relating to the physiological characteristics of the individual corresponding to the subjective information collected, the set of parameters of the functions filters the emotional classes of the classification of emotions.

The subjective information is, for example, collected by means of a user interface, capable of allowing input on a representation (for example a two-dimensional or three-dimensional representation) of the intensity felt for each of the emotional classes of the classification of emotions corresponding to an emotional spectrum, for at least one time period of the calibration experience.

The subjective information can be collected for at least one key instant of the calibration experiment, said at least key instant being predetermined before said calibration experiment. Alternatively, the subjective information can be collected for at least one key instant of the calibration experiment, said at least key instant being selected (for example during the recording of the experiment), either subjectively by said at least an individual, either by an external observer.

Alternatively, the subjective information can be collected for at least one key instant of the calibration experiment, said at least key instant being selected, automatically, according to the measurements collected during the calibration experiment and / or of data. behavior relating to said at least individual.

According to a second aspect, the invention also relates to a module suitable for implementing the method according to the first aspect. More specifically, the module adapted to identify and quantify the emotions felt by at least one individual involved in an interactive, immersive and / or sensory experience, includes:

- a module adapted to collect, during the experiment, measures relating to a set of physiological characteristics of the individual;

- a module adapted to extract a set of characteristic quantities relating to physiological characteristics, as a function of the measurements;

a module adapted to determine, for each of the emotional classes of a classification of emotions, an intensity felt by the individual obtained by calculating the image of the set of quantities characteristic of a filter function specific to said at least one individual and to said emotional class.

Other features and advantages of the present invention will appear in the following description of embodiments, with reference to the accompanying drawings, in which:

- Figure 1 is a diagram representing the steps of a process for identifying and quantifying emotions felt by at least one individual involved in an interactive, immersive and / or sensory experience;

FIG. 2 is a diagram of an annotation system suitable for implementing a method for calibrating the filter functions f, of the set F, used during the fourth step of the method of identification and quantification of emotions;

- Figure 3 shows a graphic representation of the emotional spectrum S corresponding to a bar diagram;

- Figure 4 shows a graphic representation of the emotional spectrum S corresponding to a radar diagram;

- Figure 5 is a block diagram of an annotation system, according to an embodiment of the invention multi-site and multi-user.

We refer to FIG. 1, on which are represented the stages of a process of identification and quantification of emotions felt by at least one individual involved in an interactive, immersive and / or sensory experience.

During a first step 110, raw measurements Mb are collected relating to a set of physiological characteristics Cp.j of the individual, j being an integer between 1 and an integer upper bound K. The raw measurements Mb are by example obtained by means of one or more portable devices of the bracelets, smart watches or even virtual reality headsets capable of measuring the physiological characteristics Cp.j such as the cardiac, dermal, and / or cerebral electrophysiological activity of the individual . Such a device is described in particular in French patent application No. 16/61569, and makes it possible in particular to obtain raw measurements Mb relating to the electrodermal response, to the heartbeat, to the acceleration of the movements of the individual, and the temperature at the individual's wrist.

The raw Mb measurements are obtained during the experiment in which the individual is involved. These raw Mb measurements can be obtained in real time, and / or for one or more periods of the experiment. In addition, time markers can be used to locate the raw measurements Mb thus collected over time.

During an optional second step 120, the raw measurements Mb are filtered so as to optimize their reliability. More particularly, the raw measurements Mb can in particular be smoothed temporally, and / or even be subject to correction by decomposition into a main or independent component to attenuate the artefacts linked to the movements, as described for example in the patent document US 2005 0058456 A1, or in the article "Motion artifact removal from photoplethysmographic signais by combining temporally constrained independent component analysis and adaptive filter", Peng et al., Biomed Eng Online. 2014; 13:50, Published online 2014 Apr 24. doi: 10.1186 / 1475-925X-13-50, PMCID: PMC4021027.

The raw measurements Mb can also be subject to noise reduction processing, for example by applying noise reduction processing in the frequency domain.

During a third step 130, characteristic quantities relating to the physiological characteristics Cp are determined, as a function of the raw measurements Mb, advantageously filtered. More particularly, to each of the physiological characteristics Cp.j is associated a variable Vj whose value is denoted Xj. The value Xj is determined by calculating the image of a function Fj from the raw measurements Mb, advantageously filtered, or from a subset of the raw measurements Mb relevant to the physiological characteristics Cp.j.

Thus, for the physiological characteristic corresponding to the electrodermal response, the corresponding value Xj can be determined by calculating the value of a function Fj of derived type over a determined time window for the filtered measurements provided by the sensors measuring the electrical activity in surface of the individual's skin. More generally, by way of nonlimiting example, the function Fj can correspond to an average, a variance, a slope, a time derivative, a function relating to the power of a signal contained in a frequency band, a suitable function. to determine a frequency, consistency, variability, and / or a combination of the previously mentioned functions. Thus, at the end of the third step 130, it is then possible to determine a scalar vector X of characteristic quantities relating to the physiological characteristics Cp, such as:

During a fourth step 140, an intensity e, is determined for each emotional class E, of a classification Cl as a function of the characteristic quantities relating to the physiological characteristics Cp, obtained during the third step 130. The classification Cl has an integer N of emotional classes E ,, each corresponding to an emotional state of the individual. More particularly, during a fourth step 140, for each emotional class E ,, the intensity e is determined, by means of a filter function f ,, specific to said emotional class E ,, applied to the scalar vector X of 10 characteristic quantities relating to physiological characteristics Cp.

Thus, if we note the set of filters F = {f,} î = i ... n, then fj (X) = ej. The set of intensities {e,} î = i ... n thus obtained then describes an emotional spectrum S, as shown in FIG. 3.

The following table describes an example of classification comprising a number N equal to 20 of emotional classes Ej:

i Emotional class E, Associated filter fi (X) 1 Interest f Interest 2 Fun fAmusement 3 Pride fFierté 4 Joy fjoie 5 Pleasure fpiaisir 6 Satisfaction fcontentement 7 Love famour 8 Admiration fAdmiration 9 Relief fsoulagement 10 Compassion fcompassion 11 Sadness fïristesse 12 Guilt fculpabilité 13 Regret f regret 14 Shame fhlonte 15 Disappointment fDéception 16 Fear fpeur 17 Disgust füégoût 18 Contempt flVIépris 19 Hatred fhlaine 20 Anger fcolère

The filter functions f, of the set F = {f,} î = i ... n typically include parameters, obtained or updated during a calibration phase. Thus, by way of nonlimiting example, the filter f, for the emotional class E, can correspond to a linear transformation described by the following mathematical expression:

e, = fj (X) = x. e, '+ b, with ^c i' = {A) j = i ... k a transposed vector containing coefficients for characteristic quantities relating to physiological characteristics Cp;

b, a bias, such that bj = êj - x. c / with êj an average intensity relative to the emotional class E, and x an average scalar vector through the observations;

e, ’and b, being obtained or updated during one or more calibration phases.

This mathematical expression is thus obtained by normalizing the characteristic quantities of the scalar vector X - typically by considering a zero mean and a standard deviation equal to 1 for the scalar vector X across all observations - and by estimating the coefficients c, 'in setting b = 0. The bias b, is then estimated by calculating the average of the scalar vector X before normalization and the intensity of the emotions.

In an advantageous embodiment of the method according to the invention, in particular when the latter is implemented in an autonomous device, a multiple linear regression model is used to determine the set of intensities {e,} î = i .. .n describing the emotional spectrum

An example of an adapted multiple linear regression model is for example described in the document entitled Christopher M. Bishop: Pattern Recognition and Machine Learning, Springer, 2006.

The algorithms used for determining the multiple linear regression model having a low complexity and requiring a low execution time therefore prove to be very suitable for their implementation in a processing unit for a mobile device. The time between obtaining the raw measurements Mb and the estimation of the emotional spectrum S is then reduced enough to allow the display of the latter in real time, including on a device having a low computing power.

In an embodiment of the method according to the invention advantageous in particular when the fourth step is implemented in a remote device, complex learning techniques, such as the use of recurrent or convolutional neural networks, are used to determine the set of intensities {e,} î = i ... n describing the emotional spectrum S. An example of complex learning techniques is for example described in the document entitled lan Goodfellow and Yoshua Bengio and Aaron Courville, Deep Learning , MIT Press, 2016.

Since processing latency is not a priority, it is possible to use finer and / or precise analysis methods.

During a fifth optional step, a representation of the set of intensities {e,} î = i ... n and of the emotional classes E, corresponding to the emotional spectrum S is generated, and advantageously, displayed.

FIG. 4 illustrates, according to a first embodiment, a graphic representation of the emotional spectrum S corresponding to a radar diagram, where each radial axis is used to quantify the intensity e, of one of the emotional classes E ,. The lowest intensity level, e, = 0, is located at the center point of the radar diagram. This graphical representation is notably adapted to present the emotional spectrum S when the number N of emotional classes E, is strictly greater than 1.

FIG. 3 illustrates, according to a second embodiment, a graphic representation of the emotional spectrum S corresponding to a bar diagram, where the statement describing one of the emotional classes E, is annotated visibly along a horizontal axis, and where l the vertical axis is used to represent the intensity e, of the corresponding emotional class E. The lowest intensity level, i.e. e, = 0, is located at the horizontal axis.

Alternatively, this graphic representation can be used to present the emotional spectrum S when the latter have only one emotion, that is to say when the number N of emotional classes E, is equal to 1: the graphic representation does not then has a vertical bar on the horizontal axis.

According to the first or second embodiment, the order and the statement of the axes of the diagram should preferably correspond to that of the emotional spectrum S, in order to optimize the interpretation of the representations and facilitate their use in a user interface.

The representations can also include a plurality of emotional spectra, for example to simultaneously compare several individuals or several situations.

Referring now to FIG. 2 in which is represented, an annotation system adapted to implement a method for calibrating the filter functions f, of the set F, used during the fourth step 140 of the identification method and quantification of emotions previously described. The annotation system makes it possible to adjust the model described by the filter functions f, to estimate the emotional spectrum.

The annotation system typically includes a processing unit 310, a storage unit 320, a display unit 330 and a user interface 340 to allow entry of subjective reports. The annotation system can include fixed computing means personal computer, workstation, server, etc. - or mobile - smartphone, tablet, laptop, etc.

The processing unit 310 of the annotation system is configured to collect measurements relating to the physiological characteristics Cpj of an individual as well as temporal information relating to the measurements, while the latter is involved in an interactive, immersive and / or sensory, then record these in storage unit 320.

The processing unit 310 is also configured to obtain or determine a representation of a recording of the experience having induced the emotions, so as to allow their annotation a posteriori. The representation corresponds, for example, to a video extract from the film that the individual watched, a sequence of screenshots taken while the individual was playing a video game or having a virtual reality experience, or a video recording of the route taken while driving the vehicle.

The user interface 340 is arranged so as to allow the entry of information capable of identifying the emotional classes E, corresponding to the emotions felt during the experience, as well as their intensity. The user interface 340 includes, for example, input means by motor commands - pointing device, keyboard, tactile or voice device. The voice commands include, for example, an indication of the name of the emotion felt and a description of the intensity such that I was very afraid, I am not happy, I am sad.

The display unit 330 includes a reading module capable of displaying the representation of the recording of the past experience having induced the emotions. The reading module includes a time marker, called a reading head, pointing to the instant corresponding to the current display of the representation.

The read head is temporally synchronized with the recorded measurements to allow the identification of the recorded measurements corresponding to the emotion described by means of the user interface 340 and corresponding to the current display of the representation.

The reading module is typically provided with a reading / pause / stop / forward / fast reverse type transport module coupled to the display of the representation. The playback module can also include means for adjusting the audio volume of the representation.

The display unit 330 also has a set of buttons for moving the read head at key moments in the representation of the recording of the past experience. These buttons can be represented by a miniature image of the representation of the recording corresponding to the corresponding key moment.

In a pre-established operating mode, the key images can be selected before recording: they then correspond to moments which are a priori likely to induce significant emotions. For example, the preset keyframes can correspond to the start of the most striking scenes of a film or to the moment when the player lost his game.

In a manual operating mode, the key images are selected during the recording of the experience, either subjectively by the individual, or by an external observer for example on the basis of a behavioral study of the individual . The keyframes can thus correspond to striking scenes of a video game identified by the player by pressing a button on his controller.

In an automatic operating mode, the key images are selected automatically, as a function of the measurements obtained and / or of behavioral data. The automatic operating mode is adapted to improve the performance of an already trained model, by selecting the keyframes whose emotional spectrum S exceeds an intensity threshold which can be adjusted manually or automatically. This operating mode also makes it possible to adjust the model by minimizing, for any keyframe introduced automatically, the difference between the emotional spectrum reported subjectively on the emotional equalizer and the spectrum estimated from the recorded measurements.

The display unit also includes an emotional equalizer module, coupled to the reading module, capable of displaying a representation such as that illustrated in FIG. 4 and / or 5, on which, via the user interface 340, the emotions felt by the individual and their respective intensity, at the instant pointed by the reading head of the reading module, can be reported subjectively.

The emotional equalizer module allows, by manual actions, to indicate and / or modify the intensity e, on each of the axes of the graph of the graphical representation. For example, it is possible to use a pointing device such as a computer mouse or a touch screen to adjust the intensity e, of an emotion, by pointing on the axis of the emotion corresponding to the place of the intensity felt.

The annotation system can in particular implement machine learning methods. The physiological characteristics extracted in step 4 of the treatment procedure and the subjective data are used to train the machine learning algorithm. In the particular case of a linear regression method, the quantities characteristic of the scalar vector X are applied as predictive variables while the subjective data are the variables to be predicted.

Thus, by way of nonlimiting example, the coefficients c, ’of the filter f, can be calculated by minimizing the following mathematical expression:

min _c || x. c, '- e, ||

Depending on the machine learning method applied, the minimization criterion can correspond to a classification cost function classification error, logistic function, hinge or square loss - or regression - least square error, absolute value, entropy, etc.

In an on-board embodiment, the coefficients of the adaptive filters are calculated by an optimization procedure to estimate the coefficients of a linear model as a least squares or gradient descent minimization procedure, possibly with a regularization term.

In the remote embodiment, the computing resources of the processing unit must allow the execution of an optimization procedure to estimate the coefficients of more complex, even non-linear models such as artificial neural networks.

The annotation system can in particular be used in a first operating mode in which a subjective report is established a posteriori. In this operating mode, subjective reports are entered manually by the individual involved in the experiment, by means of the user interface 340, and more particularly by means of the read head of the display module 340 and of the module. emotional equalizer, adjusting the emotional equalizer according to the intensity of the emotions felt during the experience.

The annotation system can in particular be used in a second operating mode in which a subjective report is established in real time. Subjective reports can be captured by a human operator based on observation of the individual involved in the experiment, or by a computer program that recognizes the individual's facial or vocal expressions.

In this operating mode, the subjective reports are synchronized in time, preferably by minimizing the latency, between the collected measurements relating to the physiological characteristics and behavioral information recorded during the experiment. Behavioral information such as the individual's facial or vocal expressions can thus help the human operator or the computer program to annotate the physiological data measured by the device.

Reference is now made to FIG. 5 in which an annotation system is represented, according to an embodiment of the invention multi-site and multi-user. A plurality of measurement devices is used to measure the physiological activity of several individuals, distributed over several sites and / or practicing various activities.

The measurements are collected by remote servers to be stored in databases specific to each situation. The collected measurements can in particular be used to adjust the model described by the filter functions f ,. The emotions felt by individuals 5 can be viewed by a client through a dashboard, provided with an interface to graphically represent emotional spectra.

Claims (11)

1. A suitable method for identifying and quantifying the emotions felt by at least one individual involved in an interactive, immersive and / or sensory experience, characterized in that it comprises the following stages: - a step (110) of collecting, during the experiment, measurements relating to a set of physiological characteristics of the individual; - a step (130) of extracting a set of characteristic quantities relating to the physiological characteristics, as a function of the measurements; a step (140) of determining, for each of the emotional classes of a classification of emotions, an intensity felt by the individual obtained by calculating the image of the set of quantities characteristic of a specific filter function audit at least one individual and said emotional class. 2. Method according to claim 1, further comprising a step (150) of producing and displaying a representation of the intensity felt for each of the emotional classes of the classification of emotions. 3. Method according to any one of the preceding claims, in which the filter functions, associated with the emotional classes of the classification of emotions, each comprise a set of parameters obtained and / or updated during a calibration phase. 4. Method according to claim 3, in which the set of parameters obtained and / or updated during a calibration phase for each of the filter functions comprises: - a coefficient for each characteristic quantity of the set of characteristic quantities; and, - bias. 5. Method according to any one of claims 3 or 4, in which during the calibration phase: - We collect, for said at least one individual, during a calibration experiment, measurements relating to the physiological characteristics of the individual as well as temporal information relating to said measurements, and at least one recording of a representation of said calibration experience; - subjective information is collected relating, for each of the emotional classes of the classification of emotions, to the intensity of the corresponding emotion felt by said at least individual, during at least one temporal period of the experience calibration, the subjective information being produced by said at least individual and / or by an observer following viewing of the recording of the representation of the calibration experience; - on the basis of the subjective information collected and of the measures relating to the physiological characteristics of the individual corresponding to the subjective information collected, the set of parameters of the functions filters the emotional classes of the classification of emotions. 6. Method according to claim 5, in which the subjective information is collected by means of a user interface, capable of allowing the input on a two-dimensional representation of the intensity felt for each of the emotional classes of the classification of emotions, for at least one time period of the calibration experiment. 7. Method according to any one of claims 5 and 6, in which the subjective information is collected for at least one key instant of the calibration experiment, said at least key instant being predetermined before said calibration experiment. 8. Method according to any one of claims 5 and 6, in which the subjective information is collected for at least one key instant of the calibration experience, said at least key instant being selected, either subjectively by said at least an individual, either by an external observer. 9. Method according to any one of claims 5 and 6, in which the subjective information is collected for at least one key instant of the calibration experience, said at least key instant being selected, automatically, according to the measurements. collected during the calibration experiment and / or behavioral data relating to said at least individual. 5 10. Module adapted to identify and quantify the emotions felt by at least one individual involved in an interactive, immersive and / or sensory experience, characterized in that it comprises: - a module adapted to collect, during the experiment, measurements relating to a set of physiological characteristics of 10 the individual; - a module adapted to extract a set of characteristic quantities relating to physiological characteristics, as a function of the measurements; - a module adapted to determine, for each of the classes 15 emotional of a classification of emotions, an intensity felt by the individual obtained by calculating the image of the set of quantities characteristic of a filter function specific to said at least one individual and to said emotional class.