FR3153234A1 - Method for detecting an emotion by radio frequency - Google Patents

Abstract

The present invention relates to a method for detecting an emotion in a subject (11) comprising the following steps: - transmitting a first radio frequency signal (4) towards the subject, - receiving a second radio frequency signal (5) containing a physiological signal due to the reflection of said first radio frequency signal on the subject, - digitally processing (6) the second radio frequency signal (5) so as to generate respiratory data (6a) and/or cardiac data (6b), - applying several time windows F1 of duration T1 in a sliding manner to the respiratory data and/or the cardiac data, - applying several time windows F3 of duration T3 in a sliding manner to the respiratory data and/or the cardiac data; the duration T3 being less than the duration T1, - predicting an emotion by an emotion model (12) based on artificial intelligence from the data originating from the windows F1 and the data originating from the windows F3. See Figure 3.

Description

Process for detecting an emotion by radio frequency.

The present invention relates to a method for detecting an emotion in a subject using an emotion model based on artificial intelligence. The invention also relates to a system comprising a processing unit for implementing such a method.

The invention finds a particularly interesting application in the detection of a user's emotion on a workstation. But the invention has a broader scope since it can be applied to many other professional or specific fields such as human-machine interaction, virtual reality, games and healthcare.

Society continues to evolve towards an increasingly present integration of technology into our daily lives.

Our relationship with consumption, health, education, information, and even the very functioning of our society today requires ever-greater mastery of the digital world. It must be said that the technological advances of recent decades and the advent of the Internet and a more globalized economy have greatly facilitated access to and use of high-tech products, both financially and functionally.

And yet, digital interfaces remain generally passive with regard to their users and what drives them: their intentions, their emotions beyond their interactions. They are unable to recognize episodes of frustration when the user is confronted with a poorly designed web interface. It is impossible to monitor the concentration and fatigue of an operator at a risky industrial workstation. It is impossible to recognize the despair of a student overwhelmed with knowledge in their learning or, conversely, to celebrate a moment of happiness with a player proud of their prowess in a video game. If these interfaces were capable of sensing the emotions and intentions of their users, we would no longer be obliged to use explicit commands and would benefit from a more natural and therefore even more accessible interaction. Moreover, machines would demonstrate their ability to support humans in regulating their own emotions, which would result in greater safety and well-being.

Today, most affective computing approaches rely on the exploitation of audiovisual information such as facial expressions, posture, or prosody. These are very useful because they are easily accessible, but they are not exempt from potential biases. Since facial expressions are a mode of social communication, they are rarely expressed by a user alone in front of their screen. Conversely, they may be forced to hide an emotion of opposite valence. The same is true for the expression of emotions through voice. Moreover, facial expressions are not necessarily visible in the camera's field of view, and their detection may be limited in an uncontrolled environment due to lighting conditions.

To overcome these limitations, we can focus on more physiological and therefore more unconscious signals induced by the regulation of the autonomic nervous system, which is closely linked to emotional changes. Systems are known that use an electrocardiogram on the chest to monitor cardiac activity. Electrodes distributed over the body can measure galvanic conductance and therefore variations in perspiration. An electroencephalogram can characterize neuronal activity, while an oximeter can monitor a user's pulse. By processing these signals with trained artificial intelligence modules, it is possible to extract a certain dynamic from the subject's state and feelings. Such measurement solutions, practical in the laboratory, quickly prove uncomfortable in the everyday environment.

Non-invasive sensors are known for emotion recognition. The use of thermal cameras to capture surface temperature differences on the face is increasingly popular. While this approach has the merit of focusing on the physiological component of emotion, it nevertheless has the constraint of requiring the target's face to be visible.

In recent years, some researchers have also used radar technologies in certain radio frequency ranges to remotely detect micro-movements of the rib cage and, indirectly, of the heart muscles. By processing these micro-body movements, they have accessed real-time changes in heart rate, which contain diagnostic information for assessing emotions.

Document US20170311901A1 is known, describing a radar system for emotion recognition. The radar used is a frequency-modulated continuous wave radar (FMCW). Emotion characterization is done by a discrete classification of four emotions over a 2-minute time window.

We know the paper Siddiqui H. U. R. et al., (2021), “Respiration Based Non-Invasive Approach for Emotion Recognition Using Impulse Radio Ultra Wide Band Radar and Machine Learning.”, Sensors, 21(24), Article 24. https://doi.org/10.3390/s21248336, describing a method to extract the number of respirations per minute from a signal coming from a UWB radar.

We know the document Khan et al., (2021), “Deep learning framework for subject-independent emotion detection using wireless signals.”, PLOS ONE, 16(2), e0242946. https://doi.org/10.1371/journal.pone.0242946, describing a radar system for emotion recognition. Emotion characterization is done by discrete classification of several emotions over long time windows (2 min). It describes an artificial intelligence model built directly from the raw data of the radar signal.

We know the paper Dang et al., (2022), “Emotion recognition method using millimetre wave radar based on deep learning.”, IET Radar, Sonar & Navigation, 16(11), 1796‑1808. https://doi.org/10.1049/rsn2.12297, describing a 77 Ghz FMCW millimetre radar for emotion recognition. This paper describes signal processing to extract cardiac and respiratory activities, both of which are considered in the model and combined for classification.

The present invention aims at a new real-time method of recognizing emotions using radar.

Another aim of the invention is accurate and reliable recognition of emotions.

At least one of the objectives is achieved with a method for detecting an emotion in a subject comprising the following steps:

- emission of a first radio frequency signal towards the subject,

- reception of a second radio frequency signal containing a physiological signal due to the reflection of said first radio frequency signal on the subject,

- digital processing of the second radio frequency signal so as to generate respiratory data and/or cardiac data,

- application of several F1 time windows of T1 duration in a sliding manner on the respiratory data and/or the cardiac data,

- application of several time windows F3 of duration T3 in a sliding manner on the respiratory data and/or the cardiac data; the duration T3 being less than the duration T1, and

- prediction of an emotion by an emotion model based on artificial intelligence from data from F1 windows and data from F3 windows.

The method according to the invention offers a non-invasive solution, usable in a natural environment and capable of recognizing the emotional dynamics of a subject as close as possible to their physiology without requiring the wearing of accessories in contact with the body.

With the use of large and small windows, we improve the characterization of cardiac data and the extraction of components of interest to feed the associated AI model.

Large window analysis allows for consideration of overall physiological activity throughout the user's observation period.

The analysis of small windows makes it possible to consider more local activities specific to an emotional event and to characterize in the signal resulting from physiological activity specific patterns of more local emotional events to be detected.

F1 time windows preferably have the same fixed duration, but it is possible to consider F1 time windows with different durations. The same is true for F3 time windows.

Performing analysis of physiological signals over short time windows makes it possible to obtain predictions over short periods, which allows real-time prediction of sudden emotional events during the exploitation phase of the emotion model.

We can thus evaluate the dynamics of emotions by targeting the intensity of one or more specific emotions or internal states or cognitive intentions over time on a human or animal subject.

Physiological data includes, for example, but is not limited to, heart rate fluctuations, respiratory fluctuations, eye activity, galvanic skin responses and blood pulses.

With the method according to the invention, two dynamics of emotional physiology are considered: a slow dynamic corresponding to progressive variations of the targeted emotional dimension, such as stress or joy, and a much faster dynamic, in the space of a few seconds or less, corresponding to more sudden and local variations of this same emotional dimension.

It is considered that the information specific to the emotion experienced occurs in physiological activity over latencies much shorter than 1 min, in the space of a few seconds. It is this information that is sought by using short time windows as labeled data in addition to long windows, the latter allowing, for example, the extraction of a reference emotional level.

According to an advantageous characteristic of the invention, the data from the time window F1 can be directly transmitted to the emotion model and/or can undergo a calculation of a value of at least one respiratory and/or cardiac biomarker, this value then being transmitted to the emotion model.

In particular, the calculation of a value of at least one respiratory and/or cardiac biomarker may comprise at least one of the following calculations:

- calculation of heart rate variability (HRV),

- calculation of respiratory variability (RRV for “Respiratory Rate Variability”) or of a respiratory pattern (BPA for “Breathing Pattern Analysis”) or of a respiratory amplitude and its variability (RAV for “Respiratory Amplitude and Variability”),

- calculation of respiratory sinus arrhythmia (RSA for “Respiratory Sinus Arrythmia” in English).

RSA is representative of the interaction between cardiac activity and respiratory activity, reflecting the activity of the parasympathetic autonomic nervous system.

Thus, the method according to the invention can be applied to cardiac activity alone, to respiratory activity alone or to both activities.

Depending on the activity used, one or more of the functions listed above can be calculated.

According to an advantageous characteristic of the invention, the duration T1 is at least 30 seconds with a sliding step of at least 1 millisecond; preferably the duration T1 is between 30 seconds and 5 minutes with a sliding step of between 1 millisecond and 15 seconds; ideally the duration T1 is between 30 seconds and 1 minute 15 seconds with a sliding step of between 3 seconds and 5 seconds.

The T1 duration corresponds to large time windows which are applied in a sliding manner, with a fixed time shift or not.

The physiological data obtained from these long windows, particularly beyond the minute, allow for example the calculation of heart rate variability (HRV) which requires waiting for several heartbeats to be calculated.

Advantageously, the duration T3 can be a minimum of 375 milliseconds, or between 375 milliseconds and 30 seconds, or between 1 and 5 seconds, with a step of at least 1 millisecond.

The long window can have a duration of 1 min (T1), and be applied in a sliding manner in steps of 3 seconds during the entire observation time of the subject.

The short window can have a duration of 3 seconds (T3), and be applied in a sliding manner in steps of 500ms during the entire observation time of the subject.

According to the invention, the first radio frequency signal may be a continuous wave radar signal modulated according to an FMCW protocol (for “Frequency-Modulated Continuous Wave” in English) or an ultra wide band ULB signal (or UWB for “Ultra Wide Band” in English).

For example, FMCW radar can operate in both frequency bands [57-64GHz] and [76-81GHz] up to 4 GHz. UWB radar can operate in a range between 6 and 8.5 GHz.

According to the invention, the respiratory data and/or the cardiac data may undergo a normalization step before applying the time windows.

• L’intégralité des signaux cardiaques présente dans les fenêtres successives F1
• L’intégralité des signaux respiratoires présente dans les fenêtres successives F1
• Des mesures cardiaques clé issues des temporalités F1 : par exemple HRV (“Heart Rate Variability”)
• Des mesures respiratoires clé issues des temporalités F1 : par exemple RRV (“Respiratory Rate Variability”), BPA (“Breathing Pattern Analysis”, RAV (“Respiratory Amplitude Variability”)
• Des mesures reflettant la synchronisation entre activité cardiaque et respiratoire issues des temporalités F1, par exemple RSA (“Respiratory Sinus Arrythmia”)
• L’intégralité des signaux cardiaques présente dans les fenêtres successives F3
• L’intégralité des signaux respiratoires présente dans les fenêtres successives F3.

Thus, according to the invention, the data from cardiac and/or respiratory activities are processed independently through two long sliding windows F1 and short sliding windows F3 to extract one or more data from among:

• The entirety of the cardiac signals present in the successive F1 windows
• All respiratory signals present in successive F1 windows
• Key cardiac measurements from F1 timescales: for example HRV (“Heart Rate Variability”)
• Key respiratory measures from F1 timescales: for example RRV (“Respiratory Rate Variability”), BPA (“Breathing Pattern Analysis”, RAV (“Respiratory Amplitude Variability”)
• Measures reflecting the synchronization between cardiac and respiratory activity from F1 temporalities, for example RSA (“Respiratory Sinus Arrythmia”)
• The entirety of the cardiac signals present in the successive F3 windows
• All respiratory signals present in successive windows F3.

This data is then used by the emotion model to predict emotions.

Additional data from other types of sensors than the aforementioned antennas can be used in operation to help weight the output results of the chosen Deep-Learning AI model.

With the invention, through the analysis of the characteristics of the time and frequency domain data, the periodic heart rate and respiratory data are extracted.

Radar sensor outputs are representations of physiological activities in their own dimensional, temporal, or frequency spaces embedded in the noise signal. The signal is filtered and processed to reveal as accurately as technically possible the physiological signal from the converted baseband signal.

A clutter suppression algorithm can be implemented. Through a (sliding in time) window of transmission and reception applied to the desired signature, static states as well as slow movements, therefore different from the desired modulations: rib cage, cardiac and respiratory activity, are excluded. The interference between harmonics and the fundamental frequency of the heart and respiration can be improved by implementing an empirical mode set decomposition algorithm and decomposing the signal into an Intrinsic Mode Function (IMF). The improvement of the heart and respiratory rate can be ensured by the continuous wavelet transform (CWT).

The invention makes it possible to extract complete cardiac and respiratory signals according to the chosen short and long time windows, but also classic characteristics of cardiac and respiratory activities such as HRV, RRV, BPA, RAV, RSA and other characteristics within the same heartbeat or in relation to the relationship between different cardiac characteristics of successive heartbeats.

The data is then optionally normalized and missing values can be filled in if necessary. All of this data can then be optionally converted into vectors to be used as inputs to artificial intelligence-based emotion models.

According to an advantageous characteristic of the invention, the emotion model is made up of one or more learning models, simple, deep (“deep learning” in English) or preferably hybrid.

The emotion model may include a hybrid recurrent neural network. The hybrid recurrent neural network allows the artificial intelligence model to be trained and evaluated to detect and classify significant changes in cardiac and respiratory activities from the calculated cardiac and respiratory characteristics.

The emotion model may comprise a fusion between a gated recurrent unit (GRU) and a long-short-term memory (LSTM) network. Preferably, the hybrid recurrent neural network results from such a fusion.

Such a fusion is well-suited for predictive and classification analysis of time series on a reasonably small dataset. This fusion results in an improved and more accurate model. The GRU and LSTM networks are combined to take advantage of their respective strengths, including the ability of GRU to identify short-term dependencies and the ability of LSTM to identify long-term dependencies. By combining the LSTM and GRU models, the invention identifies different types of temporal patterns or dependencies present in the data, thereby improving the emotion model's ability to make accurate predictions with preserved robustness and generalization ability. This also reduces the risks of overfitting and makes the whole system more resilient to data noise or uncertainties.

Additionally, the simplicity of GRU improves the explainability of algorithmic decisions.

Advantageously, the emotion model can include:

- a slow dynamics model dedicated to data from F1 time windows,

- a fast dynamics model dedicated to F3 time window data, and

- a decision-making algorithm common to the slow dynamics model and the fast dynamics model.

According to an advantageous characteristic of the invention, for example in the context of using an FMCW radar, the transmission of the first radio frequency signal can be carried out by means of several transmission antennas and the reception of the second radio frequency signal can be carried out by means of several reception antennas according to MIMO technology (for “Multiple-Input Multiple-Output” in English).

FMCW radar can therefore be based on a coherent MIMO system with more than one transmitting antenna and more than one receiving antenna. Coherent MIMO beamforming can be based on a synthesizer that generates chirps using the binary phase modulation method, which allows simultaneous transmission from each antenna for a better signal-to-noise ratio (SNR).

Of course, the invention can be implemented with simple antennas, without MIMO technology, with other technologies.

According to another aspect of the invention, there is provided a method for generating training data for an artificial intelligence-based emotion model, this method comprising the following steps for a subject:

- production of a training signal of duration T0 containing emotion-inducing stimuli that the subject receives,

- emission of a first radio frequency signal during the duration T0 towards the subject,

- reception of a second radio frequency signal containing a physiological signal due to the reflection of said first radio frequency signal on the subject,

- digital processing of the second radio frequency signal so as to generate respiratory data and/or cardiac data.

The method according to the invention further comprises the following steps:

- definition of a time window F1 of duration T1 less than T0,

- applying multiple F1 time windows in a sliding manner to the respiratory data and/or cardiac data; the emotion model training data comprising data from each F1 time window,

- definition of a time window F2 of duration T2 less than T1, and of a window F3 of duration T3 less than T2,

- during the emission of the first radio frequency signal, detection of events originating from the subject or calculated on the basis of the subject's behavior;

- at each event, application of several F3 time windows in a sliding manner on the respiratory data and/or the cardiac data of a F2 time window preceding the event; the training data of the emotion model comprising data from each F3 time window.

Events originating from the subject or calculated based on the subject's behavior are understood to mean any event initiated by the subject or not. This may be a response from the subject via an input interface of a processing unit or a signal automatically generated by a tool for monitoring the subject's activity. This event may be calculated based on results from other physiological sensors, for example GSR (for "Galvanic Skin Resistance" in English), EEG (Electroencephalography) sensors, EMG (Electromyography) sensors, or PPG (Photoplethysmography) sensors.

According to an advantageous characteristic of the invention, the training signal may be a virtual reality environment in which the subject is immersed. This training signal may be any other video and/or audio signal but preferably a 3D virtual reality environment.

A training signal is a signal containing one or more stimuli intended to induce a predetermined emotion in the subject. This signal includes stimuli but also calm, emotionless moments, allowing the generation of comparison labels with respect to the stimuli.

According to one embodiment of the invention, the training signal may comprise one or more stimuli capable of triggering a single stress emotion in the subject.

With the invention, we thus provide a characterization of the dynamics in real time on a single given dimension, for example stress, unlike for example a discrete classification of four emotions over a long time window of for example 2 min as in the prior art.

Other characterization techniques can be used, such as continuous dimensions Valence and Intensity in indirect measurement, then translated into emotional categories or level categories (low, medium, high stress).

According to an advantageous characteristic of the invention, the training signal can comprise several stimuli spaced apart in time, each stimulus being capable of triggering the same predetermined emotion; the stimuli being differentiated by their level of difficulty, complexity, repetitiveness, effort of attention, suddenness or duration.

The use of multimodal stimuli is planned. These stimuli are made to evolve according to a controlled independent variable that varies along a unidimensional quantitative axis. This could be a level of difficulty, complexity, repetitiveness, attention effort, suddenness or even a task duration that can be objectively measured or estimated by evaluators and that has an impact on physiology without requiring verbal feedback from the subject. Such a stimulus can lead to the induction of an associated internal state such as stress, cognitive load, concentration, tunnel effect, boredom, drowsiness, fatigue, surprise, fear.

In other words, all stimuli can, for example, induce a certain type of stress by varying the difficulty of the task undergone by the subject to induce periods of more or less strong stress, but we can also consider inducing different types of emotions: stress, fear, joy, etc. Generally speaking, we also plan periods of calm to allow for classification.

Advantageously, the duration T2 can be between 1 second and 30 seconds; the duration T3 can be at least 375 milliseconds, or between 375 milliseconds and 30 seconds, or between 1 and 5 seconds, with a step of at least 1 millisecond.

The minimum duration of T3 corresponds to a single heartbeat of a newborn, for example.

For each response from the subject, several F3 time windows are applied in a sliding manner, with a fixed time shift or not.

We can consider a training signal that lasts 3 minutes (T0). The subject is free to respond to an emotional feeling whenever he wishes, particularly on a regular basis and especially when he feels a strong emotional change.

The long window can have a duration of 1 min (T1), and be applied in a sliding manner in steps of 3 seconds during the entire observation time of the subject.

For the short window, we can pre-segment a 10-second period of interest (T2) before each response triggered by the subject, and within this period of interest, we retrieve the data from the short 3-second time windows (T3) in steps of 500 ms for example, which makes 15 3-second extracts.

With the method according to the invention, after being trained, the emotion model can be deployed on an edge computing or embedded platform connected to radar sensors pointed at a targeted workspace. Once the human target is detected, the solution starts an initialization step by fine-tuning the model on this specific target with adjusted hyperparameters and architecture based on the first captured data and additional data from optional additional sensors.

Generally, T1 can be greater than 30 seconds and T3 can be less than 30 seconds.

According to yet another aspect of the invention, there is provided a system for detecting an emotion in a subject comprising:

- a processing unit provided with an emotion model and configured to implement a method for detecting an emotion in a subject as described above,

- at least one transmitting antenna connected to the processing unit and arranged to transmit the first radio frequency signal towards the subject,

- at least one receiving antenna connected to the processing unit and arranged to receive the second radio frequency signal containing the physiological signal due to the reflection of said first radio frequency signal on the subject.

The system according to the invention includes the following advantages:

- Non-intrusive: Unlike camera-based systems, which require capturing visual images of individuals, millimeter-wave radar operates within the radio frequency spectrum. This makes the system non-intrusive and privacy-friendly, as it does not capture any identifiable visual information such as facial features, skin color, or clothing.

- Robustness to lighting conditions: Camera-based systems are sensitive to lighting conditions, and changes in lighting can significantly affect the accuracy of emotion recognition. However, millimeter-wave radar operates independently of lighting conditions because it uses radio waves that are not affected by changes in ambient light, making it suitable for both indoor and outdoor environments.

- Insensitive to occlusions: Cameras can be obstructed by various objects, such as hands, accessories, or other people, resulting in inaccurate emotion recognition results. In contrast, millimeter wave radar is able to penetrate certain materials, including fabric, plastic, and thin barriers, making it less sensitive to occlusions and providing more reliable emotion recognition results.

- focused on internal physiology: although it may be easy for a subject to control the facial and behavioral expression of emotions, the physiology of the body can hardly be consciously controlled and the present invention can therefore hardly be deceived.

According to the invention, the system may comprise a relay box arranged near the transmitting and receiving antennas; the processing unit being remote and connected to the relay box via an internet-type communication network; the processing unit communicating with the transmitting and receiving antennas via the relay box.

Proximity means a distance allowing wired or wireless communication with the antennas with a signal quality allowing the system to function properly.

The processing unit can also be embedded locally and connected directly to the transmitting and receiving antennas.

The fields of application of the invention may include:

- industry: the present invention can be installed on workstations in combination with the company's system and a dedicated rules engine to support operators in their daily work and improve their well-being. The objective may be to promote interactions, thus reducing stress at work. This results in less absenteeism, fewer accidents, increased productivity and reduced indirect costs.

- design: integrated into the development phase of a product or environment, the present invention allows each iteration to consider the human factor in order to evaluate areas of friction and optimize the interaction between users and the product.

- training: integrated into the training tool, whether the training is face-to-face, virtual or remote, the present invention allows the trainer to better understand the internal emotional state of the participants in order to personalize the format and content in the transfer of knowledge.

- social robotics: social robots must be able to understand the emotions of their interlocutors in order to adapt their behavior and thus facilitate interactions between humans and machines while promoting their acceptance.

- healthcare: the present invention can help medical and social workers to announce a difficult diagnosis by providing them with additional information about the internal feelings of patients. It can also capture the inner feelings of a patient or a young child who is unable to verbalize them.

- animal health: the present invention can help in stress management in domestic animals and in farm animals intended for slaughter.

- entertainment: coupled with a rules engine, the present invention can take into account the emotions of a player to offer a more personalized “gameplay” or video game scenario.

- the smart home: coupled with a decision rules engine, the present invention can be installed in living and working spaces to modify the aesthetic and sensory environment based on the feelings of its occupants.

Other characteristics and advantages of the invention will appear on reading the detailed description of implementations and embodiments which are in no way limiting, with regard to the appended figures in which:

FIG. 1 - there FIG. 1 is a schematic view illustrating the overall flow for the generation according to the invention of cardiac characteristics as training data for an artificial intelligence model;

FIG. 2 - there FIG. 2 is a schematic view illustrating the use according to the invention of large and small time windows when processing radiofrequency signals coming from the user;

FIG. 3 - there FIG. 3 is a schematic view illustrating the overall process for predicting an emotion from an emotion model according to the invention;

FIG. 4 - there FIG. 4 is a schematic view illustrating the generation of training data (in the training phase) and useful data (in the exploitation phase) from data representative only of cardiac activity according to the invention;

FIG. 5 - there FIG. 5 is a schematic view illustrating the generation of training data (in the training phase) and useful data (in the exploitation phase) from data representative only of the respiratory activity according to the invention;

FIG. 6 - there FIG. 6 is a schematic view illustrating the generation of training data (in the training phase) and useful data (in the exploitation phase) from data representative of cardiac activity and data representative of respiratory activity according to the invention;

FIG. 7 - there FIG. 7 is a flowchart illustrating steps of digital processing of radar signals to obtain data representative of respiratory activity and data representative of cardiac activity according to the invention; and

FIG. 8 - there FIG. 8 is a general view illustrating the working environment of a user, an environment in which components are arranged for implementing the method according to the invention.

The embodiments which will be described below are in no way limiting; it will be possible in particular to implement variants of the invention comprising only a selection of characteristics described below isolated from the other characteristics described, if this selection of characteristics is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art. This selection comprises at least one preferably functional characteristic without structural details, or with only a part of the structural details if this part alone is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art.

Generally speaking, the design of an artificial intelligence model involves a training phase during which the artificial intelligence model is fed with a large amount of specific training data. The goal is to explore as many situations as possible so that the machine develops its predictive capacity.

There FIG. 1 concerns the training phase. We can see a user 1 equipped with a virtual reality headset 2. This headset 2 is connected to a processing unit not shown in the FIG. 1 .

A training signal 3 is played in the headset 2. Preferably, the training signal 3 is a 3D environment comprising videos and audio in which the user 1 is immersed, and depending on the situations, with which he can possibly interact.

Stimuli are integrated into this environment and are intended to create a predetermined emotion in the user. These can be different stimuli placed at different times during the playback of the training signal. For example, a playback duration T0 of the training signal can be planned equal to 3 minutes. Other durations can be considered close to a minute or several minutes, up to 120 minutes for example.

We can consider a stimulus capable of provoking the same emotion and to which we apply variations throughout the reading. This is a multimodal stimulus which varies, for example, according to complexity.

When reading the training signal, the user may be asked to report whenever they feel an emotion. This reporting can be done by any means, including manipulating an input to the processing unit. This input may be a mouse, a keyboard, or any other interface.

During the reading of the training signal, a first radio frequency signal 4 is sent towards the user 1. The reflection of this first radio frequency signal, in particular on the user 1, produces a second radio frequency signal 5 which is captured for processing. On the FIG. 1 , a transmitting antenna Tx transmits the first radio frequency signal 4 and a receiving antenna Rx picks up the second radio frequency signal 5.

Preferably, the transmitting antenna Tx and the receiving antenna Rx are part of a frequency modulated continuous wave radar FMCW operating in a range between 57GHz and 64GHz or 76GHz and 81GHz with a maximum bandwidth of 4 GHz. The FMCW radar unit is based on the coherent MIMO (Multiple Input Multiple Output) system with several Tx antennas and several Rx antennas. Other technologies can be used, such as ultra-wideband. The coherent MIMO beamforming is based on a synthesizer that generates chirps (frequency and amplitude modulated pseudo-periodic signal) following the binary phase modulation method, which allows the simultaneous transmission of each antenna for a better signal-to-noise ratio (SNR). The transmitted chirps are dynamically scheduled and the frames are divided into several subframes. Two simultaneous subframes are configured independently via a serial peripheral interface (SPI) by controlling the burst period, subframe period, loop number, and duty cycle. This configuration flexibility allows for the detection of micro-motions within the radar coverage area. The configuration and generation of these subframe profiles are performed by a chirp module and a buffer within the processing unit.

The second radio frequency signal received contains physiological data of the user 1. The processing 6 on this radio frequency signal 5 aims to determine a signal 6a representative of the respiratory activity and a signal 6b representative of the cardiac activity. A window F1 is applied to the signal 6a representative of the respiratory activity so as to generate data DF1 useful as training data for the artificial intelligence model 8. A window F3 is applied to the signal 6b representative of the cardiac activity so as to generate data DF3 useful as training data for the artificial intelligence model 8. The data DF1 and DF3 contain cardiac and respiratory characteristics of the user 1 that the training of the artificial intelligence model 8 will make it possible to highlight.

Many measurements are made with many users for a single or different predetermined known emotions. On the FIG. 2 we see a time line from 0 to T0 corresponding to the duration of the training signal 3 of the FIG. 1 This training signal can be a video signal that lasts 3 minutes. Markers 9 and 10 are moments at which the user, immersed in the virtual environment, reports having felt an emotion. The user is encouraged to express themselves whenever they wish, especially when they experience a strong emotional change.

According to the invention, a large window F1 is applied in a sliding manner, for example of duration equal to 1 minute, in steps of 3 seconds, starting at the beginning of the video, i.e. 41 extracts per video. On the FIG. 1 , the DF1 signal represents the data corresponding to the F1 windows.

We also apply a short window F2, which is a period of interest of 10 seconds before each response 9 and 10 of the user. Within this period of interest, we apply short windows F3 for example of duration equal to 3 seconds, in steps of 500 ms, starting at the beginning of the window F2, i.e. 15 extracts per window F2, on which the training will make it possible to detect cardiac characteristics.

Once the artificial intelligence model is trained and configured, it constitutes the emotion model that can be used in operation.

On the FIG. 3 we distinguish a user 11 in activity permanently receiving the first radio frequency signal 4. The second radio frequency signal 5 coming from the user is captured to undergo processing 6 identical to the processing during training. The difference between the training phase and the exploitation phase is in particular the use of time windows.

The F1 time windows are applied in a sliding and continuous manner throughout a user observation period. Preferably, we are no longer limited to the duration T0 of the training signal.

F2 time windows are not used in the operating phase. In fact, F3 time windows are applied in a sliding and continuous manner.

Thus, in operation, the F1 time windows and the F3 time windows are applied continuously throughout a user observation period.

The observation period does not necessarily run continuously throughout the day. This observation period can be triggered based on information captured in the environment. For example, only activate processing and/or the application of time windows during difficult stages of a task being performed by the user or only when an external alert occurs, for example, an emergency situation.

DF1 and DF3 data on the FIG. 3 feed an emotion model 12 which is therefore capable of identifying cardiac and respiratory characteristics to enable real-time prediction of an emotion 13. The model 12 can advantageously be derived from the training of the training model 8 of the FIG. 1 .

There FIG. 4 is a schematic view illustrating the generation of training data (in the training phase) and useful data (in the exploitation phase) from data representative only of cardiac activity according to the invention.

Indeed, the invention can be implemented only from data representative of cardiac activity.

At the end of the digital processing 6, the respiratory activity 6a can be determined and not used or it can simply not be determined at all.

The 6b cardiac activity is split into two branches to follow parallel processing. On a first branch, the F3 window is applied, with the limits of the F2 windows if we are in the training phase and without limits of the F2 windows when we are in the exploitation phase. The DF3 data resulting from the application of each F3 window feeds the training model or the emotion model.

On the second branch, we apply the F1 window, with the limit of the duration T0 if we are in the training phase and without limit of the duration T0 when we are in the exploitation phase. The DF1a data directly resulting from the application of the F1 window can directly feed the training model 8 or the emotion model 12.

The invention also provides an embodiment where the DF1a data directly originating from the application of the F1 window are first transformed in step 14 into one or more DF1b values before feeding the training model 8 or the emotion model 12.

The invention further provides an embodiment where the DF1a data directly originating from the application of the F1 window and the DF1b data originating from the transformation of the DF1a data in step 14 together feed the training model 8 (in the training phase) or the emotion model 12 (in the exploitation phase).

In step 14, the transformation performed is the calculation of a value of the variability of the heart rate HRV.

There FIG. 5 is a schematic view illustrating the generation of training data (in the training phase) and useful data (in the exploitation phase) from data representative only of cardiac activity according to the invention. The same operations are carried out as in the FIG. 4 except here the initial data 6a concerns only respiratory activity. Another difference is the transformation step 14 during which at least one of the following values is calculated:

- respiratory variability (RRV for “Respiratory Rate Variability” in English),

- a breathing pattern analysis (BPA),

- respiratory amplitude and variability (RAV for “Respiratory Amplitude and Variability” in English).

There FIG. 6 is a schematic view illustrating the generation of training data (in the training phase) and useful data (in the exploitation phase) from data representative of cardiac activity and data representative of respiratory activity according to the invention.

We carry out the same operations as on the FIG. 4 except here the starting data are respiratory activity data 6a and cardiac activity data 6b. On one branch we apply window F1 to all data 6a and 6b, and on the other branch we apply window F3 to all data 6a and 6b.

Thus, at transformation step 14, all the values described with reference to figures 4 and 5 can be calculated, but a value for respiratory sinus arrhythmia (RSA) can also be calculated.

On the FIG. 7 an example of a flowchart is shown illustrating steps of a digital processing of radar signals coming from the user to obtain data representative of respiratory activity and data representative of cardiac activity according to the invention. This is processing 6 of the FIG. 1 .

Processing starts at step 15.

In step 16, FMCW signals are received from the receiving antennas before conversion to intermediate frequency in step 17.

An analog-to-digital conversion is performed in step 18. Digital signal processing is performed using one or two real-time digital signal processors with fixed or floating arithmetic that handle several instructions in parallel: interference reduction using DC component suppression (leakage between Tx and Rx), detection of an increase in the noise threshold, reduction of quantization noise due to analog-to-digital conversion, detection at a constant false alarm rate (CFAR), windowing for an FFT and a transform of the signals under a logarithmic scale.

In step 19, infinite impulse response filtering reduces electronic noise and extracts specific signal components. Quadrature conversion is based on decomposition into two orthogonal components: an in-phase component (I) and a quadrature component (Q). This allows both the amplitude and phase of the signal to be analyzed.

In step 20, a data cube construction of the FMCW signals is carried out through three-dimensional matrices defined by the characteristics of the MIMO radar (number of channels linked to the number of antennas) and the FMCW modulation (time and chirp).

In step 21, a Range-Doppler-Angle matrix is developed from the data cubes. A succession of 1-, 2- and 3-dimensional Fourier transforms is applied to estimate the 3 pieces of information: distance, speed and the angle of the target relative to the radar.

In step 22, a clutter suppression algorithm is implemented. Through a (sliding in time) window of emission and reception applied to the sought signatures, static states as well as slow movements, therefore different from the sought modulations: rib cage, cardiac and respiratory activity, are excluded. The interference between harmonics and the fundamental frequency of the heart and respiration is improved by the implementation in step 23 of an empirical decomposition algorithm into ensemble modes and the decomposition in step 24 of the signal into an Intrinsic Mode Function (IMF).

In step 25, a continuous wavelet transform (CWT) is performed to improve the identification of the heart and respiratory rhythm. This is an adaptive filtering that makes it possible to generate data 6a representative of respiratory activity and data 6b representative of cardiac activity.

The data 6a representative of respiratory activity undergo normalization in step 26. The data 6b representative of cardiac activity also undergo normalization in step 27. Normalizations 26 and 27 are optional and can be the same or different.

On the FIG. 8 , we can see the user 11 sitting in front of a workstation 28 in a workshop. Transmitting and receiving antennas can be positioned in different positions 29 of the workshop around the user 11. A processing unit 30 can be provided for implementing the method according to the invention. This processing unit can also be directly embedded in a module with the antennas. This processing unit includes the software and hardware means capable in particular of managing the antennas and the signals passing through its antennas. The processing unit is connected by wire or not to the antennas. It is also possible to envisage positioning the processing unit remotely and providing a relay box placed in the workshop to relay the signals between the processing unit and the antennas. Such a processing unit can be high-power to manage several sets of antennas for several users.

The present invention therefore relates to an emotion recognition method combining radio frequency radar and artificial intelligence models to derive emotion from the contactless measurement of at least one key physiological signal of at least one person.

The invention makes it possible to extract complete cardiac and respiratory signals according to the chosen short and long time windows, but also classic characteristics of cardiac and respiratory activities such as HRV, RRV, BPA, RAV, RSA and other characteristics within the same heartbeat or in relation to the relationship between different cardiac characteristics of successive heartbeats.

The various embodiments of the present invention comprise various steps. These steps can be implemented by instructions of a machine executable by means of a microprocessor for example.

Alternatively, these steps can be performed by specific integrated circuits including hard-wired logic to perform the steps, or by any combination of programmable components and custom components.

The present invention may also be provided in the form of a computer program product which may comprise a non-transitory computer storage medium containing instructions executable on a computing machine, which instructions may be used to program a computer (or other electronic device) to perform the method.

Of course, the invention is not limited to the examples which have just been described and numerous adjustments can be made to these examples without departing from the scope of the invention.

Claims (15)

Method for detecting an emotion in a subject (11) comprising the following steps:
- emission of a first radio frequency signal (4) towards the subject,
- reception of a second radio frequency signal (5) containing a physiological signal due to the reflection of said first radio frequency signal on the subject,
- digital processing (6) of the second radio frequency signal (5) so as to generate respiratory data (6a) and/or cardiac data (6b),
- application of several F1 time windows of T1 duration in a sliding manner on the respiratory data and/or the cardiac data,
- application of several time windows F3 of duration T3 in a sliding manner on the respiratory data and/or the cardiac data; the duration T3 being less than the duration T1, and
- prediction of an emotion by an emotion model (12) based on artificial intelligence from data from the F1 windows and data from the F3 windows. Method according to claim 1, characterized in that the data from the time window F1 are directly transmitted to the emotion model (12) and/or undergo a calculation (14) of a value of at least one respiratory and/or cardiac biomarker, this value being transmitted to the emotion model (12). Method according to claim 2, characterized in that the calculation (14) of a value of at least one respiratory and/or cardiac biomarker comprises at least one of the following calculations:
- calculation of heart rate variability (HRV),
- calculation of respiratory variability (RRV for “Respiratory Rate Variability”) or of a respiratory pattern (BPA for “Breathing Pattern Analysis”) or of a respiratory amplitude and its variability (RAV for “Respiratory Amplitude and Variability”),
- calculation of respiratory sinus arrhythmia (RSA for “Respiratory Sinus Arrythmia” in English). Method according to any one of the preceding claims, characterized in that the duration T1 is at least 30 seconds with a sliding step of at least 1 millisecond; or a duration T1 of between 30 seconds and 5 minutes with a sliding step of between 1 millisecond and 15 seconds; or a duration T1 of between 30 seconds and 1 minute 15 seconds with a sliding step of between 3 seconds and 5 seconds. Method according to any one of the preceding claims, characterized in that the duration T3 is at least 375 milliseconds, or between 375 milliseconds and 30 seconds, or between 1 and 5 seconds; with a step of at least 1 millisecond. Method according to any one of the preceding claims, characterized in that the first radio frequency signal (4) is a frequency-modulated continuous wave radar signal (FMCW) or an ultra-wideband signal (UWB). Method according to any one of the preceding claims, characterized in that the respiratory data and/or the cardiac data undergo a normalization step (26, 27) before application of the time windows. Method according to any one of the preceding claims, characterized in that the emotion model (12) consists of one or more simple or deep learning models. Method according to any one of the preceding claims, characterized in that the emotion model (12) comprises a hybrid recurrent neural network. Method according to claim 9, characterized in that the emotion model (12) comprises a fusion between a closed recurrent unit GRU (for “Gated recurrent unit” in English) and a long-term and short-term memory network LSTM (for “Long Short-term Memory” in English). Method according to any one of the preceding claims, characterized in that the emotion model (12) comprises:
- a slow dynamics model dedicated to data from F1 time windows,
- a fast dynamics model dedicated to F3 time window data, and
- a decision-making algorithm common to the slow dynamics model and the fast dynamics model. Method according to any one of the preceding claims, characterized in that the transmission of the first radio frequency signal (4) is carried out by means of several transmission antennas (29) and the reception of the second radio frequency signal is carried out by means of several reception antennas (29) according to MIMO technology (for “Multiple-Input Multiple-Output” in English). System for detecting an emotion in a subject (11) comprising:
- a processing unit (30) provided with an emotion model (12) and configured to implement a method according to any one of the preceding claims,
- at least one transmitting antenna (29) connected to the processing unit and arranged to transmit the first radio frequency signal towards the subject (11),
- at least one receiving antenna (29) connected to the processing unit and arranged to receive the second radio frequency signal containing the physiological signal due to the reflection of said first radio frequency signal on the subject. System according to claim 13, characterized in that the processing unit (30) is locally embedded and connected directly with the transmission and reception antennas (29). System according to claim 13, characterized in that it comprises a relay box arranged near the transmission and reception antennas (29); the processing unit (30) being remote and connected to the relay box via an internet-type communication network; the processing unit (30) communicating with the transmission and reception antennas via the relay box.