RU2833770C1 - Method for determining effect of information factors of virtual environment on emotional state of person according to cardiorhythmogram data - Google Patents

Abstract

FIELD: medicine.

SUBSTANCE: invention refers to medicine, namely to psychophysiology, and can be used to determine stress potential of virtual environment information factors. Disclosed is a method which employs a neural network algorithm for classifying cardiorhythmogram data. Data base of cardiorhythmograms, on which the algorithm is trained, includes cardiorhythmograms associated with positive, negative emotions when exposed to information factors of the virtual environment, as well as those associated with the absence of emotional responses to information factors of the virtual environment. All cardiac rhythmograms are obtained by event-related cardiac telemetry. Analysis of the effect of information factors of the virtual environment on the emotional state of a person according to the cardiorhythmogram includes: telemetric measurements of cardiac rhythmograms using a software and hardware complex consisting of virtual reality glasses, a miniature wireless sensor and a tablet with specialized software, analysis of the obtained data using a pre-trained neural network algorithm.

EFFECT: invention provides automatic recording of the emotional state of a person by a cardiorhythmogram under the influence of information factors of the virtual environment.

2 cl, 5 dwg

Description

The invention relates to the field of medicine, namely to the field of psychophysiology, and can be used to determine the stress-producing nature of information factors in a virtual environment.

A group of inventions is known from the prior art (RU 2774274 "Device, system and method for determining the emotional state of a user", IPC A61B 5/16, A61B 5/053, published on June 30, 2021), which relates to medicine, namely to a device, system and method for determining the emotional state of a user based on an assessment of the cortisol level caused by the emotion. The device comprises an interface and a processing unit. The interface is designed to obtain a psychophysiological signal curve. The curve indicates the measured response to stimuli corresponding to the nervous stress response. The processing unit is designed to process the psychophysiological signal curve. The system comprises a sensor for measuring the psychophysiological signal and a device for determining the emotional state of the user. The sensor indicates the measured response to stimuli corresponding to the nervous stress response of the user. When executing the method, a psychophysiological signal curve is obtained indicating the measured response to stimuli corresponding to the nervous stress response. A disadvantage of the known group of inventions is the ability to record only the emotional state of a person corresponding to the nervous stress response.

There is a known "Method for remote recognition and correction of a person's psycho-emotional state using virtual reality" (RU 2711976, IPC A61B 5/16, published on 23.01.2020), in which individual dynamic monitoring of facial expressions, movements and speech is used to recognize the psycho-emotional state, while in order to determine facial expressions, motor and speech activity, a person is presented with test material to influence his receptor field. Based on the data obtained, a person's psycho-emotional state is determined in accordance with the facial movement coding system, the resulting facial image is classified by the artificial intelligence system as an affective component of emotional disorders - by the severity and duration of emotional disorders, defined as simple emotion, mood, affect; by the nature of emotional disorders - depressed mood, elevated mood, unstable emotional sphere, qualitative distortion of emotions; by the motor component of affective disorders - increased and decreased motor activity; and the ideational component of affective disorders determined by fast and slow speech, and is stored as a database of the spectrum of emotions of an individual person. After that, remote monitoring is carried out for a specified time using artificial intelligence for constant comparison of data on the contraction of facial muscles, motor and speech activity with the formed database of the spectrum of human emotions, based on the results obtained, recommendations are issued on the possibility of referring for correction of the psychophysiological state. Correction is carried out by means of virtual reality, introducing a person into a trance state, the form of correction is determined as directive or non-directive according to an individual program based on the personality type and the predominant modality of perception of information - visual, auditory and kinesthetic, with constant monitoring by encephalography, transmitting data to artificial intelligence to determine the effectiveness of the correction. The invention provides the possibility of remote recognition of a negative psychoemotional state of a person and its timely correction using the influence of virtual reality. The disadvantage of the described analogue is the determination of the psycho-emotional state of a person in accordance with the coding system of facial movements without the use of physiological data, for example, ECG data under the influence of information factors of the virtual environment.

A method, an electronic device and a system for identifying emotions are known (CN 107992199 "Electronic equipment and emotion identification method and system for the same", IPC G06F 3/01, published 04.05.2018). The method allows using the front camera of the electronic equipment, microphone equipment, heart rate data and galvanic skin response to obtain information about the user's emotions. A disadvantage of the method is the failure to use cardiorhythmogram data for analyzing emotions, including under the influence of information factors of the virtual environment.

A method for recognizing emotions based on facial features and voice is known (CN 107705808 "Emotion recognition method based on facial features and voice features", IPC G06K 9/00, G10L 25/63, published on 25.12.2020), implemented using a camera, microphone and emotion processing unit. The disadvantage of the method is that it is based solely on the analysis of the user's facial movements and voice and does not use heart rate recording.

A method and device for determining emotions on a user's face are known (KR 101939772 "Method and apparatus for inferring facial emotion recognition system for inferring facial emotion and media for recording computer program", IPC G06K 9/46, G06T 7/00, published on 17.01.2019). The system includes a database of images of many people classified by emotional and behavioral features. The above features are determined on the basis of the facial action coding system (FACS), and in the facial action coding system, behavioral units correspond to a combination of muscles associated with changes in facial appearance or the main movement of each muscle. The disadvantage is the use (analysis) of only facial expression and the work (movement) of certain facial muscles, as well as the lack of use of ECG data, including under the influence of information factors of the virtual environment. There is also no possibility of continuous monitoring and storage of the obtained data.

A method for recognizing facial expressions is known (US 10417483 "Facial expression recognition" IPC G06N 3/08, G06N 3/12, G06V 30/194, published on September 17, 2019). The invention is an automatic geometric method for analyzing and recognizing human facial expressions based on extracted features using a neural network. Facial expressions are divided into seven different emotion categories: surprise, happiness, disgust, neutrality, fear, sadness, and anger. The method uses an algorithm that hybridizes genetic algorithms and iterative local search with a backpropagation algorithm used to train the neural network. The disadvantage of the method is the use of only facial expressions to record basic emotions, as well as the lack of use of ECG data, including when influenced by information factors of the virtual environment.

The closest to the claimed invention is the "Method for continuous monitoring of the level of human stress" (RU 2649519, IPC A61B 5/024, A61B 5/11, published on 22.09.2017). The method includes the use of a biometric detector in the form of a wristwatch or bracelet, the data from which is used to buffer the values of intervals between adjacent heartbeats during a given time window, as well as to create a histogram of the distribution of these intervals and calculate the stress level. The biometric detector data is transmitted by wire or wirelessly to a mobile application and accumulated over a certain time. Then, based on the distribution of the received data over time, conclusions are made about the daily physical activity and lifestyle of the person. The calculated stress level is associated with the physical activity of the person and his interaction with the surrounding people and objects according to the data that is collected automatically by collecting information from applications installed on the user's electronic device. Moreover, based on the collected information, they draw a conclusion about what exactly served as the source of the change in stress level, and offer the user to exclude the specified source. The invention allows to increase the reliability and safety of work, improve the methods of psycho-emotional self-control of a person, improve the quality and information content of communication between people in social networks and direct communication. The disadvantage of this method is that it measures only the user's stress level, and not his emotional state.

The task that the proposed invention is aimed at solving is the development of a method for determining the influence of information factors of a virtual environment on the emotional state of a person based on cardiogram data.

The technical result is achieved due to the fact that the developed method for determining the emotional state of a person, as well as the prototype method, includes measuring the values of intervals between adjacent heartbeats, transmitting and accumulating these data in a processing and storage device, identifying a source that influences the emotional state of a person. What is new in the developed method is that a person is affected by information factors of a virtual environment, for which a hardware and software complex is used, consisting of virtual reality glasses, a sensor with which R-R intervals of the heart are measured, and a computer with specialized software. The obtained data are analyzed using a pre-trained neural network algorithm, which is trained in advance using cardiorhythmogram data obtained from users who were presented with virtual reality scenes with emotional state control using the SAM method. In this case, a conclusion is made about the emotional state of a person after the presented information factors of the virtual environment, classifying it as corresponding to negative emotions, neutral emotions or positive emotions.

In the particular case of implementing the proposed method, the Stress Monitor software is used as specialized software.

The developed method is explained by the following figures.

Fig. 1 shows the diagram of the training stage of the neural network algorithm.

Fig. 2 shows an example of recorded R-R interval data used for further analysis.

Fig. 3 shows an example of a questionnaire on cardioactive lifestyle factors.

Fig. 4 shows an example of a questionnaire for determining the individual affective profile (IAP).

Fig. 5 shows a diagram of the developed method.

The developed method is implemented as follows.

First, the neural network algorithm is trained, and cardiograms are obtained from subjects who are shown virtual reality scenes with emotional state control using the SAM method, which describes three features: 1 - valence, 2 - activity, 3 - level of control. SAM (Self-Assessment Manikin) is a non-verbal graphic questionnaire that directly measures a person's affect and feelings in response to the impact of an object or event, such as an image. There is a database of images (IAPS - International Affective Picture System), containing a standardized set of images for studying emotions and attention, which is widely used in psychological research.

The scheme of the training stage of the neural network algorithm is shown in Fig. 1. The cardiorhythmogram is measured by a wireless sensor, and R-R intervals (1) are identified using software. Fig. 2 shows an example of recorded R-R interval data used for further analysis. Batch data transmission from the sensor to the mobile device is performed, for example, via the Bluetooth wireless protocol. Communication, transmission and storage of data are implemented on the mobile device, in a particular case (according to paragraph 2 of the formula), using the original Stress Monitor software.

In parallel with the recording of R-R intervals, the subject completes (2 min) a questionnaire on cardioactive lifestyle factors. An example of the questionnaire is shown in Fig. 3. The user also completes a questionnaire to determine the individual affective profile (IAP): it is proposed to indicate the valence of emotions associated with affective situations (2 min) (2), where valence is the degree of attraction or aversion that a person experiences to a certain object or event. An example of the questionnaire is shown in Fig. 4. Background recording of R-R intervals is performed while standing (3 min).

Next, the emotional image is registered for five pictures from the IAPS affective stimulus base using the SAM method (5 min.) to teach a formalized description of emotions based on three characteristics: 1 - valence, 2 - activity, 3 - control level.

SAM scales

1 - valence characterizes what emotions there were: from negative to positive, centrally neutral;

2 - degree of activation: from sleeping, strongly relaxed to activated, excited, central - neutral;

3 - were you able to maintain control: from failed to succeed, central neutral.

Next, the user is presented with virtual reality scenes through glasses (3). After each scene, while continuing to record R-R intervals, the user takes a survey using the SAM method. Then background recording is performed (2 min). In this way, a database of R-R intervals with labels of a person's emotional state is accumulated (4). The neural network algorithm is trained to classify the CRG data using the labeled database (5).

The database of cardiorhythmograms, on which the algorithm is trained, includes cardiorhythmograms associated with positive and negative emotions under the influence of information factors of the virtual environment, as well as those associated with the absence of emotional reactions to information factors of the virtual environment. All cardiorhythmograms were obtained by the method of event-related cardiac telemetry.

Using a neural network algorithm pre-trained according to the above-described scheme, telemetric measurements of cardiac rhythmograms are carried out using a hardware and software complex consisting of virtual reality glasses, a miniature wireless sensor and a tablet with specialized software. Then the obtained data is analyzed.

Automated determination of a person’s emotional state based on cardiorhythmogram data when presented with virtual reality scenes (Fig. 5) is carried out as follows: the R-R intervals of the subject are recorded (6), an information stimulus (scene) is simultaneously fed into the virtual reality glasses (7), and the obtained data are analyzed by a pre-trained neural network algorithm (8) to determine the emotional state of the person (9).

Example of implementation of the method.

Telemetric measurements of cardiac rhythmograms were performed using a developed hardware and software system consisting of a miniature wireless ECG sensor (NxM; Zephyr Technology, USA), a tablet based on the Android operating system with specialized Stress Monitor software, and virtual reality glasses. Batch data transmission from the sensor to the mobile device is performed via the Bluetooth wireless protocol. Communication, transmission, and storage of data are implemented on the mobile device using the original Stress Monitor software.

The subject was presented with virtual reality scenes and surveys were conducted using the SAM method in the following sequence:

1. VR story Fedorovsky embankment (2 min);

2. SAM;

3. VR story Hofburg, Vienna (2 min);

4. SAM;

5. VR plot in accordance with the indicated phobia (Fig. 4) in the questionnaire (2 min);

6. SAM;

7. VR story Roller coaster (4 min);

8. SAM.

The obtained CRG data were analyzed by a previously pre-trained neural network algorithm, classifying the CRG data into three groups: positive emotions, negative emotions, and those associated with the absence of emotional reactions to information factors of the virtual environment.

Thus, a method for determining the influence of virtual environment information factors on a person's emotional state based on cardiorhythmogram data has been developed. A neural network algorithm is used, the training of which occurs based on cardiorhythmogram data associated with positive, negative emotions when exposed to virtual environment information factors, as well as those associated with the absence of emotional reactions to virtual environment information factors.

The information provided in the description are examples that do not limit the scope of the present technical solution defined by the formula. It becomes clear to a specialist in this field that there may be other embodiments of the present technical solution that are consistent with the essence and scope of the present technical solution.

Claims (2)

1. A method for determining a person’s emotional state, including measuring the values of intervals between adjacent heartbeats, transmitting and accumulating these data in a processing and storage device, identifying a source that influences a person’s emotional state, characterized in that influence a person with information factors of a virtual environment, for which they use a hardware and software complex consisting of virtual reality glasses, a sensor with which they measure the R-R intervals of the heart, and a computer with specialized software, analyze the obtained data using a pre-trained neural network algorithm, which is trained in advance using cardiorhythmogram data obtained from users who were presented with virtual reality scenes with emotional state control using the SAM method, while drawing a conclusion about the emotional state of a person after the presented information factors of the virtual environment, classifying it as corresponding to negative emotions, neutral emotions or positive emotions. 2. A method for determining a person’s emotional state according to paragraph 1, characterized in that the specialized software used is the “Stress Monitor” software.