WO2025121869A1 - Method, device, and system for providing content including sleep state information generated on basis of information acquired from sleep environment of user - Google Patents

Abstract

The present invention relates to a method for providing interpretation content of sleep data of a user, the method comprising the steps of: acquiring sleep information of a user; generating sleep state information of the user on the basis of the acquired sleep information of the user; generating sleep data interpretation content of the user on the basis of the generated sleep state information; and outputting the generated sleep data interpretation content of the user, wherein the sleep data interpretation content of the user is expressed in at least one of a numerical manner and a non-numerical manner.

Description

Method, device and system for providing content including sleep state information generated based on information obtained from a user's sleep environment

The present invention relates to a method, device and system for providing content including sleep state information generated based on information acquired from a user's sleep environment.

There are many ways to maintain and improve your health, such as exercise and diet, but it is most important to manage your sleep, which takes up about 30% of your day.

However, despite the simple replacement of labor by machines and the leisureliness of life, modern people are unable to get enough sleep due to irregular eating and living habits and stress, and suffer from sleep disorders such as insomnia, hypersomnia, sleep apnea syndrome, nightmares, night terrors, and sleepwalking.

According to the National Health Insurance Corporation, the number of patients with sleep disorders in Korea increased by an average of approximately 8% per year from 2014 to 2018, and the number of patients treated for sleep disorders in Korea in 2018 reached approximately 570,000.

As sleep is recognized as an important factor affecting physical and mental health, interest in sleep is increasing. However, to improve sleep disorders, one must visit a specialized medical institution in person, separate examination costs are required, and ongoing management is difficult, so users' efforts for treatment are insufficient.

Korean Patent Publication No. 10-2023-0012133 discloses an electronic device that provides a user interface according to a sleep state and a method of operating the same. It is disclosed that an abnormal REM sleep event can be detected and a user interface based on designated action information can be provided in response thereto.

However, since the conventional technology detects abnormal REM sleep stages by examining the entire sleep stage, there is a concern that the accuracy of the measurement of the sleep stage may be low, and since the basic unit of measurement is composed of several minutes, there is a concern that sleep-related events may not be detected in real time.

Meanwhile, FIGS. 20a to 20e are diagrams showing hypnogram graphs of sleep stage information expressed in a conventional sleep measurement interface.

However, since the conventional technology expresses the graph representing the sleep stages by expressing the shapes assigned to the first and second sleep stages continuously rather than discretely, there is a concern that this may lead to the misunderstanding that another sleep stage must be passed through in the process of entering the second sleep stage from the first sleep stage.

In addition, as shown in FIG. 20c or FIG. 20d, even if a shape representing a wake-up stage was displayed as if it were separated from shapes representing other sleep stages, there was a problem in that it was difficult to clearly understand which sleep stage occurred at a given time because shapes corresponding to sleep stages were displayed continuously in areas allocated to other sleep stages.

Accordingly, research is currently being conducted to intuitively represent graphs of sleep state information including sleep stages.

On the other hand, in order to accurately measure sleep, polysomnography or home polysomnography should be used. However, since human sleep varies greatly from day to day, it is difficult to continuously monitor daily sleep with existing sleep measurement methods. In particular, since sleep time and sleep stages can vary due to various reasons (e.g., lifestyle habits, etc.), there was a problem that sleep analysis for multiple days (or multiple sleep sessions) was difficult.

Additionally, there has been a demand to create or provide a graphical user interface that allows one to see sleep at a glance, even when analyzing sleep over multiple days (or multiple sleep sessions).

The present invention has been made in consideration of the problems of the above-described prior art, and an object of the present invention is a method for providing a graphical user interface that shows the user's sleep state information through information about the user's sleep that can be obtained through a sleep sensor. In addition, an object of the present invention is to provide useful information about sleep to the user by increasing the accuracy of sleep analysis using sleep sound information.

The present invention has been made in consideration of the problems of the above-described prior art, and an object of the present invention is to provide a method for analyzing a user's sleep based on at least one of sensing information obtained through a sensor device and the user's sleep information, and providing a graphical user interface including information analyzed from the user's sleep. In addition, an object of the present invention is to provide useful information on sleep to the user by increasing the accuracy of sleep analysis.

Another object of the present invention is to provide a method for providing a graphical user interface that represents sleep state information generated based on sleep information acquired during one or more sleep sessions.

The present invention provides an automated sleep measurement start trigger provision. Specifically, a system is provided that automatically starts sleep measurement without a separate operation by the user, and a function can be provided that automatically starts sleep measurement based on a broadcast trigger for sleep measurement (e.g., start of charging, display OFF, etc.) and a sleep analysis trigger (e.g., movement reduction, touch input cessation, etc.).

The present invention can initiate sleep measurement by detecting not only alarms based on time reservations, but also user-initiated events (e.g., swiping) and involuntary events (e.g., decreased movement, cessation of touch).

The present invention automatically records the start and end times of sleep measurement, thereby enabling the user to collect accurate sleep data without separate operation.

The present invention solves the problem of delayed alarm or trigger execution due to the battery optimization function of the Android system, and enables execution of a sleep measurement trigger even in a low power mode.

The present invention has been made in consideration of the problems of the above-described prior art, and an object of the present invention is to analyze a user's sleep state based on information acquired from a sleep environment through a sleep sensor, and to generate and provide content related to the user's sleep state information. In addition, an object of the present invention is to generate and provide content related to the user's sleep state information by utilizing generative artificial intelligence. In addition, an object of the present invention is to provide useful sleep-related content to the user by increasing the accuracy of sleep analysis by utilizing sleep information, and to improve the quality of sleep.

A method for providing a graphical user interface that displays information about a user's sleep according to one embodiment of the present invention may include the steps of: acquiring the user's sleep information; generating the user's sleep state information based on the acquired user's sleep information; generating a sleep state information graph that displays sleep state information over time based on the generated sleep state information; generating a graphical user interface including the sleep state information graph; and outputting a graphical user interface including the generated sleep state information graph.

The above sleep state information may include at least one of sleep stage information, sleep stage probability information, sleep event information, and sleep event probability information.

The step of generating the above sleep state information graph may further include the step of generating at least one of the stacked sleep stage graphs "G stacked sleep event graphs" based on at least one of the occurrence frequencies of sleep stages and the occurrence frequencies of sleep events included in the sleep state information generated during a plurality of sleep sessions.

The graphical user interface may include a plurality of areas, each area corresponding to a different type of sleep state information.

The above multiple areas can be expressed so as to be distinct from each other, each corresponding to the sleep state information.

The graphical user interface further includes an area indicating that the sleep session has ended, and the area indicating that the sleep session has ended can be expressed to be distinct from the plurality of areas.

The step of generating the above sleep state information may further include the step of generating the sleep state information for a time corresponding to one or more epochs.

The above epoch may be set to data corresponding to 30-second units.

A device providing a graphical user interface that displays information about a user's sleep according to one embodiment of the present invention includes: an acquisition unit that acquires the user's sleep information; one or more processors; and an output unit, wherein the processor generates sleep state information of the user based on the acquired sleep information of the user, generates a sleep state information graph that displays sleep state information over time based on the generated sleep state information, and generates a graphical user interface including the sleep state information graph, and the output unit can output a graphical user interface including the generated sleep state information graph.

The above sleep state information may include at least one of sleep stage information, sleep stage probability information, sleep event information, and sleep event probability information.

The above processor can generate at least one of a stacked sleep stage graph "G stacked sleep event graph" based on at least one of an occurrence frequency of a sleep stage and an occurrence frequency of a sleep event included in the sleep state information generated during a plurality of sleep sessions.

The graphical user interface may include a plurality of areas, each area corresponding to a different type of sleep state information.

The above multiple areas can be expressed so as to be distinct from each other, each corresponding to the sleep state information.

The graphical user interface may further include an area indicating that the sleep session has ended.

The region indicating that the above sleep session has ended may be expressed to be distinct from the plurality of regions.

The above processor can generate the sleep state information for a time corresponding to one or more epochs.

The above epoch may be set to data corresponding to 30-second units.

According to one embodiment of the present invention, a method for providing interpretation content of a user's sleep data may include a step of acquiring the user's sleep information, a step of generating the user's sleep state information based on the acquired user's sleep information, a step of generating the user's sleep data interpretation content based on the generated sleep state information, and a step of outputting the generated user's sleep data interpretation content.

In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the interpretation content of the user's sleep data can be expressed in at least one of a numerical method and a non-numeric method.

In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the interpretation content of the user's sleep data may be generated based on comparing the generated sleep state information of the user with at least one of the user's sleep data generated over a predetermined period of time in the past, a medical standard, and sleep data of another user generated over a predetermined period of time in the past.

In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the interpretation content of the user's sleep data may be generated based on at least one of a lookup table or a large-scale language model.

In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the sleep state information can be generated based on combining the acquired user's sleep information and other information into multimodal data.

In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the sleep state information may be generated based on a portion of the user's acquired sleep information cut off by a predetermined length on a time axis.

According to one embodiment of the present invention, an electronic device for providing interpretation content of a user's sleep data includes a sensing unit for obtaining the user's sleep information, a wireless communication unit, one or more processors, and an output unit, wherein the wireless communication unit transmits the obtained sleep information to an external terminal, the external terminal generates sleep state information of the user based on the transmitted sleep information of the user, and generates sleep data interpretation content of the user based on the generated sleep state information, the wireless communication unit receives the generated sleep data interpretation content of the user from the external terminal, and the output unit can output the received sleep data interpretation content of the user.

In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the interpretation content of the user's sleep data can be expressed in at least one of a numerical method and a non-numeric method.

In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the interpretation content of the user's sleep data may be generated based on comparing the generated sleep state information of the user with at least one of the user's sleep data generated over a predetermined period of time in the past, a medical standard, and sleep data of another user generated over a predetermined period of time in the past.

In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the interpretation content of the user's sleep data may be generated based on at least one of a lookup table or a large-scale language model.

In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the sleep state information can be generated based on combining the acquired user's sleep information and other information into multimodal data.

In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the sleep state information may be generated based on a portion of the user's acquired sleep information cut off by a predetermined length on a time axis.

According to one embodiment of the present invention, a method for providing interpretation content of a user's sleep data comprises the steps of: obtaining sleep information of the user; generating sleep state information of the user based on the obtained sleep information of the user, wherein the sleep state information includes a sleep apnea category, a sleep onset latency category, and a first cycle sleep quality category. A method for providing a ...

In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the step of calculating an importance score between categories of the generated sleep state information further includes a step of learning an importance parameter based on human feedback-based reinforcement learning, and the importance score can be calculated based on the learned importance parameter.

In a method for providing interpretation content of a user's sleep data according to one embodiment of the present invention, the step of calculating an importance score between categories of the generated sleep state information may further include a step of calculating the importance score based on comparing the generated sleep state information of the user with at least one of the user's sleep data generated during a predetermined period in the past, a medical standard, and sleep data of another user generated during a predetermined period in the past.

According to one embodiment of the present invention, an electronic device for providing interpretation content of a user's sleep data includes a sensing unit for obtaining sleep information of the user, a wireless communication unit, one or more processors, and an output unit, wherein the wireless communication unit transmits the obtained sleep information to an external terminal, and the external terminal generates sleep state information of the user based on the transmitted sleep information of the user - the sleep state information includes a sleep apnea category, a sleep onset latency category, and a first cycle sleep quality category. The sleep data belongs to at least one category among the REM Latency category, the REM Ratio category, the Deep Ratio category, the Wake Time after Sleep Onset (WASO) category, the Number of Awakening category, and the Total Sleep Time category, and the user's sleep data interpretation content is generated based on the generated sleep state information, and an importance score between categories of the generated sleep state information is calculated, and the wireless communication unit receives the generated user's sleep data interpretation content and ranking information of the calculated importance score from the external terminal, and the output unit can output the sleep data interpretation content belonging to a category having a higher ranking of the importance score based on the importance score between categories of the received sleep state information so that the sleep data interpretation content can be easily identified more than the sleep data interpretation content belonging to another category.

In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, an importance score between categories of the generated sleep state information can be calculated based on an importance parameter learned based on human feedback-based reinforcement learning.

In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, an importance score between categories of the generated sleep state information may be calculated based on comparing the generated sleep state information of the user with at least one of the user's sleep data generated during a predetermined period in the past, a medical standard, and sleep data of another user generated during a predetermined period in the past.

According to one embodiment of the present invention, an electronic device for providing interpretation content of a user's sleep data includes a communication module, a processor, and a memory, wherein the communication module receives, from the sensor device, sleep information of the user obtained from one or more sleep information sensor devices, and the processor generates sleep state information of the user based on the received sleep information of the user, wherein the sleep state information includes a sleep apnea category, a sleep onset latency category, and a first cycle sleep quality category. The sleep data belongs to at least one category among the REM Latency category, the REM Ratio category, the Deep Ratio category, the Wake Time after Sleep Onset (WASO) category, the Number of Awakening category, and the Total Sleep Time category, and the user's sleep data interpretation content is generated based on the generated sleep state information, and an importance score between categories of the generated sleep state information is calculated, and the communication module can transmit the generated user's sleep data interpretation content and ranking information of the calculated importance score to the user terminal.

In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, an importance score between categories of the generated sleep state information can be calculated based on an importance parameter learned based on human feedback-based reinforcement learning.

In an electronic device for providing interpretation content of a user's sleep data according to one embodiment of the present invention, an importance score between categories of the generated sleep state information may be calculated based on comparing the generated sleep state information of the user with at least one of the user's sleep data generated during a predetermined period in the past, a medical standard, and sleep data of another user generated during a predetermined period in the past.

The present invention relates to a system for initiating sleep measurement through a user terminal, comprising: a permission management tool for collecting permissions required to perform sleep measurement; and a sleep data collection tool for activating a sensor module for collecting environmental sensing information of the user terminal based on the collected permissions.

Additionally, the permission management tool is configured to request at least one of the Overlay permission permission, the foreground service permission, the activation permission of the sensor module, and the time-based alarm permission.

Additionally, the permission management tool is configured to provide the user with a permission activation notification or guide them to the settings screen when the permission is disabled.

In addition, the sleep data collection tool is configured to detect an event through the activated sensor module, and a broadcast trigger for sleep measurement of the user terminal is generated based on the detected event, and when the broadcast trigger is generated, the sensor module is further activated for sleep measurement.

In addition, the detected event is at least one of initiation of wired charging of the user terminal, initiation of wireless charging of the user terminal, detection of a change in the accelerometer sensor of the sensor module, switching of the display unit of the user terminal to an OFF state, a change in the battery status of the user terminal, connection of the user terminal with another device, and unlocking of the user terminal.

Additionally, the sleep data collection tool is configured to transmit, when the broadcast trigger occurs, environmental sensing information collected by the additionally activated sensor module for sleep measurement from the time of occurrence of the broadcast trigger to another device for sleep analysis.

Additionally, the sleep data collection tool is configured to provide a sleep measurement confirmation user interface to the user terminal when the broadcast trigger occurs.

In addition, the device further includes a time reservation tool for reserving a time for initiating the sleep measurement; wherein the time reservation tool is configured to generate a broadcast trigger for initiating the sleep measurement at the reserved time, and when the broadcast trigger is generated, the device is configured to further activate the sensor module for sleep measurement.

Additionally, the sleep data collection tool is configured to forcibly generate a broadcast trigger even in the low power mode of the user terminal.

Additionally, the time scheduling tool is configured to forcibly generate a broadcast trigger for initiating the sleep measurement at the scheduled time even in the low power mode of the user terminal.

The present invention relates to a system for initiating sleep measurement through a user terminal, comprising: a permission management tool for collecting permissions required to perform sleep measurement; and a sleep data collection tool for activating a sensor module of the user terminal based on the collected permissions; wherein the sleep data collection tool is configured to detect a first event through the activated sensor module, and a broadcast trigger of sleep measurement of the user terminal is configured to occur based on the detected first event, and when the broadcast trigger occurs, the sensor module is further activated for sleep measurement, and a second event is detected through the additionally activated sensor module for sleep measurement, and a sleep analysis trigger of sleep measurement of the user terminal is configured to occur based on the detected second event, and the detected second event is configured to be detected by a stimulus higher than a threshold by the sensor module.

In addition, the first event detected is at least one of initiation of wired charging of the user terminal, initiation of wireless charging of the user terminal, detection of a change in the accelerometer sensor of the sensor module, switching of the display unit of the user terminal to an OFF state, a change in the battery status of the user terminal, connection of the user terminal with another device, and unlocking of the user terminal.

In addition, the second event configured to be detected by the sensor module by a stimulus higher than a threshold is at least one of initiation of wired charging of the user terminal, initiation of wireless charging of the user terminal, detection of a change in an accelerometer sensor of the sensor module, a change in a battery status of the user terminal, connection of the user terminal with another device, unlocking of the user terminal, a palm swipe for the user terminal, a finger swipe for the user terminal, a finger tap for the user terminal, a voice input for the user terminal, vibration detection of the user terminal, a screen wheel scroll for the user terminal, pressing a specific user interface of the user terminal, pressing the display unit of the user terminal for a predetermined period of time or longer, moving two or more fingers simultaneously for the display unit, turning the user terminal over, activation of a proximity sensor of the user terminal, detection of a temperature change of the user terminal, and detection of a change in illumination around the user terminal.

Additionally, the sleep analysis trigger includes information on when sleep analysis of the measured sleep begins.

The present invention relates to a system for initiating sleep measurement through a user terminal, comprising: a permission management tool for collecting permissions required to perform sleep measurement; and a sleep data collection tool for activating a sensor module of the user terminal based on the collected permissions; wherein the sleep data collection tool is configured to detect a first event through the activated sensor module, and a broadcast trigger of sleep measurement of the user terminal is configured to occur based on the detected first event, and when the broadcast trigger occurs, the sensor module is further activated for sleep measurement, and a second event is detected through the additionally activated sensor module for sleep measurement, and a sleep analysis trigger of sleep measurement of the user terminal is configured to occur based on the detected second event, and the detected second event is configured to be detected by a stimulus below a threshold level by the sensor module.

In addition, the second event configured to be detected by the sensor module by a stimulus below the threshold is at least one of: when there is no touch input to the display unit of the user terminal for a predetermined period of time or longer; when there is no movement of the accelerometer for the user terminal for a predetermined period of time or longer; when noise around the user terminal decreases and a sound below the threshold is detected for a predetermined period of time or longer; when illuminance around the user terminal decreases and illuminance below the threshold is detected for a predetermined period of time or longer; when the locked state of the user terminal is maintained for a predetermined period of time or longer; when data transmission and reception with a network connected to the user terminal is not performed for a predetermined period of time or longer; when the battery consumption rate of the user terminal falls below the threshold; and when a change in activity data of another device connected to the user terminal is not detected for a predetermined period of time or longer.

The present invention relates to a server which receives environmental sensing information from a user terminal that generates a broadcast trigger for initiating sleep measurement and/or a sleep analysis trigger including time point information for starting sleep analysis of the measured sleep, and performs sleep analysis, the server comprising: a communication module which receives the environmental sensing information from the user terminal; a processor which performs the sleep analysis based on the received environmental sensing information; and a memory which stores a sleep analysis model for performing the sleep analysis; wherein the communication module is configured to receive environmental sensing information acquired by the user terminal from the time point of generation of the broadcast trigger generated by the user terminal.

Additionally, the communication module is configured to receive information on the start time of sleep analysis of the measured sleep from the user terminal.

In addition, when the processor performs the sleep analysis based on the received environmental sensing information, the environmental sensing information prior to the time point at which the sleep analysis of the measured sleep starts among the environmental sensing information is configured to be deleted.

In addition, when the processor performs the sleep analysis based on the received environmental sensing information, it is configured to perform the sleep analysis only on the environmental sensing information after the point in time when the sleep analysis of the measured sleep starts among the environmental sensing information.

In addition, when the processor performs the sleep analysis based on the received environmental sensing information, it is configured to acquire sleep sound information only from environmental sensing information after the point in time from which the sleep analysis of the measured sleep starts among the environmental sensing information.

In addition, when the processor performs the sleep analysis based on the received environmental sensing information, it is configured to acquire sleep state information only from environmental sensing information after the point in time at which the sleep analysis of the measured sleep starts among the environmental sensing information.

The present invention relates to a system for initiating sleep measurement through a user terminal, comprising: a permission management tool for collecting permissions required to perform sleep measurement; a sleep data collection tool for activating a sensor module of the user terminal based on the collected permissions; and a time reservation tool for storing a time for activating the sensor module; wherein the time reservation tool is configured to detect a first event at the stored time, and a broadcast trigger of sleep measurement of the user terminal is configured to occur based on the detected first event, the sleep data collection tool is configured to detect a second event through the activated sensor module, and a transmission trigger of sleep measurement of the user terminal is configured to occur based on the detected second event, and the detected second event is configured to be detected by a stimulus greater than a threshold by the sensor module.

A method for providing content related to user's sleep state information according to one embodiment of the present invention includes the steps of: obtaining environmental sensing information through a sensor module of a user terminal; converting sleep sound information included in the obtained environmental sensing information into a spectrogram; processing the spectrogram as an input of a sleep analysis model to generate user's sleep state information; and providing content related to the sleep state information through a content provision model based on the sleep state information. The sleep analysis model may be an artificial intelligence model including one or more artificial neural networks, and may include a feature extraction model and a feature classification model. The content provision model may be an interactive artificial intelligence model based on natural language processing including one or more artificial neural networks.

In a method for providing content related to user's sleep state information according to one embodiment of the present invention, the feature extraction model may be a model configured to analyze a frequency pattern of the user's breathing included in the sleep sound information, and the feature classification model may be a model configured to analyze a periodic pattern of the user's breathing included in the sleep sound information.

In a method for providing content related to sleep state information of a user according to one embodiment of the present invention, the content provision model may include one or more prompts, and the one or more prompts may include a prompt for specifying a role of the content provision model, a prompt for specifying a purpose of the content provision model, and a prompt for presenting an output format through the content provision model.

In a method for providing content related to sleep state information of a user according to one embodiment of the present invention, the content provision model includes one or more tools, and the one or more tools include a tool for accessing an external database, and the external database can be a static database or a dynamic database.

In a method for providing content related to user's sleep state information according to one embodiment of the present invention, at least two of the environmental sensing information, the generated sleep state information, and conversation information with the user through the content provision model are mapped to each other and configured as metadata, and the metadata can be stored in at least one of a database of a sleep analysis server on which the sleep analysis model is implemented and a database of a content provision server on which the content provision model is implemented.

In a method for providing content related to a user's sleep state information according to one embodiment of the present invention, the content provision model may be configured to proactively provide content related to the sleep state information through the conversational interface even without a conversational input from the user terminal.

According to one embodiment of the present invention, a user terminal for providing content related to user's sleep state information includes a sensor module, an input unit, a processor, an output unit, wherein the output unit is configured to output an interactive interface, and a communication module, wherein the interactive interface includes content related to the sleep state information received from the content providing server through the communication module, wherein the content is generated by the content providing server through a content providing model based on the sleep state information received from a sleep analysis server, and wherein the content providing model is an interactive artificial intelligence model based on natural language processing including one or more artificial neural networks, and wherein the sleep state information is obtained by the sleep analysis server processing the transmitted sleep sound information as an input of a sleep analysis model when sleep sound information included in environmental sensing information acquired by the sensor module is transmitted to the sleep analysis server through the communication module, and the sleep analysis model is an artificial intelligence model including one or more artificial neural networks and may include a feature extraction model and a feature classification model.

In a user terminal for providing content related to sleep state information of a user according to one embodiment of the present invention, the feature extraction model may be a model configured to analyze a frequency pattern of the user's breathing included in the sleep sound information, and the feature classification model may be a model configured to analyze a periodic pattern of the user's breathing included in the sleep sound information.

In a user terminal for providing content related to sleep state information of a user according to one embodiment of the present invention, the content provision model may include one or more prompts, and the one or more prompts may include a prompt for specifying a role of the content provision model, a prompt for specifying a purpose of the content provision model, and a prompt for presenting an output format through the content provision model.

In a user terminal for providing content related to sleep state information of a user according to one embodiment of the present invention, the content provision model includes one or more tools, and the one or more tools include a tool for accessing an external database, and the external database may be a static database or a dynamic database.

In a user terminal for providing content related to sleep state information of a user according to one embodiment of the present invention, at least two of the environmental sensing information, the generated sleep state information, and conversation information with the user through the content provision model are mapped to each other and configured as metadata, and the metadata can be stored in at least one of a database of a sleep analysis server on which the sleep analysis model is implemented and a database of a content provision server on which the content provision model is implemented.

In a user terminal for providing content related to a user's sleep state information according to one embodiment of the present invention, the user terminal may be configured to proactively provide content related to the sleep state information through the conversational interface even without a conversational input from the user terminal.

According to one embodiment of the present invention, a server for providing content related to sleep state information of a user includes a memory, a communication module, and a processor, wherein the processor is configured to generate the content through a content provision model implemented in the memory based on the sleep state information received from a sleep analysis server through the communication module, the processor is configured to generate an interactive interface including the generated content, and the processor is configured to transmit the interactive interface to a user terminal through the communication module, wherein the content provision model is an interactive artificial intelligence model based on natural language processing including one or more artificial neural networks, and the sleep state information is obtained when sleep sound information included in environmental sensing information acquired by the sensor module is transmitted to the sleep analysis server through the communication module, and the sleep analysis server processes the transmitted sleep sound information as an input of the sleep analysis model, and the sleep analysis model is an artificial intelligence model including one or more artificial neural networks, and may include a feature extraction model and a feature classification model.

In a server for providing content related to user's sleep state information according to one embodiment of the present invention, the feature extraction model may be a model configured to analyze a frequency pattern of the user's breathing included in the sleep sound information, and the feature classification model may be a model configured to analyze a periodic pattern of the user's breathing included in the sleep sound information.

In a server for providing content related to sleep state information of a user according to one embodiment of the present invention, the content provision model may include one or more prompts, and the one or more prompts may include a prompt for specifying a role of the content provision model, a prompt for specifying a purpose of the content provision model, and a prompt for presenting an output format through the content provision model.

In a server for providing content related to sleep state information of a user according to one embodiment of the present invention, the content provision model includes one or more tools, and the one or more tools include a tool for accessing an external database, and the external database may be a static database or a dynamic database.

In a server for providing content related to user's sleep state information according to one embodiment of the present invention, at least two of the environmental sensing information, the generated sleep state information, and conversation information with the user through the content provision model are mapped to each other and configured as metadata, and the metadata can be stored in at least one of a database of a sleep analysis server on which the sleep analysis model is implemented and a database of a content provision server on which the content provision model is implemented.

In a server for providing content related to a user's sleep state information according to one embodiment of the present invention, the server may be configured to proactively provide content related to the sleep state information through the conversational interface even without a conversation input from the user terminal.

According to the present invention, by generating and providing information about the user's sleep as a graphical user interface, it is possible to provide information about the user's sleep to the user, thereby contributing to improving the quality of the user's sleep. In addition, by presenting the generated sleep state information in a time series from the start time of the sleep session, it is possible to easily understand the information analyzed for sleep.

According to the present invention, by analyzing the user's sleep based on acoustic information, it is possible to monitor the user's sleep state and thereby contribute to improving the quality of the user's sleep.

In addition, according to the present invention, there is no need to install a microphone in contact with the body to obtain the user's sleep information, and the sleep state can be monitored in a typical home environment only through a software update without purchasing a separate additional device, thereby providing the effect of increasing convenience.

In addition, according to the present invention, when analyzing acoustic information in the time domain, only a portion cut off by a predetermined length on the time axis is analyzed, so that the size of data used as input to the sleep analysis model can be relatively reduced, thereby reducing the sleep analysis time.

In addition, according to the present invention, when analyzing acoustic information in the time domain, only a portion cut off by a predetermined length on the time axis is analyzed, thereby contributing to improving the quality of sleep of the user by analyzing sleep for a relatively short period of time.

In addition, according to the present invention, there is also an effect that highly accurate sleep analysis is possible by performing sleep analysis in a multimodal manner.

In addition, according to the present invention, convenience and sleep quality are improved in that the user can easily recognize the sleep analysis content by providing the user's sleep data interpretation content.

In addition, according to the present invention, convenience is improved in that the user can easily recognize the sleep analysis content by providing the user's sleep data interpretation content.

In addition, according to the present invention, based on the importance scores between categories of sleep information, the user's sleep data interpretation content can be generated or provided so that data in a category of sleep information having a relatively high importance score can be easily identified more than sleep data belonging to other categories of sleep information, thereby enabling the user to easily recognize the sleep analysis content and improving convenience.

The present invention can greatly improve user convenience. It can prevent the loss of sleep data by automatically starting sleep measurement without the user having to start sleep measurement separately. In particular, it can prevent the situation where the user forgets to start sleep measurement even though he or she intends to do so, thereby enabling accurate and continuous data collection.

In addition, the present invention can record the user's actual sleep start time more precisely by combining a broadcast trigger and a sleep analysis trigger. By comprehensively analyzing time reservation, user behavior data, and environmental data, the reliability of sleep data is increased, and the user can obtain more accurate sleep analysis results.

The present invention is designed to enable triggers and alarms to operate even in the low power mode of an Android system, thereby resolving the problem of functional limitations that may arise due to battery optimization, thereby enabling stable and continuous sleep measurement in various environments.

Furthermore, the present invention provides a multi-faceted trigger function utilizing user behavior data as well as time-based triggers. It supports both user-initiated events (e.g., swiping) and initiated events (e.g., reduced motion) to adapt to various usage patterns.

According to the present invention, by analyzing the user's sleep based on acoustic information, it is possible to monitor the user's sleep state and thereby contribute to improving the quality of the user's sleep.

In addition, according to the present invention, there is no need to install a microphone in contact with the body to obtain the user's sleep information, and the sleep state can be monitored in a typical home environment only through a software update without purchasing a separate additional device, thereby providing the effect of increasing convenience.

In addition, according to the present invention, when analyzing acoustic information in the time domain, only a portion cut off by a predetermined length on the time axis is analyzed, so that the size of data used as input to the sleep analysis model can be relatively reduced, thereby reducing the sleep analysis time.

In addition, according to the present invention, when analyzing acoustic information in the time domain, only a portion cut off by a predetermined length on the time axis is analyzed, thereby contributing to improving the quality of a user's sleep by analyzing sleep for a relatively short period of time.

In addition, according to the present invention, there is also an effect that accurate sleep analysis is possible by performing sleep analysis in a multimodal manner.

In addition, the present invention is effective in that it provides content related to the user's sleep state, so that the user can easily recognize the sleep analysis results and easily obtain information for improving the quality of sleep.

FIG. 1A is a conceptual diagram illustrating a system in which various aspects of one or more graphical user interface generating devices (100) for displaying information about a user's sleep according to one embodiment of the present invention may be implemented.

FIG. 1b is a conceptual diagram illustrating a system in which various aspects of one or more graphical user interface providing devices (200) that display information about a user's sleep according to one embodiment of the present invention can be implemented.

FIG. 2a is a conceptual diagram illustrating a system in which generation and/or provision of one or more graphical user interfaces representing information about a user's sleep is implemented in a user terminal (10) according to another embodiment of the present invention.

FIG. 2b is a conceptual diagram illustrating a system in which various aspects of various electronic devices related to another embodiment of the present invention can be implemented.

Figure 3a is a block diagram for explaining a user terminal (10) according to one embodiment of the present invention.

FIG. 3b is a block diagram showing the configuration of one or more graphical user interface generating devices (100)/providing devices (200) that display information about a user's sleep according to one embodiment of the present invention.

FIG. 3c is a block diagram for explaining an external server (20) according to one embodiment of the present invention.

FIG. 4a is a diagram illustrating a graphical user interface including a hipnogram according to one embodiment of the present invention.

FIG. 4b is a graph showing the time ratio of each sleep stage measured according to one embodiment of the present invention.

Figure 4c is a drawing showing a respiratory stability graph according to an embodiment of the present invention.

FIG. 4d is a diagram illustrating a graphical user interface including a hipnogram according to another embodiment of the present invention.

FIG. 4e is a diagram illustrating a graphical user interface including a description display for respiratory instability according to one embodiment of the present invention.

FIGS. 5A and 5B are diagrams illustrating a graphical user interface including statistical information of sleep state information according to embodiments of the present invention.

FIGS. 6A and 6B are diagrams illustrating a graphical user interface including sleep status information acquired over a week according to embodiments of the present invention.

FIGS. 7A and 7B are diagrams illustrating a graphical user interface including sleep state information acquired over a predetermined period of time according to embodiments of the present invention.

FIGS. 8A through 8E are black and white drawings of a graphical user interface including a Stacked Sleep Stage Graph according to embodiments of the present invention.

Figure 9a is a drawing for explaining a process of obtaining sleep sound information in a sleep analysis method according to the present invention.

FIG. 9b is a drawing for explaining a method of obtaining a spectrogram corresponding to sleep sound information in a sleep analysis method according to the present invention.

FIG. 10 is a flowchart of a method for generating and providing one or more graphical user interfaces representing information regarding a user's sleep according to one embodiment of the present invention.

FIG. 11 is a schematic diagram showing one or more network functions for performing a sleep analysis method according to the present invention.

Figure 12 is a drawing for explaining sleep stage analysis using a spectrogram in a sleep analysis method according to the present invention.

Figure 13 is a drawing for explaining sleep disorder determination using a spectrogram in a sleep analysis method according to the present invention.

Figure 14 is a drawing showing an experimental process for verifying the performance of a sleep analysis method according to the present invention.

FIG. 15a and FIG. 15b are drawings for explaining the overall structure of a sleep analysis model according to one embodiment of the present invention.

FIG. 16 is a diagram for explaining a feature extraction model and a feature classification model according to one embodiment of the present invention.

FIG. 17a and FIG. 17b are graphs verifying the performance of a sleep analysis method according to the present invention, and are drawings comparing the results of polysomnography (PSG) and the results of analysis using an AI algorithm according to the present invention.

Figure 18 is a graph verifying the performance of a sleep analysis method according to the present invention, and is a drawing comparing the results of polysomnography (PSG) with the results of analysis using an AI algorithm according to the present invention in relation to sleep apnea and hypopnea.

FIGS. 19A to 19C are diagrams showing aspects in which a graphical user interface according to embodiments of the present invention is displayed on various display units.

Figures 20a to 20e are diagrams showing hypnogram graphs of sleep stage information expressed in a conventional sleep measurement interface.

FIG. 21 is a diagram illustrating a graphical user interface that displays in parallel a graph representing sleep stage information and a graph representing sleep event information according to one embodiment of the present invention.

FIGS. 22A through 22E are diagrams illustrating a graphical user interface including a Stacked Sleep Stage Graph according to embodiments of the present invention.

FIGS. 23A through 23D are diagrams illustrating a graphical user interface including a Stacked Sleep Stage Graph generated based on a larger number of sleep sessions compared to FIGS. 22A through 22E, according to embodiments of the present invention.

FIGS. 24a and 24b are conceptual diagrams illustrating a system in which various aspects of a sleep data interpretation content creation device or a sleep data interpretation content providing device based on user sleep information according to one embodiment of the present invention can be implemented.

Figure 24c is a conceptual diagram illustrating a system in which sleep data interpretation content creation and provision based on user sleep information according to one embodiment of the present invention is implemented in a user terminal (300).

FIG. 24d is a conceptual diagram illustrating a system of a sleep data interpretation content creation device (100a) based on user sleep information according to one embodiment of the present invention.

Figure 24e is a conceptual diagram illustrating a system of a sleep data interpretation content providing device (100b) based on user sleep information according to one embodiment of the present invention.

FIG. 24f is a conceptual diagram illustrating a system in which various aspects of various electronic devices according to embodiments of the present invention can be implemented.

FIG. 25a and FIG. 25b are block diagrams illustrating a computing device (100) according to one embodiment of the present invention.

FIG. 25c is a block diagram for explaining an external terminal (200) according to one embodiment of the present invention.

FIG. 25d is a block diagram illustrating a user terminal (300) according to one embodiment of the present invention.

FIG. 26a and FIG. 26b are graphs verifying the performance of a sleep analysis method according to the present invention, and are drawings comparing the results of polysomnography (PSG) and the results of analysis using an AI algorithm according to the present invention (AI result).

Figure 26c is a graph verifying the performance of a sleep analysis method according to the present invention, and is a drawing comparing the results of polysomnography (PSG) with the results of analysis using an AI algorithm according to the present invention in relation to sleep apnea and hypopnea.

Figure 27 is a drawing showing an experimental process for verifying the performance of a sleep analysis method according to the present invention.

FIG. 28 is an exemplary diagram illustrating a process of obtaining sleep sound information from environmental sensing information according to one embodiment of the present invention.

FIG. 29a is an exemplary diagram illustrating a method for obtaining a spectrogram corresponding to sleep sound information according to one embodiment of the present invention.

Figure 29b is a drawing for explaining sleep stage analysis using a spectrogram in a sleep analysis method according to the present invention.

Figure 29c is a drawing for explaining sleep event determination using a spectrogram in a sleep analysis method according to the present invention.

FIG. 30a is a schematic diagram illustrating one or more network functions according to one embodiment of the present invention.

FIG. 30b is a diagram for explaining the structure of a sleep analysis model utilizing deep learning to analyze a user's sleep according to one embodiment of the present invention.

FIG. 31A is a diagram for explaining a method for generating interpretation content of sleep data based on a lookup table according to one embodiment of the present invention.

FIG. 31b is a diagram for explaining a method for generating interpretation content of sleep data based on a large-scale language model according to one embodiment of the present invention.

FIG. 31c is a flowchart of a method for providing non-numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

FIG. 31d is a flowchart of a method for providing numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

FIG. 32 is a flowchart illustrating a method for learning importance parameters that serve as a basis for calculating importance scores of sleep categories according to embodiments of the present invention.

FIG. 33A is an exemplary diagram illustrating one or more graphical user interfaces representing non-numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

FIG. 33b is an exemplary diagram illustrating one or more graphical user interfaces representing non-numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

FIG. 33c is an exemplary diagram illustrating one or more graphical user interfaces representing non-numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

FIG. 33d is an exemplary diagram illustrating one or more graphical user interfaces representing numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

FIG. 33e is an exemplary diagram illustrating one or more graphical user interfaces representing numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

FIGS. 34 and 35 are graphs for explaining a method of calculating importance scores of categories of sleep information based on medical criteria according to embodiments of the present invention.

FIG. 36 is a graph for explaining a method of calculating an importance score of a category of sleep information based on information compared with sleep data of a user generated over a predetermined period of time in the past according to embodiments of the present invention.

Figure 37 is a diagram for explaining consistency training according to one embodiment of the present invention.

FIG. 38 is a flowchart illustrating a method for analyzing sleep state information, including a process of combining sleep sound information and sleep environment information into multimodal data according to one embodiment of the present invention.

FIG. 39 is a flowchart illustrating a method for analyzing sleep state information, including a step of combining each of inferred sleep sound information and sleep environment information into multimodal data according to one embodiment of the present invention.

FIG. 40 is a flowchart illustrating a method for analyzing sleep state information, including a step of combining inferred sleep sound information with sleep environment information and multimodal data according to one embodiment of the present invention.

FIG. 41 is a diagram for explaining a linear regression analysis function utilized to analyze AHI, an index of sleep apnea occurrence, through sleep events occurring during sleep according to one embodiment of the present invention.

Figures 42a to 42c are conceptual diagrams showing exemplary systems in which services according to the present invention can be implemented.

FIGS. 42d and 42e are conceptual diagrams illustrating an exemplary system in which a service for providing content related to a user's sleep state information according to the present invention can be implemented.

FIG. 42f is a conceptual diagram illustrating a system in which various aspects of various electronic devices according to embodiments of the present invention can be implemented.

FIG. 43a and FIG. 43b are block diagrams illustrating a computing device (100) according to one embodiment of the present invention.

Figure 43c is a block diagram for explaining a server (200) according to one embodiment of the present invention.

FIG. 43d is a block diagram illustrating a user terminal (300) according to one embodiment of the present invention.

Figure 44 is a schematic diagram of a data set for sleep analysis according to one embodiment of the present invention.

FIG. 45a is a diagram for explaining noise reduction according to one embodiment of the present invention.

Figure 45b is a diagram for explaining a process of preprocessing and converting data according to one embodiment of the present invention.

Figure 45c is a diagram for explaining a data conversion process according to one embodiment of the present invention.

Figure 46 is a drawing for explaining a method of obtaining a spectrogram corresponding to sleep sound information in a sleep analysis method according to the present invention.

FIG. 47 is a flowchart exemplarily showing a method for analyzing a user's sleep state through acoustic information according to one embodiment of the present invention.

FIGS. 48a and 48b are drawings for explaining the overall structure of a sleep analysis model according to one embodiment of the present invention.

FIG. 49 is a diagram for explaining a feature extraction model and a feature classification model according to one embodiment of the present invention.

Figure 50 is a drawing for explaining in detail the operation of a sleep analysis model according to one embodiment of the present invention.

FIG. 51a and FIG. 51b are diagrams for explaining the performance of sleep event determination and noise addition using a spectrogram in a sleep analysis method according to the present invention.

Figure 52 is a drawing for explaining sleep stage analysis using a spectrogram in a sleep analysis method according to the present invention.

Figure 53 is a drawing for explaining sleep event determination using a spectrogram in a sleep analysis method according to the present invention.

Figure 54 is a drawing showing an experimental process for verifying the performance of a sleep analysis method according to the present invention.

Figure 55 is a graph verifying the performance of a sleep analysis method according to the present invention, and is a drawing comparing the polysomnography (PSG) result (PSG result) and the analysis result (AI result) using the AI algorithm according to the present invention.

Figure 56 is a graph verifying the performance of a sleep analysis method according to the present invention, and is a drawing comparing the results of polysomnography (PSG) with the results of analysis using an AI algorithm according to the present invention in relation to sleep apnea and hypopnea.

FIG. 57 is a drawing for explaining the configuration of an automatic measurement system according to one embodiment of the present invention.

FIG. 58 is a diagram showing a screen where an app requests and manages specific permissions in an Android system according to one embodiment of the present invention.

FIG. 59a is a diagram showing a main screen of a user interface for automatic sleep measurement through an app according to one embodiment of the present invention. FIG. 59b is a diagram showing an Auto Tracking setting screen for setting an automatic measurement schedule according to one embodiment of the present invention. FIG. 59c is a diagram showing a user interface for displaying results after automatic sleep measurement is completed according to one embodiment of the present invention.

FIG. 60 is a diagram for explaining a process for setting up automation of sleep measurement using a Shortcuts app in an iOS environment according to one embodiment of the present invention.

FIG. 61 is a diagram for explaining a process of setting up automation in an iOS Shortcuts App according to one embodiment of the present invention.

FIG. 62 is a diagram illustrating a process by which a user adds a new action (e.g., starting sleep measurement) to an automated task to be executed at a specific time in a shortcut app according to one embodiment of the present invention.

FIG. 63 is a drawing showing a screen for setting a state in which a sleep measurement termination task according to one embodiment of the present invention can be automatically registered and executed via App Intent.

FIG. 64 is a drawing showing an embodiment of providing an activation notification of a sensor module when a broadcast trigger occurs according to the present invention.

Figure 65 is a diagram explaining the structure of the Transformer model that is the basis of the Large Language Model.

FIG. 66 is a diagram for explaining an inverse diffusion model (Inverter Model) of a DIFFUSION model in a content-generating artificial intelligence according to an embodiment of the present invention.

FIG. 67 is a diagram for explaining a generator and a discriminator of a GAN (Generative Adversarial Network) in a content-generating artificial intelligence according to one embodiment of the present invention.

FIG. 68 is a diagram for explaining a method for generating interpretation content of sleep data based on a large-scale language model according to one embodiment of the present invention.

FIG. 69a is a flowchart of a method for providing non-numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

FIG. 69b is a flowchart of a method for providing numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

The following description sets forth exemplary methods, parameters, etc. However, it should be recognized that such description is not intended to be a limitation on the scope of the present invention, but instead is provided as a description of exemplary embodiments.

There is a need for a method for providing one or more graphical user interfaces that efficiently display information about a user's sleep. Providing information about a user's sleep in an efficient manner can provide information for understanding the user's sleep information from a medical perspective in a user-friendly manner and can provide more suitable information to the user.

Although the following description uses terms such as "first," "second," etc. to describe various elements, these elements should not be limited by these terms. These terms are used to distinguish one element from another.

Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device may include a portable communication device (e.g., a mobile phone, a smart watch) that includes other functions, such as PDA and music player functions. Alternatively, an embodiment may include an electronic device that includes a touch-sensitive surface and display.

Additionally, in some embodiments of the present invention, an electronic device including a display (or touch-sensitive surface) may be included.

Overall composition

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

Although each step described in this specification is described as being performed by a computer, the subject of each step is not limited thereto, and at least some of each step may be performed by different devices depending on the embodiment.

User terminal (10)

Figure 3a is a block diagram for explaining a user terminal (10) according to one embodiment of the present invention.

According to one embodiment of the present invention, the user terminal (10) is a terminal that can receive information related to the user's sleep through information exchange with at least one of various electronic devices (e.g., a graphical user interface generating device, a providing device, or an external server) according to embodiments of the present invention, and may refer to a terminal possessed by the user. For example, the user terminal (10) may be a terminal related to a user who wants to improve his/her health through information related to his/her sleep habits. The user may obtain monitoring information related to his/her sleep through the user terminal (10). The monitoring information related to sleep may include, for example, sleep state information related to the time when the user fell asleep, the time he/she slept, the time when he/she woke up from sleep, or sleep stage information related to changes in sleep stages during sleep. For a specific example, the sleep stage information may refer to information on changes in the user's sleep to light sleep, normal sleep, deep sleep, or REM sleep at each time point during the user's 8 hours of sleep last night. The specific description of the sleep stage information described above is only an example, and the present invention is not limited thereto.

The above user terminal (10) may include a mobile phone, a smart phone, a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a smart glass, a head mounted display (HMD)), etc.

The above user terminal (10) may include a wireless communication unit (11), an input unit (12), a sensing unit (14), an output unit (15), an interface unit (16), a memory (17), a control unit (18), and a power supply unit (19). The components illustrated in FIG. 2f are not essential for implementing the user terminal, and thus the user terminal described in this specification may have more or fewer components than the components listed above.

Wireless Communication Department (11)

The wireless communication unit (11) may include one or more modules that enable wireless communication between a user terminal (10) and a wireless communication system, between a user terminal (10) and a graphical user interface generating device (100), a graphical user interface providing device (200), or between a user terminal (10) and an external server (20). In addition, the wireless communication unit (11) may include one or more modules that connect the user terminal (10) to one or more networks.

This wireless communication unit (11) may include at least one of a broadcast reception module (311), a mobile communication module (312), a wireless Internet module (313), a short-range communication module (314), and a location information module (315).

Input section (12)

According to the present invention, the input unit (12) may include a camera (321) or an image input unit for inputting an image signal, a microphone (322) or an audio input unit for inputting an audio signal, and a user input unit (323, for example, a touch key, a mechanical key, etc.) for receiving information from a user. Voice data or image data collected by the input unit (12) may be analyzed and processed into a user's control command.

Sensing part (14)

According to the present invention, the sensing unit (14) may include one or more sensors for sensing at least one of information within the user terminal, information regarding the surrounding environment surrounding the user terminal, and user information. For example, the sensing unit (14) may include at least one of a proximity sensor (341), an illumination sensor (342), a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor), a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor (e.g., a camera (see 321)), a battery gauge, an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor, etc.), and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric recognition sensor, etc.). Meanwhile, the user terminal disclosed in this specification can utilize information sensed by at least two or more of these sensors in combination.

Output section (15)

According to the present invention, the output unit (15) is for generating output related to vision, hearing, or tactile sensations, and may include at least one of a display unit (351), an audio output unit (352), a haptic module (353), and an optical output unit (354).

According to the present invention, the display unit (351) can implement a touch screen by forming a mutual layer structure with the touch sensor or by forming an integral structure. This touch screen can function as a user input unit (323) that provides an input interface between the user terminal (10) and the user, and at the same time, provide an output interface between the user terminal (10) and the user.

Interface section (16)

According to the present invention, the interface unit (16) serves as a passageway for various types of external devices connected to the user terminal (10). This interface unit (16) may include at least one of a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I/O (Input/Output) port, a video I/O (Input/Output) port, and an earphone port. In the user terminal (10), appropriate control related to the connected external device (e.g., an electronic device) may be performed in response to the connection of the external device to the interface unit (16).

Memory (17)

According to the present invention, the memory (17) stores data supporting various functions of the user terminal (10). The memory (17) can store a plurality of application programs (or applications) running on the user terminal (10), data, commands, or instructions for the operation of the user terminal (10). At least some of these application programs can be downloaded from an external server via wireless communication. In addition, at least some of these application programs can exist on the user terminal (10) from the time of shipment for the basic functions of the user terminal (10) (e.g., call reception, call transmission, message reception, call transmission). Meanwhile, the application program can be stored in the memory (17), installed on the user terminal (10), and driven by the control unit (18) to perform the operation (or function) of the user terminal. The memory (17) can store instructions for the operation of the control unit (18).

Control unit (18)

According to the present invention, the control unit (18) controls the overall operation of the user terminal (10) in addition to the operation related to the application program. The control unit (18) processes signals, data, information, etc. input or output through the components discussed above, or operates the application program stored in the memory (17), thereby providing or processing appropriate information or functions to the user.

According to the present invention, the control unit (18) can control at least some of the components examined with reference to Fig. 2f in order to drive the application program stored in the memory (17). Furthermore, the control unit (18) can operate at least two or more of the components included in the user terminal (10) in combination with each other in order to drive the application program.

Power supply unit (19)

According to the present invention, the power supply unit (19) receives external power and internal power under the control of the control unit (18) and supplies power to each component included in the user terminal (10). The power supply unit (19) includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the above components may operate in cooperation with each other to implement the operation, control, or control method of the user terminal according to various embodiments described below. In addition, the operation, control, or control method of the user terminal may be implemented on the user terminal by driving at least one application program stored in the memory (17).

Graphical User Interface Generation Device (100)

A graphical user interface generation device (100) according to the present invention can generate a graphical user interface for delivering information to a user based on the user's sleep information (e.g., environmental sensing information, etc.).

Alternatively, the graphical user interface generation device (100) according to embodiments of the present invention may generate a graphical user interface for conveying information to the user based on sleep state information generated based on the user's sleep information.

FIG. 1A is a conceptual diagram illustrating a system in which various aspects of one or more graphical user interface generating devices (100) representing information about a user's sleep according to one embodiment of the present invention may be implemented. As illustrated in FIG. 1A, a system according to embodiments of the present invention may include a graphical user interface generating device (100), a user terminal (10), an external server (20), and a network.

Here, a system implementing one or more graphical user interface generating devices (100) for displaying information about a user's sleep as illustrated in FIG. 1a is according to one embodiment, and its components are not limited to the embodiment illustrated in FIG. 1a, and may be added, changed, or deleted as needed.

Device providing graphical user interface (200)

The graphical user interface providing device (200) according to the present invention can output a graphical user interface generated based on the user's sleep information (e.g., environmental sensing information, etc.) or sleep state information to provide the user with the graphical user interface.

FIG. 1b is a conceptual diagram illustrating a system in which various aspects of one or more graphical user interface providing devices (200) that display information about a user's sleep according to one embodiment of the present invention can be implemented. As illustrated in FIG. 1b, a system according to embodiments of the present invention may include a graphical user interface providing device (200), a user terminal (10), an external server (20), and a network.

Here, a system implementing one or more graphical user interface providing devices (200) for displaying information about a user's sleep as illustrated in FIG. 1b is according to one embodiment, and its components are not limited to the embodiment illustrated in FIG. 1b, and may be added, changed, or deleted as needed.

Meanwhile, FIG. 2a is a conceptual diagram illustrating a system in which, according to another embodiment of the present invention, generation and/or provision of one or more graphical user interfaces representing information about a user's sleep is implemented in a user terminal (10). As illustrated in FIG. 2a, generation and/or provision of one or more graphical user interfaces representing information about a user's sleep may be performed in a user terminal (10) without a separate generation device (100) and/or a separate provision device (200).

Additionally, FIG. 2b illustrates a conceptual diagram of a system in which various aspects of various electronic devices related to another embodiment of the present invention may be implemented.

The electronic devices illustrated in FIG. 2b can perform at least one of the operations performed by various devices according to embodiments of the present invention.

For example, operations performed by various electronic devices according to embodiments of the present invention may include an operation of acquiring environmental sensing information, an operation of performing learning for sleep analysis, an operation of performing inference for sleep analysis, and an operation of acquiring sleep state information.

Or, for example, it may include an operation of receiving information related to the user's sleep, transmitting or receiving environmental sensing information, determining environmental sensing information, extracting acoustic information from environmental sensing information, processing or manipulating data, processing a service, providing a service, constructing a learning data set based on the environmental sensing information or the information related to the user's sleep, storing acquired data or a plurality of data that become inputs to a neural network, transmitting or receiving various pieces of information, exchanging data for a system according to embodiments of the present invention through a network, generating one or more graphical user interfaces representing information related to the user's sleep, or providing one or more graphical user interfaces representing information related to the user's sleep.

External Server (20)

FIG. 3c is a block diagram for explaining an external server (20) according to one embodiment of the present invention.

According to the present invention, the external server (20) may include a processor (21), a memory (22), and a communication module (27). The external server (20) may be a cloud server or an edge server. The external server (20) may be a digital device, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone, which has a computing ability equipped with a processor and a memory. The external server (20) may be a web server that processes a service. The types of servers described above are merely examples, and the present invention is not limited thereto.

Processor (21)

According to the present invention, the processor (21) controls the external server (20) as a whole. The processor (21) may include an AI processor (215).

The AI processor (215) can learn a neural network using a program stored in the memory (22). In particular, the AI processor (215) can learn a neural network for recognizing data related to the operation of the user terminal (10). Here, the neural network can be designed to simulate the human brain structure (e.g., the neuron structure of the human neural network) on a computer. The neural network can include an input layer, an output layer, and at least one hidden layer. Each layer includes at least one neuron having a weight, and the neural network can include a synapse connecting neurons to neurons. In the neural network, each neuron can output an input signal input through a synapse as a function value of an activation function for a weight and/or a bias.

A plurality of network modes can send and receive data according to their connection relationships, respectively, so as to simulate the synaptic activity of neurons in which neurons send and receive signals through synapses. Here, the neural network may include a deep learning model developed from a neural network model. In the deep learning model, a plurality of network nodes are located in different layers and can send and receive data according to convolutional connection relationships. Examples of neural network models include various deep learning techniques such as a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network, a restricted Boltzmann machine, a deep belief network, and a deep Q-network, and can be applied in fields such as vision recognition, speech recognition, natural language processing, and speech/signal processing.

Meanwhile, the processor (21) performing the function as described above may be a general-purpose processor (e.g., CPU), but may be an AI-only processor for artificial intelligence learning (e.g., GPU, TPU).

Memory (22)

According to the present invention, the memory (22) can store various programs and data required for the operation of the user terminal (10) and/or the external server (20). The memory (22) is accessed by the AI processor (215), and data reading/recording/modifying/deleting/updating, etc. can be performed by the AI processor (215). In addition, the memory (22) can store a neural network model (e.g., a deep learning model) generated through a learning algorithm for data classification/recognition. Furthermore, the memory (22) can store not only the learning model (221), but also input data, learning data, learning history, etc.

Meanwhile, the AI processor (215) may include a data learning unit (215a) that learns a neural network for data classification/recognition. The data learning unit (215a) may learn criteria regarding which learning data to use to determine data classification/recognition and how to classify and recognize data using the learning data. The data learning unit (215a) may acquire learning data to be used for learning and learn a deep learning model by applying the acquired learning data to the deep learning model.

The data learning unit (215a) may be manufactured in the form of at least one hardware chip and mounted on an external server (20). For example, the data learning unit (215a) may be manufactured in the form of a dedicated hardware chip for artificial intelligence, and may be manufactured as a part of a general-purpose processor (CPU) or a graphics processor (GPU) and mounted on an external server (20). In addition, the data learning unit (215a) may be implemented as a software module. When implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer readable media that can be read by a computer. In this case, at least one software module may be provided to an operating system (OS) or provided by an application.

The data learning unit (215a) can learn to have a judgment criterion on how to classify/recognize a given data by using the acquired learning data. At this time, the learning method by the model learning unit can be classified into supervised learning, unsupervised learning, and reinforcement learning. Here, supervised learning refers to a method of learning an artificial neural network in a state where a label for learning data is given, and the label may mean the correct answer (or result value) that the artificial neural network should infer when learning data is input to the artificial neural network. Unsupervised learning may mean a method of learning an artificial neural network in a state where a label for learning data is not given. Reinforcement learning may mean a method of learning to select an action or action sequence that maximizes the cumulative reward in each state for an agent defined in a specific environment. In addition, the model learning unit may learn the neural network model using a learning algorithm including error backpropagation or gradient descent. When a neural network model is trained, the trained neural network model can be called a training model (221). The training model (221) can be stored in memory (22) and used to infer results for new input data that is not training data.

According to the present invention, the AI processor (215) may further include a data preprocessing unit (215b) and/or a data selection unit (215c) to improve analysis results using a learning model (221) or to save resources or time required for generating a learning model (221).

According to the present invention, the data preprocessing unit (215b) can preprocess the acquired data so that the acquired data can be used for learning/inference for situation judgment. For example, the data preprocessing unit (215b) can extract feature information as preprocessing for input data received through the communication module (27), and the feature information can be extracted in a format such as a feature vector, a feature point, or a feature map.

According to the present invention, the data selection unit (215c) can select data required for learning from among the learning data or the learning data preprocessed in the preprocessing unit. The selected learning data can be provided to the model learning unit. For example, the data selection unit (215c) can select only data for an object included in a specific area as learning data by detecting a specific area from an image acquired through a camera of an electronic device. In addition, the data selection unit (215c) can select data required for inference from among the input data acquired through an input device or the input data preprocessed in the preprocessing unit.

In addition, the AI processor (215) may further include a model evaluation unit (215d) to improve the analysis results of the neural network model. The model evaluation unit (215d) may input evaluation data into the neural network model, and if the analysis results output from the evaluation data do not satisfy a predetermined criterion, cause the model learning unit to learn again. In this case, the evaluation data may be preset data for evaluating the learning model (221). For example, the model evaluation unit (215d) may evaluate that the predetermined criterion is not satisfied if the number or ratio of evaluation data for which the analysis results are inaccurate among the analysis results of the learned neural network model for the evaluation data exceeds a preset threshold.

Communication module (27)

The communication module (27) can transmit and receive data and/or information to and from at least one of the user terminal (10), the graphical user interface generating device (100), the graphical user interface providing device (200), and other electronic devices according to one embodiment of the present invention. For example, it can receive environmental sensing information from the user terminal (10) and transmit the AI processing result by the AI processor (215) to the user terminal (10). The manner in which data is transmitted and/or received is not limited thereto.

Various embodiments of the present invention

The electronic devices illustrated in FIG. 2b may individually perform operations performed by various electronic devices according to embodiments of the present invention, but may also perform one or more operations simultaneously or in time series. At least one of the electronic devices illustrated in FIG. 2b may be a user terminal (10). At least one of the electronic devices illustrated in FIG. 2b may be a graphical user interface generating device (100). At least one of the electronic devices illustrated in FIG. 2b may be a graphical user interface providing device (200). At least one of the electronic devices illustrated in FIG. 2b may be an external server (20).

Referring to FIG. 2b, the electronic devices (10a to 10d) illustrated in FIG. 2b may be electronic devices within the range of an area (11a) capable of acquiring environmental sensing information. Hereinafter, for convenience, the area (11a) capable of acquiring environmental sensing information will be referred to as “area (11a).”

Meanwhile, referring to FIG. 2b, the electronic devices (10a and 10d) may be devices formed by a combination of two or more electronic devices.

Meanwhile, referring to FIG. 2b, the electronic devices (10a and 10b) may be electronic devices connected to a network within the area (11a).

Meanwhile, referring to FIG. 2b, electronic devices (10c and 10d) may be electronic devices not connected to a network within the area (11a).

Meanwhile, referring to FIG. 2b, the electronic devices (20a and 20b) may be electronic devices outside the range of the area (11a).

Meanwhile, referring to FIG. 2b, there may be a network that interacts with electronic devices within the scope of area (11a), and there may be a network that interacts with electronic devices outside the scope of area (11a).

Here, a network interacting with electronic devices within the scope of the area (11a) can play a role in transmitting and receiving information for controlling smart home appliances.

Additionally, the network interacting with the electronic devices within the scope of the area (11a) may be, for example, a short-range network or a local network. Here, the network interacting with the electronic devices within the scope of the area (11a) may be, for example, a long-range network or a global network.

Since the specific description of the operation of the networks illustrated in Fig. 2b is the same as that described above, redundant description will be omitted.

Meanwhile, referring to FIG. 2b, there may be one or more electronic devices connected through a network outside the scope of the area (11a), and in this case, the electronic devices may perform distributed processing of data or divide and perform one or more operations. Here, the electronic devices connected through a network outside the scope of the area (11a) may include a server device.

Alternatively, if there is one or more electronic devices connected via a network outside the scope of area (11a), the electronic devices may perform various operations independently of one another.

As illustrated in FIGS. 1A and 1B, one or more graphical user interface generating devices (100) representing information about a user's sleep and one or more graphical user interface providing devices (200) representing information about a user's sleep can mutually transmit and receive data for a system according to embodiments of the present invention with a user terminal (10) through a network.

As illustrated in FIG. 2A, even if one or more graphical user interface generating devices (100) representing information about the user's sleep and one or more graphical user interface providing devices (200) representing information about the user's sleep are not separately provided, a user terminal (10) may perform the role of one or more graphical user interface generating devices (100) representing information about the user's sleep and/or one or more graphical user interface providing devices (200) representing information about the user's sleep through a network, thereby mutually transmitting and receiving data for a system according to embodiments of the present invention.

network

As illustrated in FIG. 2b, various electronic devices according to the present invention can mutually transmit and receive data for a system according to embodiments of the present invention through a network.

The network according to embodiments of the present invention can use various wired communication systems such as Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), Very High Speed DSL (VDSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN). In addition, the network presented herein can use various wireless communication systems such as Code Division Multi Access (CDMA), Time Division Multi Access (TDMA), Frequency Division Multi Access (FDMA), Orthogonal Frequency Division Multi Access (OFDMA), Single Carrier-FDMA (SC-FDMA), and other systems.

The network according to embodiments of the present invention can be configured regardless of its communication mode, such as wired or wireless, and can be configured with various communication networks, such as a personal area network (PAN) and a wide area network (WAN). In addition, the network can be the well-known World Wide Web (WWW), and can also use a wireless transmission technology used for short-distance communication, such as infrared (IrDA: Infrared Data Association) or Bluetooth. The technologies described in this specification can be used not only in the networks mentioned above, but also in other networks.

FIG. 3b is a block diagram showing the configuration of one or more graphical user interface generating devices (100)/providing devices (200) that display information about a user's sleep according to one embodiment of the present invention.

According to one embodiment of the present invention, one or more graphical user interface generating devices (100)/providing devices (200) for displaying information about a user's sleep may include a display (120)/(220), a memory (140)/(240) storing one or more programs configured to be executed by one or more processors, and one or more processors (160)/(260).

According to one embodiment of the present invention, the memory (140)/(240) storing one or more programs includes a high-speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and a non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid state storage devices. Additionally, the memory can store instructions for performing a method of providing one or more graphical user interfaces that display information about a user's sleep.

According to one embodiment of the present invention, the processor (160)/(260) may be configured as one or more. Additionally, the processor may execute a memory storing one or more programs.

Sleep Information

According to one embodiment of the present invention, sleep information may be acquired from one or more sleep information sensor devices to generate one or more graphical user interfaces representing information about the user's sleep. The sleep information may include sleep sound information of the user acquired in a non-invasive manner during the user's activity or sleep. In addition, the sleep information may include the user's lifestyle information and the user's log data.

Meanwhile, in the present invention, sleep information may include environmental sensing information and user's lifestyle information. The user's lifestyle information may include information affecting the user's sleep. Specifically, information affecting the user's sleep may include the user's age, gender, disease, occupation, bedtime, wake-up time, heart rate, electrocardiogram, and sleep time. For example, if the user's sleep time is less than the reference time, it may affect the user's sleep the next day by requiring more sleep time. On the other hand, if the user's sleep time is more than the reference time, it may affect the user's sleep the next day by requiring less sleep time.

One or more sleep information sensor devices

According to one embodiment of the present invention, one or more sleep sensor devices may include a microphone module, a camera, and a light sensor provided in a user terminal (10).

For example, information related to the user's activities in a work space can be obtained through a microphone module equipped in the user terminal (10).

In addition, since the microphone module must be equipped in a relatively small-sized user terminal (10), it may be configured as a MEMC (Micro-electro Mechanical System).

Environmental sensing information

In an embodiment, environmental sensing information of the present invention can be acquired through a user terminal (10). Environmental sensing information may refer to sensing information acquired in a space where a user is located. Environmental sensing information may be sensing information acquired in relation to a user's activity or sleep in a non-contact manner.

According to one embodiment of the present invention, it may mean sensing information acquired in an area (11a) capable of acquiring environmental sensing information as illustrated in FIG. 2b, but is not limited thereto.

For example, the environmental sensing information may be sleep sound information obtained in a bedroom where the user is sleeping. According to an embodiment, the environmental sensing information obtained through the user terminal (10) may be information that serves as a basis for obtaining the user's sleep state information in the present invention. For a specific example, sleep state information regarding whether the user is before sleep, during sleep, or after sleep may be obtained through the environmental sensing information obtained in relation to the user's activity.

For another example, the environmental sensing information may include sounds generated as the user tosses and turns during sleep, sounds related to muscle movement, or sounds related to the user's breathing during sleep. The sleep sound information may refer to sound information related to movement patterns and breathing patterns that occur during the user's sleep. Environmental sensing information related to the user's activities in a space may be acquired through a microphone equipped in the user terminal (10).

Alternatively, the environmental sensing information may include the user's breathing and movement information. The user terminal (10) may include a radar sensor as a motion sensor. The user terminal (10) may perform signal processing on the user's movement and distance measured through the radar sensor to generate a discrete waveform (breathing information) corresponding to the user's breathing. A quantitative index may be obtained based on the discrete waveform and movement.

The environmental sensing information may include measurements obtained through sensors that measure temperature, humidity, and lighting levels in the space where the user is sleeping. For this purpose, the user terminal (10) may be equipped with sensors that measure temperature, humidity, and lighting levels in the bedroom.

According to one embodiment of the present invention, a graphical user interface generating device (100) or a graphical user interface providing device (200) can obtain sleep state information based on environmental sensing information obtained through a microphone module composed of MEMS.

Specifically, a graphical user interface generating device (100) or a graphical user interface providing device (200) can convert environmental sensing information that is acquired unclearly, including a lot of noise, into data that can be analyzed, and can perform learning for an artificial neural network by utilizing the converted data.

According to one embodiment of the present invention, when pre-learning for an artificial neural network is completed, the learned neural network can obtain the user's sleep state information based on a spectrogram obtained in response to sleep sound information. Specifically, the learned neural network may be an artificial intelligence sound analysis model, but is not limited thereto.

That is, the graphical user interface generation device (100)/providing device (200) according to embodiments of the present invention can obtain sound information having a low signal-to-noise ratio through a user terminal (e.g., an artificial intelligence speaker, a bedroom IoT device, a mobile phone, a wearable device, etc.) that is generally widely used to collect sound.

Alternatively, acoustic information or sleep acoustic information may be obtained through a microphone module provided in the graphical user interface generating device (100)/providing device (200) according to embodiments of the present invention.

When a graphical user interface generating device (100) or a graphical user interface providing device (200) or a user terminal (10) obtains acoustic information or sleep acoustic information having a low signal-to-noise ratio, it can process the acquired information into data suitable for analysis and process the processed data to provide sleep state information.

Acoustic information or sleep acoustic information according to an embodiment of the present invention may be obtained through a microphone module configured to come into contact with the user's body, but may also be obtained through a microphone module not configured to come into contact with the user's body.

This can provide the effect of increasing convenience by eliminating the need to install a microphone in contact with the user's body to obtain clear sound, and by enabling monitoring of sleep status in a typical home environment with only a software update without purchasing a separate additional device with a high signal-to-noise ratio.

Although the graphical user interface generation device (100) in FIG. 1a is expressed as a separate entity from the user terminal (10), according to an embodiment of the present invention, as shown in FIG. 2a, the graphical user interface generation device (100) may be included in the user terminal (10) to perform the functions of measuring sleep status and generating a graphical user interface in a single integrated device.

Similarly, although the graphical user interface providing device (200) in FIG. 1b is expressed as a separate entity from the user terminal (10), according to an embodiment of the present invention, as shown in FIG. 2a, the graphical user interface providing device (200) may be included in the user terminal (10) to perform the functions of measuring sleep status and providing a graphical user interface in a single integrated device.

The user terminal (10) may mean any form of entity(ies) in a system having a mechanism for communicating with an external server (20) or a separate electronic device. For example, the user terminal (10) may include a personal computer (PC), a notebook, a mobile terminal, a smart phone, a tablet PC, an artificial intelligence (AI) speaker, an artificial intelligence TV, and a wearable device, and may include all types of terminals that can connect to a wired/wireless network. In addition, the user terminal (10) may include any server implemented by at least one of an agent, an application programming interface (API), and a plug-in. In addition, the user terminal (10) may include an application source and/or a client application.

When an external server (20) according to one embodiment of the present invention receives sleep sound information, the received sleep sound information can be processed into appropriate data so that sleep state information can be generated based on the received sleep sound information.

According to one embodiment of the present invention, the external server (20) may be a server that stores information on a plurality of learning data for learning a neural network. The plurality of learning data may include, for example, health checkup information or sleep checkup information. For example, the external server (20) may be at least one of a hospital server and an information server, and may be a server that stores information on a plurality of polysomnography records, electronic health records, electronic medical records, and the like. For example, the polysomnography records may include information on breathing and movement during sleep of a sleep examination subject and information on sleep diagnosis results (for example, sleep stages, etc.) corresponding to the information. The information stored in the external server (20) may be utilized as learning data, verification data, and test data for learning a neural network in the present invention.

At least one of the user terminal (10), the graphical user interface generating device (100), or the graphical user interface providing device (200) according to embodiments of the present invention may receive health checkup information or sleep checkup information, etc. from an external server (20), and construct a learning data set based on the corresponding information. In addition, at least one of the user terminal (10), the graphical user interface generating device (100), or the graphical user interface providing device (200), or the external server (20) according to embodiments of the present invention may perform learning for one or more network functions through the learning data set, thereby generating a sleep analysis model for obtaining sleep state information corresponding to environmental sensing information, or may obtain sleep state information through the implemented sleep analysis model. A specific description of a configuration for constructing a learning data set for neural network learning of the present invention and a learning method utilizing the learning data set will be described later.

Acquisition of environmental sensing information

Environmental sensing information or sleep information according to embodiments of the present invention may be obtained from one or more sensor devices. In addition, a sensor device according to one embodiment of the present invention may be implemented in the form of a user terminal (10).

Environmental sensing information may refer to sensing information obtained in the space where the user is located. Environmental sensing information may be sensing information obtained in the space where the user is located using a non-contact method.

For example, the environmental sensing information may be sound information obtained in a bedroom where the user is sleeping. According to an embodiment, the environmental sensing information obtained through the user terminal (10) may be information that serves as a basis for obtaining the user's sleep state information in the present invention. For a specific example, sleep state information regarding whether the user is before, during, or after sleep may be obtained through environmental sensing information obtained in relation to the user's activity.

In addition, the environmental sensing information may be at least one of noise information commonly occurring in daily life (sound information related to cleaning, sound information related to cooking, sound information related to watching TV, cat sounds, dog sounds, bird sounds, car sounds, wind sounds, rain sounds, etc.) or other biometric information (electrocardiogram, brain waves, pulse information, information related to muscle movement, etc.).

The user terminal (10) may mean any form of entity(ies) in a system having a mechanism for communicating with the graphical user interface generating device (100)/providing device (200). For example, the user terminal (10) may include a personal computer (PC), a notebook, a mobile terminal, a smart phone, a tablet PC, an artificial intelligence (AI) speaker, an artificial intelligence TV, a wearable device, etc., and may include all types of terminals that can connect to a wired/wireless network. In addition, the user terminal (10) may include any server implemented by at least one of an agent, an application programming interface (API), and a plug-in. In addition, the user terminal (10) may include an application source and/or a client application.

According to one embodiment, the processor (160) can obtain environmental sensing information. Specifically, the environmental sensing information can be obtained through a user terminal (10) carried by the user. For example, environmental sensing information related to a space in which the user is active can be obtained through the user terminal (10) carried by the user, and the processor (160) can receive the corresponding environmental sensing information from the user terminal (10).

In addition, an external server according to one embodiment of the present invention may record an artificial intelligence model for analyzing sleep state information. In this case, if environmental sensing information is obtained from a user terminal (10) or the like and the obtained environmental sensing information is transmitted to an external server (20), the external server (20) may generate sleep state information based on the environmental sensing information through the implemented artificial intelligence model.

Alternatively, according to one embodiment of the present invention, environmental sensing information may be acquired from a user terminal (10), sleep sound information may be acquired through preprocessing of the environmental sensing information from the user terminal (10), the user terminal (10) may transmit the acquired sleep sound information to an external server, and the external server may generate sleep state information based on the received sleep sound information.

A graphical user interface generating device (100)/providing device (200) according to one embodiment of the present invention can receive health checkup information or sleep checkup information, etc. from an external server and construct a learning data set based on the information.

A graphical user interface generating device (100)/providing device (200) according to one embodiment of the present invention can generate a sleep analysis model for obtaining sleep state information based on environmental sensing information by performing learning on one or more network functions through a learning data set. A specific description of a configuration for constructing a learning data set for neural network learning of the present invention and a learning method utilizing the learning data set will be described later.

The external server may be a digital device, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone, which has a processor and a computing power with memory. The external server may be a web server that processes services. The types of servers described above are only examples and the present invention is not limited thereto.

According to one embodiment of the present invention, a graphical user interface generating device (100)/providing device (200) can obtain sleep state information of a user, and generate and/or provide one or more graphical user interfaces that display information about the user's sleep based on the sleep state information of the user.

Specifically, a graphical user interface generating device (100)/providing device (200) according to one embodiment of the present invention can obtain sleep state information related to whether the user is before, during, or after sleep based on environmental sensing information, and can generate and/or provide one or more graphical user interfaces that display information about the user's sleep based on the obtained sleep state information of the user.

Meanwhile, the specific description related to the aforementioned sleep state information is only an example, and the present invention is not limited thereto.

According to one embodiment of the present invention, one or more sleep sensor devices may include a microphone module, a camera, and a light sensor provided in a user terminal (10).

For example, information related to the user's activities in a space can be obtained through a microphone module equipped in the user terminal (10). In addition, when sensing information through a microphone module equipped in the user terminal (10), the microphone module must be equipped in a relatively small-sized user terminal (10), and thus may be configured as a MEMS (Micro-electro Mechanical System).

The microphone module according to embodiments of the present invention can be manufactured very compactly, but may have a low signal-to-noise ratio (SNR) compared to a condenser microphone or a dynamic microphone. A low signal-to-noise ratio may mean that the ratio of noise, which is a sound that is not intended to be identified, to the sound that is intended to be identified is high, and thus the sound is not easy to identify (i.e., unclear).

In addition, the environmental sensing information to be analyzed in the present invention may include acoustic information related to the user's breathing and movement acquired during sleep. Such acoustic information is information about very small sounds (i.e., sounds that are difficult to distinguish) such as the user's breathing and movement, and is acquired together with other sounds during the sleep environment. Therefore, if acquired through a microphone module with a low signal-to-noise ratio as described above, detection and analysis may be very difficult.

In such a case, an electronic device according to an embodiment of the present invention can transform and/or adjust environmental sensing information that is acquired unclearly, including a lot of noise, into data that can be analyzed, and can perform learning for an artificial neural network by utilizing the transformed and/or adjusted data. When pre-learning for the artificial neural network is completed, the learned neural network (e.g., an acoustic analysis model) can acquire information on the user's sleep state based on data (e.g., a spectrogram) acquired (e.g., transformed and/or adjusted) corresponding to sleep sound information.

In an embodiment, the sleep state information may include not only information regarding whether the user is sleeping, but also sleep stage information regarding changes in the user's sleep stage during sleep. For a specific example, the sleep state information may include sleep stage information indicating that the user was in REM sleep at a first time point and that the user was in shallow sleep at a second time point that is different from the first time point. In this case, through the sleep state information, information may be obtained that the user was in relatively deep sleep at the first time point and was in shallower sleep at the second time point.

That is, when the graphical user interface generating device (100)/providing device (200) according to embodiments of the present invention obtains sleep sound information having a low signal-to-noise ratio through a user terminal (e.g., an artificial intelligence speaker, a bedroom IoT device, a mobile phone, a wearable device, etc.) that is generally widely distributed to collect sound, it can process the acquired sleep sound information into data suitable for analysis, and process the processed data to provide sleep state information related to changes in sleep stages.

In an embodiment, the graphical user interface generating device (100)/providing device (200) may be a terminal or a server, and may include any type of device. The graphical user interface generating device (100)/providing device (200) may be a digital device, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone, which has a computing capability equipped with a processor and a memory. Alternatively, the graphical user interface generating device (100)/providing device (200) may be a web server that processes a service.

According to one embodiment of the present invention, the graphical user interface generation device (100)/providing device (200) may be a server providing a cloud computing service. More specifically, the graphical user interface generation device (100)/providing device (200) may be a server providing a cloud computing service that processes information with another computer connected to the Internet rather than the user's computer as a type of Internet-based computing.

The above cloud computing service may be a service that stores data on the Internet and allows users to use it anytime and anywhere through Internet access without having to install necessary data or programs on their computers, and allows users to easily share and transmit data stored on the Internet with simple operations and clicks. In addition, the cloud computing service may be a service that not only simply stores data on a server on the Internet, but also allows users to perform desired tasks by utilizing the functions of application programs provided on the Web without having to install separate programs, and allows multiple people to simultaneously share documents and work on them.

In addition, the cloud computing service can be implemented in the form of at least one of IaaS (Infrastructure as a Service), PaaS (Platform as a Service), SaaS (Software as a Service), a virtual machine-based cloud server, and a container-based cloud server. That is, the graphical user interface generation device (100)/providing device (200) of the present invention can be implemented in the form of at least one of the above-described cloud computing services. The specific description of the above-described cloud computing service is merely an example, and may include any platform that constructs the cloud computing environment of the present invention. Meanwhile, the types of servers described above are merely examples, and the present invention is not limited thereto.

Sleep sound information

Meanwhile, in the present invention, sleep information may include environmental sensing information and user's life information. The environmental sensing information may be sound information about the user's sleep. One or more sleep information sensor devices may collect raw data about sounds generated during sleep for analysis of sleep. The raw data about sounds generated during sleep may be information in the time domain.

Meanwhile, in the present invention, by performing preprocessing on sound information acquired by a sleep information sensor device, it is also possible to acquire sleep sound information containing information on breathing and/or movement patterns.

Specifically, sleep sound information may contain information about the user's breathing and/or movement patterns related to sleep. For example, when awake, since all nervous systems are activated, breathing patterns may be irregular and there may be a lot of body movement. In addition, since the neck muscles do not relax, breathing sounds may be very small. On the other hand, when the user is sleeping, the autonomic nervous system becomes stabilized, so breathing changes regularly, body movement may also decrease, and breathing sounds may also become louder. In addition, when apnea occurs during sleep, a loud breathing sound may occur immediately after apnea as a compensatory mechanism. Since sleep sound information contains such information about breathing and/or movement, raw data about sleep can be collected to obtain sleep sound information, and analysis of sleep can be performed based on the obtained sleep sound information.

Information in the time domain

Meanwhile, in the present invention, sleep information may include at least one of environmental sensing information, sleep environment information, or user's life information. Here, the environmental sensing information or sleep environment information may be sound information for analyzing the user's sleep. One or more sleep information sensor devices may collect raw data regarding sounds generated during sleep for analyzing sleep. The raw data regarding sounds generated during sleep may be information in the time domain. The term "raw sound information" described below means raw data regarding sounds generated during sleep.

Preprocessing of information in the time domain

According to embodiments of the present invention, acquired environmental sensing information and acoustic information are information in the time domain and may undergo a preprocessing process of noise reduction.

In the noise reduction process, noise (e.g., white noise) included in raw data is removed or reduced. The noise reduction process can be performed using algorithms such as spectral gating and spectral subtraction to remove or reduce background noise. Furthermore, in the present invention, the process of removing or reducing noise can be performed using a deep learning-based noise reduction algorithm. The deep learning-based noise reduction algorithm can use a noise reduction algorithm specialized for the user's breathing or respiration sound, in other words, a noise reduction algorithm learned through the user's breathing or respiration sound.

Preprocessing like the above can be performed during the learning process of sleep state information or during the inference process.

Spectral noise gating

Spectral gating or spectral noise gating is a preprocessing method for audio information. Noise reduction can be performed on all acquired audio information, but splitting can also be performed at regular time intervals (e.g., 5 minutes), and noise reduction can be performed on each of the split audio information. In order to perform noise reduction on audio information split at regular time intervals, a method of first calculating a spectrum for each frame can be included.

Among each spectrum frame produced, the frame with the frequency spectrum with the lowest energy can be specified.

The method may include assuming that a frame having a frequency spectrum with the lowest energy among each spectrum frame is static noise, and attenuating a frequency of the frequency spectrum frame assumed as static noise from the spectrum frame.

Deep Learning Based Noise Reduction

According to the present invention, in order to perform noise reduction preprocessing, a deep learning-based noise reduction method that is performed on raw sound information in the time domain rather than the frequency domain may be used. For deep learning-based noise reduction, information such as sleep sound information, which is information necessary for inputting a sleep analysis model, may be maintained, and other sounds may be attenuated.

According to the present invention, noise reduction can be performed not only on acoustic information obtained through PSG test results, but also on acoustic information obtained through a microphone built into a user terminal such as a smartphone.

Generation/conversion and preprocessing of information or spectrograms in the frequency domain

In one embodiment of the present invention, raw information in the time domain can be converted into information in the frequency domain in order to analyze sleep sound information. Or, according to embodiments of the present invention, raw information in the time domain can be converted into information including changes in frequency components of the raw information along the time axis.

According to an embodiment of the present invention, raw sound information from which noise is removed or reduced can be converted into information in the frequency domain or information including changes in frequency components of the raw sound information along the time axis. Preferably, it can be converted into a spectrogram. At this time, a method of converting into a spectrogram based only on the amplitude excluding the phase from the raw sound information can be used, and through this method, not only privacy can be protected, but also the processing speed can be improved by reducing the data capacity. However, in another embodiment, it is also possible to generate a spectrogram using both the phase and the amplitude.

One embodiment of the present invention can create a sleep analysis model using a spectrogram (SP) converted based on sleep sound information (SS).

One embodiment of the present invention removes or reduces noise in sleep sound information using the above-described method, converts it into a spectrogram, and trains the spectrogram to create a sleep analysis model, thereby reducing the amount of computation and computation time and even protecting an individual's privacy.

For example, among the sound information acquired through microphones, etc., the sleep sound information (e.g., the user's breathing sound, etc.) required for sleep stage analysis may be relatively smaller than other noises, but when converted to a spectrogram, the identification of the sleep sound information may be relatively superior compared to other surrounding noises.

Meanwhile, when converting to a spectrogram according to an embodiment of the present invention, personal information cannot be identified by converting the resolution of the frequency domain to a low level. In other words, when configured with a frequency resolution (frequency bins) of less than a certain number (e.g., 20), personal information cannot be identified from the restored signal.

Additionally, according to an embodiment of the present invention, a method of converting acquired acoustic information into a spectrogram in real time may be included.

Additionally, since compression of the frequency resolution of the spectrogram can be performed on the user's smartphone rather than on a server or cloud, it can also prevent leakage of personal information.

Meanwhile, a spectrogram according to an embodiment of the present invention may be a mel spectrogram to which a mel scale is applied.

Method for converting information or spectrograms in the frequency domain

Figure 9a is a drawing for explaining a process of obtaining sleep sound information in a sleep analysis method according to the present invention.

FIG. 9b is a drawing for explaining a method of obtaining a spectrogram corresponding to sleep sound information in a sleep analysis method according to the present invention.

The processor (160) can generate a spectrogram (SP) corresponding to sleep sound information (SS), as illustrated in FIG. 9b. Raw data (raw sound information in the time domain) that serves as the basis for generating the spectrogram (SP) can be input, and the raw data according to the present invention can be collected through polysomnography (PSG) in a hospital environment, or can be collected through a microphone built into a user terminal, such as a wearable device or a smartphone, in a home environment.

In addition, raw data may be acquired through the user terminal (10) from the start time input by the user to the end time, or may be acquired from the time when the user operates the device (e.g., alarm setting) to the time corresponding to the device operation (e.g., alarm setting time), or may be acquired by automatically selecting a time point based on the user's sleep pattern, or may be acquired by automatically determining the time point based on the user's intended sleep time point (e.g., user's voice, breathing sound, peripheral device (TV, washing machine) sound, etc.) or change in illumination. Meanwhile, the intended sleep time point according to one embodiment of the present invention may be calculated from the user's intended sleep time point.

According to the present invention, the processor (160) can perform a fast Fourier transform on sleep sound information (SS) to generate a sleep spectrogram (SP). The spectrogram (SP) is intended to visualize and understand sounds or waves, and may be a combination of waveform and spectrum characteristics. The spectrogram (SP) may represent the difference in amplitude according to changes in the time axis and frequency axis as a difference in printing density or display color.

According to the present invention, preprocessed sound-related raw data can be divided into 30-second units and converted into a spectrogram. Accordingly, a 30-second spectrogram has a dimension of 20 frequency bins x 1201 time steps. In the present invention, various methods such as reshaping, resizing, and split-cat can be used to change a rectangular spectrogram into a shape close to a square, thereby converting it into a shape close to a square. Alternatively, by using these methods, it is possible to relatively preserve the amount of information.

Meanwhile, the present invention can utilize a method for simulating breathing sounds measured in various home environments by adding various noises generated in a home environment to clean breathing sounds. Since sounds have an additive property, they can be added to each other. However, if original audio signals such as mp3 or pcm are added and converted into spectrograms, the consumption of computing resources becomes very large. Therefore, the present invention proposes a method for converting breathing sounds and noises into spectrograms respectively and adding them. Through this, breathing sounds measured in various home environments can be simulated and utilized for learning an artificial intelligence model, thereby ensuring the robustness of an artificial intelligence model for information on various home environments.

Preprocessing of information or spectrograms in the frequency domain

The purpose of converting data according to one embodiment of the present invention into a spectrogram is to use the spectrogram as input to a sleep analysis model and infer through a learned model which sleep state or sleep stage a pattern in the spectrogram corresponds to. However, before using the spectrogram as input to the sleep analysis model, several preprocessing processes may be required. These preprocessing processes may be performed only during the learning process, or may be performed not only during the learning process but also during the inference process. Or, they may be performed only during the inference process.

The preprocessing process of the spectrogram according to embodiments of the present invention may include data augmentation preprocessing techniques, such as a pitch shifting preprocessing method that inflates the amount of data by adding Gaussian noise to the data, a pitch shifting preprocessing method that slightly raises or lowers the overall pitch of the sound, a so-called TUT (Tile UnTile) augmentation method in which a spectrogram or mel spectrogram is converted into a vector in the learning process, and the converted vector is randomly cut (tile) at the input stage of a node (neuron) and recombined (untile) after the output of the node (neuron).

In addition, the data augmentation preprocessing method according to an embodiment of the present invention may include a noise addition augmentation method that adds noise generated in various environments (e.g., external sounds, natural sounds, fan sounds, door opening or closing sounds, animal sounds, human conversation sounds, movement sounds, etc.) rather than Gaussian noise.

In order to shorten the learning time when the spectrogram is used as an input for the learning model, the noise-added augmentation according to the embodiment of the present invention may include a method of artificially adding noise information on the sleep sound information and the spectrogram after converting the noise information into a spectrogram. In this case, there may not be a significant difference between the spectrogram obtained by converting the entirety of the original sound information domain into which the noise information is added to the sleep sound information, and the spectrogram obtained by adding the noise information on the domain in which the sleep sound information and the noise information are each converted into a spectrogram.

In addition, the noise augmentation according to an embodiment of the present invention protects the user's privacy by making it difficult to convert back from the spectrogram to the original signal by maintaining only the amplitude among the amplitude and phase in the spectrogram of each of the sleep sound information and the noise information, and adding the phase by making it an arbitrary phase, thereby making it difficult to convert back from the spectrogram to the original signal.

Alternatively, the noise augmentation according to an embodiment of the present invention may include a method of adding on a domain converted into a spectrogram with acoustic information, as well as a method of adding on a domain converted into a mel spectrogram with mel scale applied.

In addition, according to the method of adding in the mel scale according to an embodiment of the present invention, the time required for hardware to process data can be shortened.

Additionally, a preprocessing method may be performed to convert information in the transformed frequency domain, information including changes in frequency components along the time axis, or a spectrogram into a form close to a square.

Sleep status information

In one embodiment, the sleep state information may include information related to whether the user is sleeping. Specifically, the sleep state information may include at least one of first sleep state information indicating that the user is sleeping, second sleep state information indicating that the user is sleeping, and third sleep state information indicating that the user is sleeping. In other words, when the first sleep state information is inferred with respect to the user, the processor (160) may determine that the user is in a state before sleeping (i.e., before going to bed), when the second sleep state information is inferred, the processor (160) may determine that the user is in a state during sleeping, and when the third sleep state information is obtained, the processor (160) may determine that the user is in a state after sleeping (i.e., waking up).

Additionally, the sleep state information may include, in addition to information related to the user's sleep stage, information about at least one of sleep apnea, snoring, tossing and turning, coughing, sneezing, or teeth grinding (e.g., sleep event information).

In order to learn or infer sleep stage information according to embodiments of the present invention, acoustic information acquired over a long time interval may be required.

On the other hand, in order to learn or predict sleep state information (e.g., snoring or apnea information, etc.) other than sleep stage information according to embodiments of the present invention, acoustic information acquired during a relatively short time interval (e.g., 1 minute) before and after the time when the corresponding sleep state occurs may be required.

Such sleep state information may be characterized as being acquired based on environmental sensing information. The environmental sensing information may include sensing information acquired in a non-contact manner in a space where the user is located.

According to the present invention, the processor (160) can obtain sleep state information based on at least one of acoustic information, actigraphy, biometric information, environmental sensing information, and sleep information obtained from the user terminal (10).

Meanwhile, the processor (160) according to one embodiment of the present invention can identify a singular point in the acoustic information. Here, the singular point of the acoustic information may be related to breathing and movement patterns related to sleep. For example, in a wakeful state, since all nervous systems are activated, the breathing pattern may be irregular and there may be a lot of body movement. In addition, since the neck muscles are not relaxed, there may be very little breathing sound. On the other hand, when the user is sleeping, the autonomic nervous system is stabilized, so breathing changes regularly, body movement may also be reduced, and breathing sound may also be loud. That is, the processor (160) can identify a point in time when acoustic information of a pattern related to regular breathing, little body movement, or little breathing sound is detected in the acoustic information as a singular point. In addition, the processor (160) can acquire sleep acoustic information based on the acoustic information acquired based on the identified singular point. The processor (160) can identify a singular point related to the user's sleeping time in the acoustic information acquired in time series, and acquire sleep acoustic information based on the singular point.

FIG. 9A is a diagram for explaining a process of acquiring sleep sound information in a sleep analysis method according to the present invention. Referring to FIG. 9A, a processor (160) can identify a singular point (P) related to a point in time when sound information of a pattern related to regular breathing, little body movement, or little breathing sound is detected from sound information (E). Based on the identified singular point (P), the processor (160) can acquire sleep sound information (SS) based on sound information acquired after the singular point (P). The waveform and singular point related to sound in FIG. 4 are merely examples for understanding the present invention, and the present invention is not limited thereto.

That is, the processor (160) can extract and acquire only sleep sound information (SS) from a large amount of environmental sensing information (i.e., sound information) based on the singular point (P) by identifying a singular point (P) related to the user's sleep from the sound information. This can provide convenience by automating the process of the user recording his/her sleep time, and can contribute to improving the accuracy of the acquired sleep sound information.

In addition, in the embodiment, the processor (160) can obtain sleep state information related to whether the user is before or during sleep based on the singular point (P) identified from the acoustic information (E). Specifically, if the singular point (P) is not identified, the processor (160) can determine that the user is before sleep, and if the singular point (P) is identified, the processor can determine that the user is during sleep after the singular point (P). In addition, the processor (160) can identify a point in time (e.g., a time of waking up) at which the identified pattern is no longer observed after the singular point (P) is identified, and if the point in time is identified, the processor can determine that the user is after sleep, that is, has woken up.

That is, the processor (160) can obtain sleep state information related to whether the user is before, during, or after sleep based on whether a singular point (P) is identified in the acoustic information (E) and whether a preset pattern is continuously detected after the singular point is identified.

Meanwhile, the processor (160) can obtain sleep state information based on Actigraphy or biometric information, not acoustic information (E). It may be advantageous to obtain the user's movement information through a sensor unit in contact with the body. In the present invention, since the user's sleep state information is determined in advance using Actigraphy or biometric information during the first sleep analysis, the reliability of the analysis of the sleep state can be further improved.

According to one embodiment of the present invention, the processor (160) can obtain environmental sensing information. Alternatively, according to one embodiment of the present invention, the environmental sensing information can be obtained through a user terminal (10) carried by a user. For example, environmental sensing information related to a space where a user is active can be obtained through a user terminal (10) carried by a user, and the processor (160) can receive the corresponding environmental sensing information from the user terminal (10). The processor (160) according to one embodiment of the present invention can obtain sleep sound information based on the environmental sensing information.

According to the present invention, the environmental sensing information may be acoustic information acquired in a non-contact manner in the user's daily life. For example, the environmental sensing information may include various acoustic information acquired according to the user's life, such as acoustic information related to cleaning, acoustic information related to cooking, acoustic information related to watching TV, and sleep acoustic information acquired during sleep. In an embodiment, the sleep acoustic information acquired during the user's sleep may include sounds generated as the user tosses and turns during sleep, sounds related to muscle movements, or sounds related to the user's breathing during sleep. That is, the sleep acoustic information in the present invention may mean acoustic information related to movement patterns and breathing patterns related to the user's sleep.

Alternatively, according to one embodiment of the present invention, at least one of the user terminal (10) or the external server (20) may generate or infer sleep state information. When first sleep state information is inferred with respect to the user, the processor equipped in the user terminal (10) or the external server (20) may determine that the user is in a state before sleep (i.e., before going to bed), when second sleep state information is inferred, the processor may determine that the user is in a state during sleep, and when third sleep state information is obtained, the processor may determine that the user is in a state after sleep (i.e., waking up). Meanwhile, although the operation of generating sleep state information has been described as the operation of the processor (160), the above-described operation may be performed by at least one of the processors of various electronic devices according to embodiments of the present invention.

Sleep Stage Information

According to one embodiment, the processor (160) can extract sleep stage information. The sleep stage information can be extracted based on the user's environmental sensing information. The sleep stage can be divided into NREM (non-REM) sleep and REM (Rapid eye movement) sleep, and NREM sleep can be further divided into multiple stages (e.g., 2 stages of Light and Deep, 4 stages of N1 to N4). The sleep stage setting can be defined as a general sleep stage, but can also be arbitrarily set to various sleep stages depending on the designer. Through sleep stage analysis, not only the quality of sleep related to sleep but also sleep disorders (e.g., sleep apnea) and their fundamental causes (e.g., snoring) can be predicted. Meanwhile, the operation of extracting sleep stage information has been described as the operation of the processor (160), but the above-described operation can also be performed by at least one of the processors of various electronic devices according to embodiments of the present invention.

In addition, the processor (160) according to one embodiment of the present invention can generate environment creation information based on sleep stage information. For example, when the sleep stage is in the Light stage or the N1 stage, environment creation information for controlling an environment creation device (lighting, air purifier, etc.) to induce deep sleep or REM sleep can be generated. Meanwhile, the operation of generating environment creation information has been described as the operation of the processor (160), but the above-described operation may be performed by at least one of the processors of various electronic devices according to embodiments of the present invention.

According to one embodiment of the present invention, when a word indicating a light sleep stage is displayed in Korean, it may be displayed as 'light sleep' or 'normal sleep'. Users who are not experts in sleep stages may misunderstand that they did not sleep properly during the sleep period due to the word 'light sleep', but in contrast, if it is displayed as the word 'normal sleep', such misunderstandings are likely to be reduced.

Hypnogram

According to the present invention, a hypnogram is a graph that represents sleep stage information as a function of time. Through the hypnogram graph, it is possible to express REM sleep stage and NON-REM sleep from the time of falling asleep to the time of waking up. Alternatively, through the hypnogram, it is possible to express sleep stage information of a total of four stages: REM sleep, deep sleep, light sleep, and wake-up stage.

FIG. 17a and FIG. 17b are graphs verifying the performance of a sleep analysis method according to the present invention, and are drawings comparing the results of polysomnography (PSG) and the results of analysis using an AI algorithm according to the present invention.

As illustrated in FIG. 17b, the sleep stage information obtained according to the present invention is not only highly consistent with polysomnography, but also includes more precise and meaningful information related to sleep stages (Wake, Light, Deep, REM).

The hypnodensity graph illustrated at the bottom of Fig. 17a is a graph representing sleep stage probability information that represents the probability of which of the four sleep stage classes a particular sleep stage belongs to. According to an embodiment of the present invention, when predicting sleep stage information through the hypnodensity graph, it is possible to represent the probability (i.e., sleep stage probability information) of which of the four classes (Wake, Light, Deep, REM) a particular sleep stage belongs to in a periodic unit according to one or more epochs, and further, it is possible to represent the probability of which of the five classes (Wake, N1, N2, N3, REM) a particular sleep stage belongs to, the probability of which of the three classes (Wake, Non-REM, REM) a particular sleep stage belongs to, and the probability of which of the two classes (Wake, Sleep) a particular sleep stage belongs to. Here, the sleep stage probability information may mean a numerical representation of the proportion of a given sleep stage in a given epoch when classifying sleep stages.

The hypnogram, which is a graph illustrated above the hypnodensity graph, can be obtained by determining the sleep stage with the highest probability from the hypnodensity graph according to one embodiment of the present invention. As illustrated in Fig. 17b, the sleep analysis results obtained according to the present invention showed very consistent performance when compared with labeling data obtained through a polysomnography test.

Meanwhile, FIG. 4a is a drawing for explaining another example of a hypnogram indicating sleep stages within a user's sleep period according to one embodiment of the present invention.

Hypnograms can usually be obtained through electroencephalograms (EEGs), electrooculography (EOGs), electromyography (EMGs), and polysomnography (PSG).

As shown in Fig. 4a, the hypnogram displayed can express sleep stages by dividing them into REM sleep and NON-REM sleep. For example, it can be expressed as four stages: REM sleep, deep sleep, light sleep, and wakefulness. Details on the graphical user interface displaying the hypnogram will be described later.

Sleep intention information

According to one embodiment of the present invention, the processor (160) can obtain sleep intention information based on environmental sensing information. According to one embodiment, the processor (160) can identify the type of sound included in the environmental sensing information.

In addition, the processor (160) can calculate sleep intention information based on the number of types of identified sounds. The processor (160) can calculate lower sleep intention information as the number of types of sounds increases, and can calculate higher sleep intention information as the number of types of sounds decreases.

For example, if there are three types of sounds included in the environmental sensing information (e.g., vacuum cleaner sound, TV sound, and user voice), the processor (160) can calculate sleep intention information as two points. Also, if there is one type of sound included in the environmental sensing information (e.g., washing machine), the processor (160) can calculate sleep intention information as six points.

That is, the processor (160) can obtain sleep intention information related to how much the user intends to sleep based on the number of types of sounds included in the environmental sensing information. For example, the more types of sounds are identified, the lower the sleep intention information (i.e., low-score sleep intention information) that the user has is output.

The specific numerical description of the type of sound and sleep intention information included in the aforementioned environmental sensing information is only an example, and the present invention is not limited thereto. In addition, although the operation of calculating sleep intention information has been described as the operation of the processor (160), the above-described operation may be performed by a processor equipped in another electronic device disclosed in the present invention (e.g., a processor of a user terminal (10) or an external server (20), etc.).

Obtaining sleep intention information using the intention score table method

In addition, in the embodiment, the processor (160) may pre-match different intent scores to each of the plurality of sound information to generate or record an intent score table. For example, an intent score of 2 points may be matched to the first sound information related to a washing machine, an intent score of 5 points may be pre-matched to the second sound information related to the sound of a humidifier, and an intent score of 1 point may be matched to the third sound information related to a voice. The processor (160) may pre-match a relatively high intent score to sound information related to the user's sleep (e.g., sounds generated by the user's activities, such as vacuum cleaner, dishwashing, and voice sounds) and may pre-match a relatively low intent score to sound information unrelated to the user's sleep (e.g., sounds unrelated to the user's activities, such as vehicle noise, rain sounds) to generate an intent score table. The specific numerical description of the intent scores matched to each of the aforementioned sound information is merely an example, and the present invention is not limited thereto.

According to the present invention, the processor (160) can obtain sleep intention information based on the environmental sensing information and the intention score table. Specifically, the processor (160) can record an intention score matched to the identified sound in response to a time point when at least one of a plurality of sounds included in the intention score table is identified in the environmental sensing information. For a specific example, when a vacuum cleaner sound is identified in response to a first time point in a process of obtaining environmental sensing information in real time, the processor (160) can record two intention scores matched to the corresponding vacuum cleaner sound by matching them to the first time point. In the process of obtaining environmental sensing information, the processor (160) can record an intention score matched to the identified sound by matching it to the corresponding time point whenever each of various sounds is identified.

In an embodiment, the processor (160) may obtain sleep intention information based on the sum of intention scores obtained during a predetermined time period (e.g., 10 minutes). For example, the higher the intention score obtained during 10 minutes, the higher the sleep intention information may be obtained, and the lower the intention score obtained during 10 minutes, the lower the sleep intention information may be obtained. The specific numerical description of the predetermined time period described above is only an example, and the present invention is not limited thereto.

That is, the processor (160) can obtain sleep intention information related to how much the user intends to sleep based on the characteristics of the sound included in the environmental sensing information. For example, the more sound related to the user's activity is identified, the lower the user's sleep intention can be output (i.e., low-score sleep intention information).

Meanwhile, although the above operation has been described as an operation of a processor (160), the above operation may also be performed by a processor equipped in another electronic device disclosed in the present invention (e.g., a processor of a user terminal (10) or an external server (20), etc.).

Sleep event information

Sleep events according to one embodiment of the present invention include various events that may occur during sleep, such as snoring, sleep breathing (e.g., including information related to sleep apnea), and teeth grinding.

According to one embodiment of the present invention, it is possible to generate sleep event information indicating that a predetermined sleep event has occurred, or sleep event probability information indicating a probability of determining that a predetermined sleep event has occurred. Hereinafter, sleep respiration information, which is an example of sleep event information, will be described.

Figure 18 is a graph verifying the performance of a sleep analysis method according to the present invention, and is a drawing comparing the results of polysomnography (PSG) with the results of analysis using an AI algorithm according to the present invention in relation to sleep apnea and hypopnea.

The probability graph shown at the bottom of Figure 18 shows the probability of which of the two diseases (sleep apnea, hypopnea) a person belongs to in 30-second units when predicting sleep disorders by inputting user sleep sound information.

Among the three graphs shown in Fig. 18, the first graph shown in the AI result can be obtained by determining the disease with the highest probability from the probability graph shown below it.

Using sleep analysis according to the present invention, as shown in Fig. 18, the sleep state information obtained according to the present invention showed performance very consistent with polysomnography. In addition, it showed performance that included more precise analysis information related to apnea and hypopnea.

According to the present invention, the point where sleep disorders (sleep apnea, sleep hyperventilation, sleep hypoventilation) occur can be identified by analyzing the user's sleep analysis in real time. If stimulation (tactile, auditory, olfactory stimulation, etc.) is provided to the user at the moment when the sleep disorder occurs, the sleep disorder can be temporarily alleviated. That is, according to the present invention, the user's sleep disorder can be stopped and the frequency of sleep disorders can be reduced based on accurate event detection related to sleep disorders.

According to one embodiment of the present invention, a probability graph may indicate the probability of which of two diseases (sleep apnea, hypopnea) a user belongs to in units of 30 seconds when predicting a sleep disorder by inputting user sleep sound information, but is not limited to 30 seconds.

Sleep analysis model

FIG. 15a and FIG. 15b are drawings for explaining the overall structure of a sleep analysis model according to one embodiment of the present invention.

According to one embodiment of the present invention, the sleep analysis model may include a feature extraction model that extracts one or more features for each predetermined epoch, and a feature classification model that classifies each of the features extracted through the feature extraction model into one or more sleep stages to generate sleep stage information.

In the present invention, sleep state information can be obtained through a sleep analysis model that analyzes the user's sleep stage based on acoustic information (sleep acoustic information).

In the present invention, since the sleep sound information (SS) is a sound related to breathing and body movement acquired during the user's sleep time, it can be a very small sound, and accordingly, the present invention can perform an analysis of the sound by converting the sleep sound information (SS) into a spectrogram (SP) as described above. In this case, since the spectrogram (SP) includes information showing how the frequency spectrum of the sound changes over time, it is possible to easily identify a breathing or movement pattern related to a relatively small sound, and thus the efficiency of the analysis can be improved. Specifically, it may be difficult to predict whether the state is at least one of an awake state, a REM sleep state, a light sleep state, and a deep sleep state based only on a change in the energy level of the sleep sound information, but by converting the sleep sound information into a spectrogram, it is possible to easily detect a change in the spectrum of each frequency, and thus analysis corresponding to a small sound (e.g., breathing and body movement) can be made possible.

According to the present invention, the processor (160) can process a spectrogram (SP) as input to a sleep analysis model to obtain sleep state information. Here, the sleep analysis model is a model for obtaining sleep state information related to changes in the user's sleep stage, and can output sleep state information by inputting sleep sound information obtained during the user's sleep. In an embodiment, the sleep analysis model can include a neural network model configured through one or more network functions.

According to one embodiment of the present invention, a sleep analysis model (symbol A) utilizing deep learning for analyzing a user's sleep disclosed in FIG. 15b can perform sleep information inference (symbol E) through a feature extraction model (symbol B), an intermediate layer (symbol C), and a feature classification model (symbol D). The sleep analysis model (symbol A) utilizing deep learning configured through the feature extraction model (symbol B), the intermediate layer (symbol C), and the feature classification model (symbol D) performs both time-series feature learning and feature learning for multiple images, and the sleep analysis model (symbol A) utilizing deep learning learned through this can infer sleep stages for the entire sleep time and infer sleep events occurring in real time.

Network functions and neural networks

FIG. 11 is a schematic diagram showing one or more network functions for performing a sleep analysis method according to the present invention.

According to the present invention, a sleep analysis model is composed of one or more network functions, and the one or more network functions may be composed of a set of interconnected computational units, which may generally be referred to as 'nodes'. These 'nodes' may also be referred to as 'neurons'. The one or more network functions are composed of at least one node. The nodes (or neurons) constituting the one or more network functions may be interconnected by one or more 'links'.

According to the present invention, in a neural network, one or more nodes connected through a link can relatively form a relationship between input nodes and output nodes. The concept of input nodes and output nodes is relative, and any node in an output node relationship with respect to one node can be in an input node relationship with respect to another node, and vice versa. As described above, the input node-to-output node relationship can be created based on a link. One or more output nodes can be connected to one input node through a link, and vice versa.

In a relationship between input nodes and output nodes connected through a single link, the value of the output node can be determined based on the data input to the input node. Here, the node interconnecting the input node and the output node can have a weight. The weight can be variable and can be varied by a user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node through each link, the output node can determine the output node value based on the values input to the input nodes connected to the output node and the weight set for the link corresponding to each input node.

As described above, a neural network is a network in which one or more nodes are interconnected through one or more links to form input node and output node relationships within the neural network. Depending on the number of nodes and links within the neural network, the relationship between the nodes and links, and the weight values assigned to each link, the characteristics of the neural network can be determined. For example, if there are two neural networks with the same number of nodes and links and different weight values between the links, the two neural networks can be recognized as being different from each other.

Some of the nodes that make up the neural network can form a layer based on the distances from the initial input node. For example, a set of nodes that are n distances from the initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links that must be passed through to reach the node from the initial input node. However, this definition of a layer is arbitrary for explanation, and the order of a layer in the neural network can be defined in a different way than described above. For example, a layer of nodes can also be defined by the distance from the final output node.

The initial input node may refer to one or more nodes to which data is directly input without going through a link in a relationship with other nodes in the neural network. Or, in a neural network, in a relationship between nodes based on a link, it may refer to nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in a relationship with other nodes in the neural network. In addition, the hidden node may refer to nodes that constitute the neural network other than the initial input node and the final output node. The neural network according to one embodiment of the present invention may be a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the hidden layer close to the output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer.

A neural network may include one or more hidden layers. The hidden nodes of the hidden layers may take as input the output of the previous layer and the output of the surrounding hidden nodes. The number of hidden nodes for each hidden layer may be the same or different. The number of nodes of the input layer may be determined based on the number of data fields of the input data and may be the same as or different from the number of hidden nodes. The input data input to the input layer may be operated by the hidden nodes of the hidden layer and may be output by the fully connected layer (FCL), which is the output layer.

A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to input and output layers. Using a deep neural network, you can identify latent structures of data. That is, you can identify latent structures of photos, text, videos, voices, and music (for example, what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the voice are, etc.).

Deep neural networks may include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, generative adversarial networks (GANs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), Q networks, U networks, Siamese networks, transformers, vision transformers (ViTs), mobile vision transformers (Mobile ViTs), etc. The description of the above-described deep neural networks is only an example and the present invention is not limited thereto.

In the present invention, the network function may include an autoencoder. The autoencoder may be a type of artificial neural network for outputting output data similar to input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be arranged between input and output layers.

The number of nodes in each layer can be reduced from the number of nodes in the input layer to an intermediate layer called the bottleneck layer (encoding), and then expanded symmetrically from the bottleneck layer to the output layer (symmetrical to the input layer).

Nodes of the dimension reduction layer and the dimension restoration layer may or may not be symmetrical. The autoencoder can perform nonlinear dimension reduction. The number of input layers and output layers can correspond to the number of sensors remaining after preprocessing of the input data.

In the autoencoder structure, the number of nodes in the hidden layer included in the encoder can have a structure that decreases as it gets farther away from the input layer. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and the decoder) is too small, a sufficient amount of information may not be transmitted, so it may be maintained at a certain number or more (for example, more than half of the input layer, etc.).

A neural network can be trained in at least one of supervised learning, unsupervised learning, and semi-supervised learning. The training of a neural network is to minimize the error of the output. In the training of a neural network, training data is repeatedly input into the neural network, the output of the neural network and the target error for the training data are calculated, and the error of the neural network is backpropagated from the output layer of the neural network to the input layer in the direction of reducing the error, thereby updating the weights of each node of the neural network. In the case of supervised learning, training data in which the correct answer is labeled for each training data is used (i.e., labeled training data), and in the case of unsupervised learning, the correct answer may not be labeled for each training data. That is, for example, in the case of supervised learning for data classification, the training data may be data in which each training data is labeled with a category. Labeled training data is input into the neural network, and the error can be calculated by comparing the output (categories) of the neural network with the labels of the training data.

As another example, in the case of unsupervised learning for data classification, the input learning data can be compared with the output of the neural network to calculate the error. The calculated error is backpropagated in the backward direction (i.e., from the output layer to the input layer) in the neural network, and the connection weights of each node in each layer of the neural network can be updated according to the backpropagation. The amount of change in the connection weights of each node to be updated can be determined according to the learning rate. The calculation of the neural network for the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of repetitions of the learning cycle of the neural network. For example, a high learning rate can be used in the early stage of learning of the neural network to allow the neural network to quickly acquire a certain level of performance, thereby increasing efficiency, and a low learning rate can be used in the later stage of learning to increase accuracy.

In learning neural networks, learning data can generally be a subset of real data (i.e., data to be processed using the learned neural network), and therefore, there may be a learning cycle in which errors in the learning data decrease but errors in the real data increase. Overfitting is a phenomenon in which excessive learning on the learning data increases errors in the real data. For example, a neural network that learned cats by showing a yellow cat may not recognize cats when it sees a cat that is not yellow, which may be a type of overfitting. Overfitting can act as a cause of increasing errors in AI algorithms. Various optimization methods can be used to prevent overfitting. To prevent overfitting, methods such as increasing the learning data, regularization, and dropout, which omits some nodes of the network during the learning process, can be applied.

Throughout this specification, the terms computational model, neural network, network function, and neural network may be used interchangeably. (Hereinafter, they are collectively referred to as neural network.) The data structure may include a neural network. And the data structure including the neural network may be stored in a computer-readable medium. The data structure including the neural network may also include data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for learning the neural network. The data structure including the neural network may include any of the components among the above-described configurations.

That is, a data structure including a neural network may be configured to include all or any combination of data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, loss functions for training the neural network, etc. In addition to the aforementioned configurations, a data structure including a neural network may include any other information that determines the characteristics of the neural network.

In addition, the data structure may include all forms of data used or generated in the computational process of the neural network, and is not limited to the foregoing. The computer-readable medium may include a computer-readable recording medium and/or a computer-readable transmission medium. The neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. The neural network is composed of at least one node.

Feature extraction model and feature classification model

FIG. 16 is a diagram for explaining a feature extraction model and a feature classification model according to one embodiment of the present invention.

The sleep analysis model used in the present invention may include a feature extraction model that extracts one or more features for each predetermined epoch, and a feature classification model that classifies each of the features extracted through the feature extraction model into one or more sleep stages to generate sleep state information. The feature extraction model may extract features related to breathing sounds, breathing patterns, and movement patterns by analyzing a time-series frequency pattern of a spectrogram (SP). In one embodiment, the feature extraction model may be configured through a part of a neural network model that is pre-learned through a learning data set.

The sleep analysis model used in the present invention may include a feature extraction model and a feature classification model. The feature extraction model may be a deep learning learning model based on a natural language processing model capable of learning the time-series correlation of given data. The feature classification model may be a learning model based on a natural language processing model capable of learning the time-series correlation of given data. Here, the deep learning learning model based on a natural language processing model capable of learning the time-series correlation may include, but is not limited to, Tarnsformer, ViT, MobileViT, and MobileViT2.

A learning data set according to one embodiment of the present invention may be composed of data in the frequency domain and a plurality of pieces of sleep state information corresponding to each piece of data.

Alternatively, a learning data set according to one embodiment of the present invention may be composed of a plurality of spectrograms and a plurality of sleep state information corresponding to each spectrogram.

Alternatively, a learning data set according to one embodiment of the present invention may be composed of a plurality of Mel spectrograms and a plurality of sleep state information corresponding to each Mel spectrogram.

For convenience of explanation, the configuration and execution of a sleep analysis model according to one embodiment of the present invention will be described in detail based on a data set of spectrograms. However, the learning data utilized in the sleep analysis model of the present invention is not limited to spectrograms, and information in the frequency domain, information including changes in frequency components of information in the time domain along the time axis, spectrograms, or mel spectrograms can be utilized as learning data.

Among the sleep analysis models according to the embodiment of the present invention, the feature extraction model can be pre-trained by a one-to-one proxy task that inputs one spectrogram and learns to predict sleep state information corresponding to one spectrogram. In the case of adopting a CNN deep learning model in the feature extraction model according to the embodiment of the present invention, learning may be performed by adopting a structure of a FC (Fully Connected Layer) or a FCN (Fully Connected Neural Network). In the case of adopting a MobileViTV2 deep learning model in the feature extraction model according to the embodiment of the present invention, learning may be performed by adopting a structure of an intermediate layer.

Among the sleep analysis models according to an embodiment of the present invention, a feature classification model can be trained to predict sleep state information of each spectrogram by inputting a plurality of continuous spectrograms, and to predict or classify overall sleep state information by analyzing a sequence of the plurality of continuous spectrograms.

In addition, according to an embodiment of the present invention, after pre-learning is performed through a One-to-one proxy task for a feature extraction model, fine tuning can be performed through a Many-to-Many task for the pre-learned feature extraction model and feature classification model. For example, a sequence of 40 consecutive spectrograms can be input into multiple feature extraction models learned through a One-to-one proxy task, and 20 pieces of sleep state information can be output, thereby inferring sleep stages. The specific numerical descriptions related to the number of spectrograms, the number of feature extraction models, and the number of sleep state information described above are merely examples, and the present invention is not limited thereto.

As described above, an inference model is generated to extract the user's sleep state and sleep stage through deep learning of environmental sensing information. Briefly, environmental sensing information including acoustic information, etc. is converted into a spectrogram, and an inference model is generated based on the spectrogram.

The inference model can be built into the graphical user interface generation device (100)/providing device (200), as described above.

Thereafter, environmental sensing information including user sound information acquired through the user terminal (10) is input to the corresponding inference model, and sleep state information and/or sleep stage information is output as a result value. At this time, learning and inference may be performed in the same subject, but learning and inference may be performed in separate subjects. That is, both learning and inference may be performed by the graphical user interface generation device (100) of Fig. 1a or the graphical user interface provision device (200) of Fig. 1b, and learning may be performed in the graphical user interface generation device (100) of Fig. 1a or the graphical user interface provision device (200) of Fig. 1b, but inference may be performed in the user terminal (10). Alternatively, at least one of learning and inference may be performed in an external server (20).

Alternatively, as in Fig. 2a, when the graphical user interface generating device (100) and the graphical user interface providing device (200) are implemented as a user terminal (10), both learning and inference may be performed by the user terminal (10).

Alternatively, as shown in FIG. 2b, when the present invention is implemented through a system composed of various electronic devices, learning or inference may be performed by at least one of the electronic devices illustrated in FIG. 2b. For example, learning or inference may be performed in an external server (20).

Hereinafter, a feature extraction model and a feature classification model according to an embodiment of the present invention will be described in detail.

Feature extraction model

According to the present invention, the feature extraction model can be composed of an independent deep learning model learned through a learning data set. The feature extraction model can be learned through a supervised learning or unsupervised learning method. The feature extraction model can be learned to output output data similar to the input data through the learning data set. In detail, only the core feature data (or feature) of the input spectrogram can be learned through the hidden layer. In this case, during the decoding process through the decoder, the output data of the hidden layer can be an approximation of the input data (i.e., the spectrogram) rather than a perfect copy value.

According to the present invention, each of a plurality of spectrograms included in a learning data set may be tagged with sleep state information. Each of the plurality of spectrograms may be input to a feature extraction model, and an output corresponding to each spectrogram may be stored by matching it with the tagged sleep state information. Specifically, when first learning data sets (i.e., a plurality of spectrograms) tagged with first sleep state information (e.g., shallow sleep) are input, features related to the output for the corresponding input may be stored by matching it with the first sleep state information. In an embodiment, one or more features related to the output may be displayed in a vector space. In this case, since the feature data output corresponding to each of the first learning data sets is an output through a spectrogram related to the first sleep stage, they may be located at a relatively close distance in the vector space. That is, learning may be performed so that the plurality of spectrograms output similar features corresponding to each sleep stage.

According to the present invention, a feature extraction model through the aforementioned learning process can extract features corresponding to a spectrogram when a spectrogram (e.g., a spectrogram converted corresponding to sleep sound information) is input.

In an embodiment, the processor (160) may process a spectrogram (SP) generated corresponding to the sleep sound information (SS) as an input of a feature extraction model to extract features. Here, since the sleep sound information (SS) is time-series data acquired in time series during the user's sleep, the processor (160) may divide the spectrogram (SP) into predetermined epochs. For example, the processor (160) may divide the spectrogram (SP) corresponding to the sleep sound information (SS) into 30-second units to obtain a plurality of spectrograms. For example, if the sleep sound information is acquired during the user's 7-hour (i.e., 420 minutes) sleep, the processor (160) may divide the spectrogram into 30-second units to obtain 840 spectrograms. The specific numerical descriptions of the above-mentioned sleep time, spectrogram division time unit, and division number are merely examples, and the present invention is not limited thereto.

According to the present invention, the processor (160) can process each of the divided plurality of spectrograms as an input of a feature extraction model to extract a plurality of features corresponding to each of the plurality of spectrograms. For example, when the number of the plurality of spectrograms is 840, the number of the plurality of features extracted by the feature extraction model corresponding thereto can also be 840. The specific numerical descriptions related to the above-described number of spectrograms and the plurality of features are only examples, and the present invention is not limited thereto.

Meanwhile, the feature extraction model according to the embodiment of the present invention may perform learning by a one-to-one proxy task. In addition, in the process of learning to extract sleep state information for one spectrogram, the feature extraction model may be learned to extract sleep state information by combining another NN (Neural Network).

According to an embodiment of the present invention, if learning is performed through a simple pre-learned Neural Network, the learning time of a feature extraction model can be shortened or the learning efficiency can be increased.

For example, according to one embodiment of the present invention, a spectrogram divided into 30-second units may be trained to output sleep state information by using the output vector as an input to a feature extraction model and using the output vector as an input to another NN.

Although sleep stages are predicted by using sleep sound information as input, sleep stage analysis and inference can be performed by using sleep sound information converted into information in the frequency domain, information including changes in frequency components of sleep sound information along the time axis, a spectrogram, or a melo spectrogram as input. Therefore, according to the embodiments of the present invention, sleep sound information is converted into information in the frequency domain, information including changes in frequency components of sleep sound information along the time axis, a spectrogram, or a melo spectrogram and used as input for a sleep analysis model. Therefore, unlike the existing sleep analysis model, sleep stages can be sensed or acquired in real time by analyzing the specificity of sleep patterns.

A graphical user interface that presents information about sleep.

According to the present invention, a graphical user interface may be provided that displays information about a user's sleep.

FIGS. 19A to 19C are diagrams showing aspects in which a graphical user interface according to embodiments of the present invention is displayed on various display units.

As illustrated in FIGS. 19a to 19c, according to embodiments of the present invention, a graphical user interface indicating information about a user's sleep may be displayed on an electronic device or user terminal (10) implemented with various types of displays.

Hereinafter, a graphical user interface including a graph representing information about sleep according to embodiments of the present invention will be described using drawings.

Graphical user interface including hypnogram

FIG. 4a is a diagram illustrating a graphical user interface including a hipnogram according to one embodiment of the present invention.

A graphical user interface including a hypnogram according to an embodiment of the present invention may display the phrase “sleep stage” together, as indicated by reference numeral 101.

According to one embodiment of the present invention, a hypnogram may include a shape corresponding to each sleep stage information, with the x-axis representing time (reference number 114) and the y-axis representing sleep stage information.

According to FIG. 4a, the sleep stage information according to one embodiment of the present invention may include a total of four stages, and may include multiple areas allocated to each sleep stage information. For example, it may be expressed by dividing into an area allocated to a wake stage (reference number 102), an area allocated to a REM sleep stage (reference number 103), an area allocated to a light sleep (or 'normal sleep') stage (reference number 104), and an area allocated to a deep sleep stage (reference number 105).

According to one embodiment of the present invention, the color of the shape corresponding to each sleep stage information may be expressed differently. For example, the color of the shape corresponding to the deep sleep stage may be a relatively darker color than other shapes (reference number 113). Alternatively, the color of the shape corresponding to the REM sleep stage may be a relatively brighter color than other shapes (reference number 111).

Alternatively, according to embodiments of the present invention, the colors of the shapes and backgrounds corresponding to each sleep stage information may be colors expressed at different locations in the color space.

For example, according to one embodiment of the present invention, the background may be expressed in a black color series, a shape corresponding to a deep sleep stage may be expressed in a relatively dark blue color series, a shape corresponding to a light sleep or normal sleep stage may be expressed in a relatively light blue color series (reference number 112), a shape corresponding to a REM sleep stage may be expressed in a purple color series, and a shape corresponding to a wake-up stage may be expressed in a yellow color series.

According to embodiments of the present invention, shapes corresponding to sleep stage information may be shapes expressed discretely from each other.

In the present invention, the term 'discretely expressed figure' may mean that each figure is not connected to each other by a solid line or a dotted line, etc.

In addition, in accordance with one embodiment of the present invention, since shapes are expressed discretely, unlike in the prior art, there may be cases where shapes corresponding to different sleep stages share only one intersection point.

Additionally, according to one embodiment of the present invention, the fact that a shape is discretely expressed may include the meaning that it includes a plurality of shapes, each of which corresponds to a plurality of sleep stages, and at least one shape among the plurality of shapes is isolated from the remaining shapes.

Figures 20a to 20e are diagrams showing hypnogram graphs of sleep stage information expressed in a conventional sleep measurement interface.

As shown in FIGS. 20a to 20e, in a hypnogram graph included in a conventional sleep measurement interface, there were cases where shapes representing sleep stages were displayed so that each shape was connected to another shape by a line, such as a dotted line or solid line, even if the shapes were separated from each other.

For example, as can be seen from FIGS. 20A to 20E, in conventional techniques, when expressing a graph representing sleep stages, there were cases where, when a change occurred between the first sleep stage and the second sleep stage, it was expressed continuously rather than discretely.

In addition, as illustrated in Fig. 20e, in order to express that there is continuity between the shapes even though the intersections of the shapes do not exist with each other, there were cases where a line (dotted line or solid line) was connected between the point indicating the end of the first shape and the point indicating the start of the second shape.

However, there was a concern that the representation of hypnograms through these continuous graphs could lead to the misunderstanding that another sleep stage must be passed through in the process of transitioning from the first sleep stage to the second sleep stage. In sleep stage analysis, the transition pattern between sleep stages is important, but the representation of continuous hypnograms may have difficulty accurately representing this transition pattern between sleep stages.

According to the present invention, unlike the prior art, when a shape corresponding to each sleep stage information is expressed discretely according to one embodiment of the present invention, there is an effect of reducing the misunderstanding that a different sleep stage must be passed through in the process of transitioning from the first sleep stage to the second sleep stage.

According to the present invention, in the process of transitioning from the first sleep stage to the second sleep stage, it is not necessary to go through another sleep stage, so when a shape corresponding to each sleep stage information is discretely expressed, there is an advantage in that the transition between sleep stages can be intuitively understood.

For example, referring to the figure indicated by reference numeral 110 in Fig. 4a (a figure corresponding to the awakening stage), the transition occurs from the light sleep (normal sleep) stage to the awakening stage and then back to the light sleep (normal sleep) stage. If this transition or change in sleep stage is expressed as a 'continuous' hypnogram graph, it may cause the misunderstanding that the REM sleep stage must be passed when the sleep stage is switched. However, if it is expressed as a 'discrete' hypnogram graph according to the present invention, such misunderstanding can be reduced.

In addition, when the shape corresponding to each sleep stage information is discretely expressed in the hypnogram, the frequency of each sleep stage is clearly expressed, so the frequency of the sleep stage belonging to a specific section can be directly confirmed, and it can be easy to grasp the relative proportion of each sleep stage. This can be helpful in identifying the characteristics or abnormalities of sleep patterns.

Meanwhile, according to one embodiment of the present invention, a shape corresponding to sleep stage information may be expressed as at least one shape among a trapezoid, an isosceles trapezoid, a kite, a parallelogram, a rhombus, a rectangle, a square, and other general quadrilaterals. Preferably, it may be expressed as a rectangle.

As illustrated in FIG. 4A, multiple sleep stage information may be expressed as multiple rectangles. At this time, at least one of the multiple rectangles may be isolated from the remaining rectangles, such as the rectangle indicated by reference numeral 110 in FIG. 4A. In the embodiment illustrated in FIG. 4A, the rectangle (110) corresponding to the wake-up stage of the sleep stage information is isolated from the remaining rectangles.

When performing sleep analysis, the awakening stage may appear at least once during the sleep period. When expressing sleep stage information as a hypnogram according to embodiments of the present invention, a rectangle representing the awakening stage can be displayed separately from rectangles representing other sleep stages. In this display, there is an advantage in that sleep can be analyzed by clearly distinguishing the occurrence of the awakening stage during sleep. According to the interface of the prior art, when information on the awakening stage during sleep is frequently obtained, since each sleep stage and the awakening stage during sleep are connected by a line (solid line or dotted line), there was a concern that the hypnogram would be expressed relatively unclearly, and that the occurrence of the awakening stage would not be clearly distinguished.

However, when representing sleep stage information as a discrete and/or separately expressed figure according to the present invention, the occurrence frequency of sleep wake-up stages can be clearly confirmed, thereby resolving the unclear errors of the prior art.

Additionally, according to one embodiment of the present invention, areas allocated to the wake stage, REM sleep stage, light sleep stage, and deep sleep stage may be displayed in order from top to bottom (see reference numbers 102 to 105).

Meanwhile, as shown in FIG. 20c or FIG. 20d, even if a shape representing a wake-up stage was displayed as if it were separated from shapes representing other sleep stages, there was a problem in that it was difficult to clearly understand which sleep stage occurred at a given time because shapes corresponding to sleep stages were displayed continuously in areas allocated to other sleep stages.

For example, referring to FIG. 20c, although multiple shapes representing the wake stage during the entire sleep period are displayed as if they are separated from shapes corresponding to other sleep stages, there is a problem in that it is unclear whether the person is sleeping or awake at that time because graphs are also displayed in areas representing the REM sleep stage, the Light sleep stage, or the Deep sleep stage at that time.

On the other hand, in an interface according to embodiments of the present invention, if at least one shape among shapes corresponding to multiple sleep stages is isolated from the remaining shapes, there is an effect of making it possible to clearly understand which sleep stage has occurred at a given time by ensuring that no shape exists in an area allocated to another sleep stage.

In addition, in the interface according to embodiments of the present invention, shapes corresponding to multiple sleep stages are expressed as shapes of the same shape (e.g., rectangles), and at least one of the shapes can be isolated from the remaining shapes. In this embodiment, there is an effect that it is possible to clearly understand which sleep stage has occurred at a given time by making it so that no shape exists in an area allocated to another sleep stage expressed as a shape of the same shape.

Additionally, among the shapes corresponding to a plurality of sleep stages included in the interface according to embodiments of the present invention, at least one shape corresponding to a sleep stage may have the same shape as shapes corresponding to other sleep stages.

In addition, according to embodiments of the present invention, a shape corresponding to at least one sleep stage may be displayed separately from other shapes corresponding to the same sleep stage. In addition, a shape corresponding to at least one sleep stage may be displayed separately from shapes corresponding to other sleep stages.

According to an embodiment of the present invention, when a wake-up stage occurs, as shown in reference numeral 110 in FIG. 4A, a shape corresponding to the sleep stage is displayed only in the reference numeral 102 region at that time, and is not displayed in the regions allocated to the remaining sleep stages (reference numerals 103 to 105), so that a user can clearly understand which sleep stage (specifically, the wake-up stage) has occurred at that time just by looking at the hypnogram graph according to the present invention.

In addition, as illustrated in FIG. 4a, according to one embodiment of the present invention, when a word indicating a light sleep stage is displayed in Korean, it may be displayed as 'light sleep' or 'normal sleep'. Users who are not experts in sleep stages may misunderstand that they did not sleep properly during the sleep period due to the word 'light sleep', but in contrast, when it is displayed as the word 'normal sleep', such misunderstanding is likely to be reduced (reference number 104).

According to one embodiment of the present invention, a shape corresponding to each sleep stage can be displayed only in a plurality of areas allocated to each sleep stage information (see reference numbers 102 to 105). Through this display method, sleep stage information during a sleep period can be clearly distinguished and expressed, so that a user can distinguish sleep stage information during a sleep period and accurately understand a graph of sleep stage information.

According to one embodiment of the present invention, there may be a boundary between a plurality of areas allocated to each sleep stage information, and this boundary may be indicated by a dotted line or a solid line (reference number 108). The line indicating the boundary may be a straight line or a curved line.

In addition, according to one embodiment of the present invention, lines indicating boundaries of multiple areas allocated to each sleep stage information may be displayed with different colors or brightnesses. Specifically, the color of the boundary line separating the wake stage area and the remaining sleep stage areas may be displayed relatively brightly.

According to one embodiment of the present invention, each of the plurality of regions assigned to each piece of sleep stage information may have a height assigned thereto.

In addition, according to one embodiment of the present invention, the heights assigned to the plurality of areas assigned to each of the sleep stage information may be the same or different. Specifically, the height assigned to the area assigned to the wake stage information may be different from the heights assigned to other areas (see reference numeral 102).

According to one embodiment of the present invention, there may be a pitch assigned between a plurality of regions assigned to each sleep stage information.

According to one embodiment of the present invention, a shape corresponding to each sleep stage information can be discretely expressed based on a pitch assigned between a plurality of regions assigned to each sleep stage information. By this discrete expression, a user can distinguish sleep stage information during a sleep period and accurately understand a graph for sleep stage information.

As illustrated in FIG. 4A, according to one embodiment of the present invention, at least one piece of information among the time of falling asleep and the time of waking up may be displayed together in the hypnogram. In this case, at least one piece of information among the time of falling asleep and the time of waking up may be displayed as time information on the x-axis (reference numbers 106 and 107). In this case, when at least one piece of information among the time of falling asleep and the time of waking up is displayed together in the hypnogram, the time interval from the time of falling asleep to the time of waking up can be easily identified by looking at only the hypnogram. According to one embodiment of the present invention, when at least one piece of information among the time of falling asleep and the time of waking up is displayed as time information on the x-axis, it may be displayed by connecting a shape and a line corresponding to each piece of sleep stage information. According to embodiments of the present invention, a line connecting at least one piece of information among the time of falling asleep and the time of waking up to a shape corresponding to the sleep stage information may be displayed as a solid line or a dotted line, a straight line, or a curved line.

In addition, according to one embodiment of the present invention, as illustrated in FIG. 4a, comment information based on sleep stage information or sleep state information may be displayed together with a hypnogram graph representing sleep stages (reference number 109).

FIG. 4d is a diagram illustrating a graphical user interface including a hipnogram according to another embodiment of the present invention.

According to one embodiment of the present invention, if sleep measurement has not been performed at all after the start of sleep measurement, only a shape corresponding to the waking stage may be displayed on the hypnogram, as shown in FIG. 4d.

According to one embodiment of the present invention, a hypnogram graph can be generated in real time during a sleep period. According to the present invention, sleep information including a user's sleep sound is acquired in real time, and the acquired sleep sound information can be immediately converted into a spectrogram. The converted spectrogram can be used as an input for a sleep analysis model to immediately analyze sleep stages, and accordingly, a hypnogram graph representing sleep stage information can be generated in real time simultaneously with the analysis of sleep stages.

Graph showing the percentage of time in each sleep stage

FIG. 4b is a graph showing the time ratio of each sleep stage measured according to one embodiment of the present invention.

A graphical user interface including a graph showing the time ratio of each sleep stage according to an embodiment of the present invention may be accompanied by the phrase “My sleep at a glance,” as indicated by reference numeral 201.

The time ratio of each sleep stage according to the present invention can be calculated as the ratio of the time corresponding to each sleep stage to the total sleep period. Specifically, the total sleep time can be calculated as the time from the time of falling asleep to the time of waking up. The time corresponding to each sleep stage can be calculated based on the sleep stage inferred by the artificial intelligence model according to the embodiment of the present invention.

For example, as illustrated in reference numeral 203 of FIG. 4B, if the total sleep time from the time of falling asleep to the time of waking up is 7 hours and 22 minutes, the period inferred as the general sleep (light sleep) stage is 4 hours and 34 minutes, the period inferred as the REM sleep stage is 1 hour and 37 minutes, the period inferred as the deep sleep stage is 58 minutes, and the period inferred as the wake-up stage is 13 minutes, the ratio of each sleep stage can be displayed as a percentile value obtained by dividing the time corresponding to each period by the total sleep time. However, the numerical values for the specific times described above are merely examples, and the present invention is not limited thereto.

In addition, in a graphical user interface according to an embodiment of the present invention, time information corresponding to each sleep stage and the ratio of each sleep stage can be displayed side by side in parallel on the left and right (reference number 202). In addition, as illustrated in reference numbers 204 to 207 of FIG. 4b, the time ratio corresponding to each sleep stage can be visually represented using a predetermined shape.

Here, the color of the shape corresponding to each sleep stage can be displayed in the same way as the color of the shape corresponding to the sleep stage information shown in the hypnogram of Fig. 4a.

In addition, according to one embodiment of the present invention, as illustrated in reference numerals 202 and 204 to 207 of FIG. 4b, the shapes corresponding to each sleep stage may be displayed in order of increasing time ratios corresponding to each sleep stage; however, according to other embodiments, the order of the displayed sleep stages may be changed.

For example, the deep sleep stage may be configured to be displayed first and the wake stage may be displayed last. Alternatively, the wake stage may be configured to be displayed first and the deep sleep stage may be configured to be displayed last. Alternatively, the REM sleep stage may be configured to be displayed first and the normal sleep stage may be configured to be displayed first. This order may be selected directly by the user or may be automatically configured by an algorithm. The above-described order is merely an example, and the present invention is not limited thereto.

Respiratory stability graph

Figure 4c is a drawing showing a respiratory stability graph according to an embodiment of the present invention.

A graphical user interface including a respiratory stability graph according to an embodiment of the present invention may display the phrase “respiratory stability” together, as indicated by reference numeral 301.

Through sleep analysis according to one embodiment of the present invention, sleep disorders (e.g., sleep apnea) and their underlying causes (e.g., snoring) can be predicted.

According to an embodiment of the present invention, the stability of breathing can be determined by criteria such as breathing cycle, breathing frequency fluctuation, and breathing pattern.

If your breathing pattern is irregular or changes suddenly during sleep, your breathing may be considered unstable.

Alternatively, breathing may be considered unstable if the frequency of breathing fluctuates significantly or shows irregular changes during sleep.

According to one embodiment of the present invention, when breathing is determined to be unstable in this way, the breathing stability of the corresponding period can be expressed as 'breathing unstable' (reference number 303). In addition, the breathing stability of a section that does not correspond to 'breathing unstable' can be expressed as 'breathing stable' (reference number 302).

As indicated by reference numerals 302 and 303 in Fig. 4c, the breathing stability can be represented as a graph over time. In this case, if the breathing stability of sleep is judged to be unstable during sleep, the phrase “unstable” can also be displayed (reference numeral 306).

The apnea-hypopnea index (AHI) is calculated by dividing the sum of the frequency of apneas and hypopneas during sleep recorded through sleep analysis such as polysomnography by the total sleep time. An AHI of less than 5 is classified as normal, 5 to 15 as mild, 15 to 30 as moderate, and 30 or more as severe.

According to one embodiment of the present invention, an AHI value can be determined through sleep analysis using an artificial intelligence model, and a breathing stability ratio can be calculated and displayed on an interface based on the determined AHI value (reference numbers 304 and 305).

In addition, according to one embodiment of the present invention, the color of the shape corresponding to the point in time when breathing is judged to be unstable may be displayed brighter. In this case, there is an advantage in that the occurrence of an event that is problematic from the perspective of breathing stability can be better identified.

FIG. 21 is a diagram illustrating a graphical user interface that displays in parallel a graph representing sleep stage information and a graph representing sleep event information according to one embodiment of the present invention.

According to one embodiment of the present invention, as shown in Fig. 21, a hypnogram graph representing sleep stage information and a graph representing breathing stability may be displayed together in parallel, above and below, on the same time axis. When displayed in this way, a user can see at a glance how breathing stability was in which sleep stage, which can be effective in analyzing sleep.

For example, if a breathing instability event occurs during the REM sleep stage, it can be interpreted as being more serious. If a hypnogram graph representing sleep stage information and a graph representing breathing stability are displayed side by side on the same time axis, as shown above, the occurrence of such an event can be easily identified at a glance, making it easier to interpret and/or judge sleep information.

In addition, when a hypnogram graph representing sleep stage information and a graph representing respiratory stability are displayed together in parallel above and below on the same time axis according to embodiments of the present invention, there is an advantage in that such information can be utilized for providing various services, or such graphic image information can be utilized for training a deep learning model.

In addition, there is an advantage in that the correlation between sleep information identified from graphs displayed in parallel as above can be utilized to generate sleep evaluation information.

FIG. 4e is a diagram illustrating a graphical user interface including a description display for respiratory instability according to one embodiment of the present invention.

A graphical user interface including a description display of respiratory instability according to an embodiment of the present invention may display the phrase “What is respiratory instability?” as indicated by reference numeral 401.

As illustrated in FIG. 4e, a graphical user interface may display a description of respiratory instability (reference numeral 402). Additionally, when a user clicks on a portion allocated to a predetermined area indicated by reference numeral 403, the user may be directed to an external website describing respiratory instability or to a screen displaying information with a more detailed description.

As illustrated in FIG. 4e, when a description of respiratory instability is displayed according to one embodiment of the present invention, a graphical user interface located in the background portion may be expressed relatively dark (reference numeral 404).

Screen showing sleep statistics

FIGS. 5A and 5B are diagrams illustrating a graphical user interface including statistical information of sleep state information according to embodiments of the present invention.

A graphical user interface including statistical information of sleep state information according to an embodiment of the present invention may display the phrase “sleep statistics” together, as indicated by reference numeral 501.

According to an embodiment of the present invention, daily sleep time can be expressed as a bar graph. Specifically, the date on which sleep status information is acquired can be displayed on the x-axis (reference number 503), and the sleep time value can be displayed on the y-axis (reference number 504).

According to an embodiment of the present invention, the sleep time calculated based on the sleep state information acquired on the day corresponding to the date on the x-axis can be displayed in the form of a bar graph (reference number 502).

In addition, according to one embodiment of the present invention, information on the day of the week on which the corresponding sleep state information was acquired may be displayed on the upper portion of the bar graph (reference number 505).

In addition, according to one embodiment of the present invention, information indicating an average of actual sleeping time may be displayed at the bottom of the bar graph (reference numbers 506 to 508). Specifically, at least one of an average of sleeping time acquired over a predetermined period, an average of sleeping time measured during a weekday, or an average of sleeping time measured during a weekend may be displayed together.

Additionally, according to one embodiment of the present invention, the average of sleep time obtained over a given period of time, the average of sleep time measured during a weekday, or the average of sleep time measured during a weekend may all be displayed together.

In addition, according to one embodiment of the present invention, a graphical user interface may be provided that expresses the average of sleep time measured during a weekday and the average of sleep time measured during a weekend as a line between the average values, as indicated by reference numeral 509, in order to compare the average of sleep time measured during a weekday and the average of sleep time measured during a weekend with the average of sleep time measured during a predetermined period. In this case, the line for dividing may be a dotted line or a solid line, and may be expressed as a curve or a straight line.

According to one embodiment of the present invention, if sleep state information is acquired only on weekdays and sleep analysis is not performed on weekends, and thus sleep state information is not acquired, the graphical user interface may be displayed as illustrated in reference numeral 508 of FIG. 5A. In addition, in this case, bar graphs corresponding to the 11th (Saturday) and the 12th (Sunday) of FIG. 5A may not be formed. The specific description of the days of the week and dates described above are merely examples and are not limited thereto. For example, if sleep analysis is not performed on the 9th (Thursday), a bar graph corresponding to the 9th (Thursday) may not be formed.

In addition, as illustrated in FIG. 5b, according to one embodiment of the present invention, when a specific bar graph among the bar graphs is clicked, sleep time information acquired on a date corresponding to the bar graph may be displayed (reference number 513). In this case, the corresponding bar graph may be displayed brightly (reference number 512), and other bar graphs may be displayed relatively darkly.

In addition, as shown in FIGS. 5A and 5B, average information on sleep time acquired over a predetermined period of time may be displayed together with a bar graph. For example, if the average sleep time acquired over a predetermined period of time is 6 hours and 26 minutes, the average sleep time may be displayed as a shape such as a line (e.g., a dotted line) at a position corresponding to 6 hours and 26 minutes on the y-axis of the bar graph (reference number 510).

A graphical user interface including statistical information of sleep state information according to one embodiment of the present invention may display the phrase “sleep statistics” together with information on the period during which the sleep state information was acquired (e.g., period information “2022.6.6 ~ 6.12”), as indicated by reference numeral 511.

In addition, a graphical user interface including statistical information of sleep state information according to one embodiment of the present invention may display a brief description of the graph (e.g., a description such as “I slept this much this week” or “The more consistent the height of the sleep bar, the better”) as indicated at reference numeral 511. The brief description of the graph described above is merely an example, and the present invention is not limited thereto.

Sleep status information obtained over a week

FIGS. 6A and 6B are diagrams illustrating a graphical user interface including sleep status information acquired over a week according to embodiments of the present invention.

As illustrated in FIGS. 6A and 6B, sleep status information acquired over a week according to embodiments of the present invention may be expressed as a bar graph in which the x-axis represents the day of the week (reference number 601) and the y-axis represents time information. Here, the time information of the y-axis may be expressed in 2-hour units (reference number 602). In addition, a line (solid line or dotted line) corresponding to each time of the y-axis may also be displayed (reference number 603).

As illustrated in FIG. 6a, according to one embodiment of the present invention, the bar graph may be expressed entirely in a single color (reference number 604).

In addition, as illustrated in FIG. 6b, according to one embodiment of the present invention, a bar graph may display sections corresponding to sleep stages of REM sleep, deep sleep, light sleep, and wakefulness separately (reference numbers 605 to 608). Specifically, when the colors of shapes corresponding to each sleep stage information in the hypnogram graph are expressed differently, the color of the shape corresponding to each sleep stage information in the bar graph of the corresponding date may be displayed in proportion to the corresponding sleep stage.

In addition, in the case where the parts corresponding to each sleep stage are displayed separately according to one embodiment of the present invention, the part corresponding to the wake stage may be arranged at the topmost part of the bar graph (reference number 608), and the part corresponding to the deep sleep may be arranged at the bottommost part of the bar graph (reference number 605). Alternatively, the part corresponding to the sleep stage with a higher time ratio may be arranged at the bottommost part of the bar graph, and the part corresponding to the sleep stage with a lower time ratio may be arranged at the topmost part of the bar graph. Meanwhile, this arrangement order is merely an example, and the present invention is not limited thereto. For example, the part corresponding to the deep sleep stage, the part corresponding to the light sleep stage, the part corresponding to the REM sleep stage, and the part corresponding to the wake stage may be arranged in that order from the bottom.

Sleep status information acquired over a specified period of time

FIGS. 7A and 7B are diagrams illustrating a graphical user interface including sleep state information acquired over a predetermined period of time according to embodiments of the present invention.

As illustrated in FIGS. 7a and 7b, sleep status information acquired over a week according to embodiments of the present invention can be expressed as a bar graph in which the x-axis represents the date (reference numbers 701a and 701b) and the y-axis represents time information.

Here, referring to reference numbers 701a and 701b of FIG. 7a, when obtaining sleep state information through sleep analysis performed from the date displayed at the top to the date displayed at the bottom, a bar graph can be displayed based on the falling-at-sleep time information and the waking-up time information included in the obtained sleep state information. For example, if it is determined that the person fell asleep after the evening of the 12th and woke up after the dawn of the 13th, it can be expressed in the shape of the leftmost bar graph among the bar graphs illustrated in FIG. 7a. At this time, as illustrated in FIGS. 7a and 7b, the two ends of the bar graph of sleep state information obtained during a predetermined period may correspond to the falling-at-sleep time and the waking-up time, respectively.

Here, the time information of the y-axis may be expressed in units of time, but may also be expressed by the words noon, dawn, midnight, evening, and noon, as shown in FIGS. 7a and 7b (reference number 702). In addition, a line (solid line or dotted line) corresponding to each time of the y-axis may also be displayed (reference number 703).

As illustrated in FIG. 7a, in a case where lines corresponding to each time on the y-axis are displayed together according to one embodiment of the present invention, the line corresponding to midnight time can be displayed as a relatively thicker or brighter line than the lines corresponding to other times, thereby allowing them to be distinguished.

As illustrated in FIG. 7a, according to one embodiment of the present invention, the bar graph may be expressed entirely in a single color (reference number 704).

In addition, as illustrated in FIG. 7b, according to one embodiment of the present invention, a bar graph may display portions corresponding to sleep stages of REM sleep, deep sleep, light sleep, and wakefulness. Specifically, when the colors of shapes corresponding to each sleep stage information in the hypnogram graph are expressed differently, the color of the shape corresponding to each sleep stage information in the bar graph of the corresponding date may be displayed in proportion to the corresponding sleep stage.

In addition, as illustrated in FIG. 7b, in a case where a portion corresponding to sleep stages of REM sleep, deep sleep, light sleep, and wakefulness is displayed separately on a bar graph according to one embodiment of the present invention, the portion corresponding to each sleep stage can be arranged in the time order in which the corresponding sleep stage was detected.

Graphical user interface including stacked sleep stage graphs

Hereinafter, a Stacked Sleep Stages Graph will be described using FIGS. 8a to 8e, FIGS. 22a to 22e, and FIGS. 23a to 23d. Stacked Sleep Stage Information refers to information that represents the occurrence frequency of sleep stages included in sleep state information acquired in each sleep session in a time-series manner when sleep state information is acquired during multiple sleep sessions.

FIGS. 8A through 8E are black and white drawings of a graphical user interface including a Stacked Sleep Stage Graph according to embodiments of the present invention.

FIGS. 22A through 22E are diagrams illustrating a graphical user interface including a Stacked Sleep Stage Graph according to embodiments of the present invention.

FIGS. 23A to 23D are diagrams illustrating a graphical user interface including a Stacked Sleep Stage Graph generated based on a larger number of sleep sessions than FIGS. 22A to 22E, according to embodiments of the present invention. According to one embodiment of the present invention, a method is disclosed in which sleep state information of a user is generated based on environmental sensing information acquired during one or more sleep sessions, and a graph representing the frequency of a given sleep state over time along a time axis is generated and provided based on the generated sleep state information. Here, a sleep session may refer to a period from when sleep measurement is initiated to when sleep measurement is terminated. One sleep session means that sleep measurement is initiated once and terminated once.

Meanwhile, according to one embodiment of the present invention, one or more sleep sessions that serve as a basis for generating the stacked sleep stage graph may or may not be consecutive sleep sessions.

For example, sleep state information can be generated over 30 sleep sessions, and a sleep stage graph can be generated based on the total sleep state information generated over the 30 sleep sessions.

As another example, some sleep sessions out of 30 sleep sessions can be selected, and a sleep stage graph can be generated based on the sleep state information generated during some sleep sessions. Here, some sleep sessions can be consecutive sleep sessions or non-consecutive sleep sessions.

When generating a stacked sleep stage graph based on non-consecutive sleep sessions, a stacked sleep stage graph may be generated based on sleep sessions measured during a specific period. For example, only sleep sessions measured on weekdays may be collected separately, and a stacked sleep stage graph based on sleep state information generated on weekdays may be generated. Alternatively, only sleep sessions measured on weekends may be collected separately, and a stacked sleep stage graph based on sleep state information generated on weekends may be generated. Alternatively, only sleep sessions measured during an arbitrary period may be collected separately, and a stacked sleep stage graph may be generated, or only sleep sessions measured under specific conditions may be collected separately.

By selecting such discontinuous sleep sessions and generating a stacked sleep stage graph based on the sleep state information generated from the selected sleep sessions, a so-called conditioned stacked sleep stage graph can be generated. Through this, a graph that makes it easy to understand sleep state information during multiple sleep sessions under specific conditions at a glance can be generated, and a graphical user interface including the generated graph can be provided.

According to embodiments of the present invention, a graphical user interface including a stacked sleep stage graph such as FIGS. 8A to 8E may be output to a display screen provided on various electronic devices such as a user terminal (10), a graphical user interface generating device (100), or a graphical user interface providing device (200).

The x-axis of each graph illustrated in FIGS. 8A to 8E represents the flow of time, and may be expressed in units of epochs, for example. According to one embodiment of the present invention, each epoch may be set to data corresponding to a unit of 30 seconds. For example, 200 epochs represent data corresponding to 10 minutes (i.e., 600 seconds), 400 epochs represent data corresponding to 20 minutes (i.e., 1200 seconds), ..., 1000 epochs represent data corresponding to 500 minutes (i.e., 30000 seconds). Meanwhile, the unit time of the epoch is merely an example and is not limited thereto. For example, each epoch may be set to data corresponding to a unit of 60 seconds, in which case, the x-axis scale of the graphs may be expressed differently when compared to the graphs illustrated in FIGS. 8A to 8E.

In addition, the unit of ticks displayed on the x-axis of the stacked sleep stage graph according to embodiments of the present invention may be 200 epochs as shown in FIGS. 8A to 8E and FIGS. 22A to 22E, or 60 epochs as shown in FIGS. 23A to 23D, but this is merely an example and is not limited thereto.

Meanwhile, the y-axis of each graph shown in FIGS. 8a to 8e represents the frequency with which the corresponding sleep state (e.g., sleep stage) appears.

Referring to FIGS. 8A to 8E, in a graphical user interface including a stacked sleep stage graph, one or more areas corresponding to sleep stages may be expressed. Here, the sleep stage information may include information corresponding to a Wake sleep stage, information corresponding to a Light sleep stage, information corresponding to a Deep sleep stage, or information corresponding to a REM sleep stage.

In the stacked sleep stage graph according to embodiments of the present invention, the order in which areas indicating information corresponding to each sleep stage are displayed can be changed, and the order can also be set arbitrarily.

For example, referring to the graphical user interfaces illustrated in FIGS. 8A to 8E and FIGS. 22A to 22E, in a stacked sleep stage graph, an area indicating information corresponding to a Wake sleep stage may be displayed at the bottom of the graph, and above that, an area indicating information corresponding to a Light sleep stage, an area indicating information corresponding to a Deep sleep stage, and an area indicating information corresponding to a REM sleep stage may be displayed in that order.

As another example, referring to the graphical user interface illustrated in FIGS. 23a to 23d, an area indicating information corresponding to the Wake sleep stage may be displayed at the very bottom of the graph, and above that, an area indicating information corresponding to the REM sleep stage, an area indicating information corresponding to the Light sleep stage, and an area indicating information corresponding to the DEEP sleep stage may be displayed in that order.

Meanwhile, according to one embodiment of the present invention, an area that is not measured further after sleep measurement is terminated in a sleep session may be expressed as an NA (Not Applicable) area in the stacked sleep stage graph. Meanwhile, the NA area may be an area where no value exists at all, or may be an area that is displayed so as to be distinguished from an area expressing a sleep stage by using different colors, brightness, etc. In the stacked sleep stage graph according to embodiments of the present invention, the total length of the x-axis is formed based on the longest sleep session among multiple sleep sessions. Since sleep state information to be displayed in the stacked sleep stage graph is not generated after sleep measurement is terminated in the remaining sleep sessions (i.e., sleep sessions having a shorter sleep session length than the longest sleep session), such an area may be expressed as NA. The NA area may be an area indicating that a sleep session has terminated.

Referring to FIGS. 8a to 8e, in a graphical user interface including a stacked sleep stage graph, classes of each sleep stage can be displayed to be distinguished from each other. For example, the Wake sleep stage can be displayed in yellow, the Light sleep stage in light blue, the Deep sleep stage in dark blue, and the REM sleep stage in red. Alternatively, the areas representing each sleep stage can be expressed as lines. For example, if each of the graphical user interfaces illustrated in FIGS. 8a to 8e is colored, it can be expressed correspondingly to each of the graphical user interfaces illustrated in FIGS. 22a to 22e.

According to one embodiment of the present invention, a user can be provided with accumulated sleep stage information through a graphical user interface including an accumulated sleep stage graph.

Hereinafter, a detailed description will be given of a stacked sleep stage graph using FIG. 8A and FIG. 22A. According to one embodiment of the present invention, it is assumed that the graphs illustrated in FIG. 8A and FIG. 22A are graphs obtained by a user measuring sleep 100 times. In the present invention, the fact that a user measured sleep 100 times is referred to as "100 sleep sessions." In addition, it is assumed that one epoch is set as data corresponding to 30 seconds. The specific figures for the number of measured sleep sessions and the time corresponding to each epoch are merely examples and are not limited thereto.

In the graphs shown in FIG. 8a and FIG. 22a, the graphs are displayed in which the area from 0 to 100 of the y-axis is filled with the Wake sleep stage while the value of the x-axis increases from 0 to 10. Through this, it can be confirmed that, among the 100 sleep sessions of this user, during the time corresponding to the first 10 epochs (i.e., 300 seconds from the start of each sleep session), the sleep stage of this user was detected as the Wake stage all during the 100 sleep sessions.

Next, a graph was displayed in which, while the values of the x-axis increased from 10 to 20, the area from the values 0 to 41 of the y-axis was filled with the Wake sleep stage, but the area from the values 41 to 100 of the y-axis was filled with the Light sleep stage. Through this, it can be confirmed that among the 100 sleep sessions of this user, there were a total of 41 sleep sessions that detected the Wake sleep stage during the time corresponding to the 10th to the 20th epoch (i.e., during the time reaching 600 seconds after 300 seconds from the start of each sleep session), and there were a total of 59 (=100-41) sleep sessions that detected the Light sleep stage.

When the value of the x-axis is 30, a graph is displayed in which the area from 0 to 32 of the y-axis is filled with the Wake sleep stage, the area from 32 to 90 of the y-axis is filled with the Light sleep stage, and the area from 90 to 100 of the y-axis is filled with the Deep sleep stage. Through this, it can be confirmed that among the 100 sleep sessions of this user, at the time corresponding to the 30th epoch (i.e., when 900 seconds have passed from the start of each sleep session), there were a total of 32 sleep sessions that detected the Wake sleep stage, a total of 58 (=90-32) sleep sessions that detected the Light sleep stage, and a total of 10 (100-90) sleep sessions that detected the Deep sleep stage.

When the value of the x-axis is 60, the area from 0 to 15 of the y-axis is filled with the Wake sleep stage, the area from 15 to 62 of the y-axis is filled with the Light sleep stage, the area from 62 to 95 of the y-axis is filled with the Deep sleep stage, and the area from 95 to 100 of the y-axis is filled with the REM sleep stage. Through this, it can be confirmed that among the 100 sleep sessions of this user, at the time corresponding to the 60th epoch (i.e., when 1800 seconds had passed from the start of each sleep session), there were a total of 15 sleep sessions that detected the Wake sleep stage, a total of 47 (= 62-15) sleep sessions that detected the Light sleep stage, a total of 33 (= 95-62) sleep sessions that detected the Deep sleep stage, and a total of 5 (100-95) sleep sessions that detected the REM sleep stage.

Meanwhile, if a sleep stage graph is generated based on sleep analysis over a sufficient number of sleep sessions, even if the user does not measure sleep, there is an effect in which the probability of entering a certain sleep stage at a given point in time after sleep can be estimated simply by looking at the generated sleep stage graph.

Here, when a stacked sleep stage graph is generated based on sleep analysis for a sufficient number of sleep sessions, the sufficient number of sleep sessions may be continuous or discontinuous. When a stacked sleep stage graph is generated based on discontinuous sleep sessions, a stacked sleep stage graph may be generated based on sleep sessions measured during a specific period. Only sleep sessions measured on weekdays may be collected separately, or only sleep sessions measured on weekends may be collected separately. Alternatively, only sleep sessions measured during an arbitrary period may be collected separately, or only sleep sessions measured under specific conditions may be collected separately.

By selecting a sufficient number of discontinuous sleep sessions in this way and generating a stacked sleep stage graph based on the sleep state information generated from the selected sleep sessions, a conditional stacked sleep stage graph can be generated. Even if the user does not measure sleep, there is an effect in that the probability of entering a certain sleep stage at a certain point in time after sleep under certain conditions can be estimated simply by looking at the previously generated conditional stacked sleep stage graph.

For example, if a sufficient number of sleep sessions measured on a public holiday are aggregated to generate a conditional stacked sleep stage graph, then even if sleep is not measured directly on the public holiday, it is possible to estimate the probability of which sleep stage a person will be in at a given point in time after falling asleep during the public holiday, based on the previously generated conditional stacked sleep stage graph.

In addition, through the accumulated sleep stage graph according to the present invention, the sleep cycle during the user's sleep session can also be estimated. For example, as shown in FIG. 8A and FIG. 22A, through the boundary line between the REM sleep stage area and the Deep sleep stage area repeatedly increasing and decreasing over time, it can be confirmed that the frequency of the REM sleep stage repeatedly increases and decreases during the user's sleep session.

The stacked sleep stage graph according to the present invention has the effect of being able to identify the user's sleep time during multiple sessions. As shown in the graphs of FIG. 8a and FIG. 22a, there is a vertical boundary line between the NA region and the region occupied by each sleep stage (Wake, Light, Deep, REM). Based on where this boundary line is located in the graph and the length of the boundary line, the duration of each sleep session can be identified.

For example, in the graphs illustrated in FIG. 8a and FIG. 22a, the values on the x-axis at which a vertical boundary (y-axis direction) exists between the sleep stage region and the NA region correspond to x=80, 100 680, 740, 760, ..., 920, respectively. For example, when the value of the x-axis is 680, there exists a vertical boundary between the NA region and the sleep stage (REM) region, and the minimum y-value of this vertical boundary is 85. Through this, it can be seen that, among the user's 100 sleep sessions, the number of times the user's sleep session ended at a time point corresponding to the 680th epoch (i.e., a time point after 20400 (=680×30) seconds have elapsed from the start of each sleep session) was 15 (=100-85) times in total.

In addition, when the value of the x-axis is 760, there is a vertical boundary line between the NA area and the sleep stage (REM, Light) area, and the minimum y-value of this vertical boundary line is 75. Through this, it can be seen that among the user's 100 sleep sessions, the number of times the user's sleep session ended at the point corresponding to the 760th epoch (i.e., after 22800 (=760×30) seconds had passed from the start of each sleep session) was 25 (=100-75) times in total.

In the stacked sleep stage graph according to embodiments of the present invention, since the end of a sleep session is indicated by an NA region, the region corresponding to the sleep stage can form a form in which the y value monotonically decreases according to the passage of time on the x-axis.

Meanwhile, the end of a sleep session may mean that the user has finished measuring data for sleep analysis. In other words, it may mean the point in time when the user has completely woken up from sleep. Through the accumulated sleep stage graph in this way, there is an advantage in that the frequency of the end points of sleep sessions in multiple sleep sessions can be easily identified. In other words, there is an effect in that information about which epoch sleep ended can be easily identified. In addition, there is an effect in that the distribution of the length of sleep time (or the length of sleep session) in multiple sleep sessions can be identified at a glance.

Recently, research is being conducted to intuitively represent graphs of sleep state information including sleep stages.

In order to accurately measure sleep, a polysomnography test or a home polysomnography test must be used. However, since human sleep varies greatly from day to day, it is difficult to continuously measure daily sleep using the existing sleep measurement method. Sleep time and sleep stages can vary due to various reasons (e.g., lifestyle habits, etc.). Therefore, in the past, there was a problem that it was difficult to analyze sleep for multiple days (or multiple sleep sessions). In addition, even if sleep analysis was performed for multiple sleep sessions, there was also a problem that it was difficult to easily understand the results of sleep analysis for multiple sleep sessions at a glance. As explained using FIGS. 8A to 8E, the present invention has the effect of solving the above-described problems by allowing the results of sleep analysis for a user for multiple sleep sessions to be understood at a glance through a stacked sleep stage graph.

Ease of understanding sleep through accumulated sleep stage graphs

Meanwhile, since the sleep stage appears from the beginning of the sleep session and shows periodicity or a certain pattern as the sleep session progresses, there is an effect of making it easy to understand the information analyzed for sleep by presenting the sleep state information generated from the start time of the sleep session in a time series manner.

For normal sleep

For example, referring to the graphs illustrated in FIG. 8b and FIG. 22b, it can be easily seen at a glance that, as a result of analyzing sleep during multiple sleep sessions, the time from the start of the user's sleep session to the time of falling asleep was short, and the Wake stage did not appear much after sleep began. In addition, it can be seen at a glance that the Deep sleep stage was often detected in the early part of each sleep session after sleep began, and that the frequency of Deep sleep decreased over time and the REM sleep stage tended to be detected. In addition, since the sleep time identified from the borderline position with the NA area was observed to be more constant than the sleep time in FIG. 8a, it can be easily seen that FIG. 8b and FIG. 22b are graphs analyzing the sleep of a healthy person.

In addition, as shown in Fig. 23a, if the results of analyzing sleep over a larger number of sleep sessions than the graph shown in Fig. 22b are displayed as a stacked sleep stage graph, the periodicity according to the frequency of occurrence of sleep stages over time during sleep can be more clearly identified.

Referring to the graph illustrated in Fig. 23a, it can be seen that in multiple sleep sessions, the Deep sleep stage initially increases in a cycle of about 1 to 1.5 hours after sleep begins and then decreases. In addition, in the first Deep sleep stage cycle, it can be seen that the frequency of Deep sleep decreases around 60 epochs and the frequency of REM sleep increases, and that the frequency of Deep sleep increases around 150 epochs and the frequency of REM sleep decreases.

Additionally, referring to the graph illustrated in Figure 23a, it can be seen that over time in each sleep session, the periodicity of REM sleep tends to weaken as REM sleep lasts longer.

For those suffering from insomnia,

Referring to the graphs shown in Fig. 8c and Fig. 22c, it can be seen that the Wake stage was detected a lot throughout the sleep sessions, and the NA region occupies nearly half of the entire graph, indicating that the duration of each sleep session was short. This may mean that the sleep time was short, so it can be easily understood that Fig. 8c and Fig. 22c are graphs analyzing the sleep of a person suffering from insomnia.

In addition, as shown in Fig. 23b, if the results of analyzing sleep for a greater number of sleep sessions than the graph shown in Fig. 22c are displayed as a stacked sleep stage graph, the frequency of occurrence of sleep stages over time during sleep can be more clearly identified.

Referring to the graph shown in Figure 23b, it can be seen that the graph analyzes the sleep of a person suffering from insomnia, as the Wake stage is detected frequently throughout sleep in multiple sleep sessions and the NA region also appears frequently.

For sleep apnea

Referring to the graphs shown in FIGS. 8d and 22d, it can be seen that the Light sleep stage was detected frequently throughout the sleep sessions, the detection frequency of the Deep sleep stage was low, and the Wake stage was continuously detected at every time point throughout each sleep session. In addition, it can be seen that the detection frequency of the Deep sleep stage was low at the beginning of each sleep session, but the periodicity of the Deep sleep stage was not distinct. It is known that people with sleep apnea usually do not sleep deeply and the periodicity of the Deep sleep stage is not distinct. In light of this, it can be inferred that the figures in FIGS. 8d and 22d indicate that, before transitioning from the Light sleep stage to the Deep sleep stage, there are frequent awakenings during sleep, deep sleep is not achieved, and the Light sleep ratio is high due to obstructive apnea caused by muscle relaxation.

In addition, as shown in Fig. 23c, if the results of analyzing sleep for a greater number of sleep sessions than the graph shown in Fig. 22d are displayed as a stacked sleep stage graph, the frequency of occurrence of sleep stages over time during sleep can be more clearly identified.

Referring to the graph shown in Fig. 23c, it can be confirmed that the frequency of occurrence of the Deep sleep stage is not clearly periodic throughout sleep in multiple sleep sessions. It can also be confirmed that the frequency of occurrence of the Light sleep stage is high throughout sleep. In light of this, it can be inferred that the graph shown in Fig. 23c is a graph analyzing the sleep of a person suffering from sleep apnea.

For sleepers who suffer from both insomnia and sleep apnea

Referring to the graphs shown in Fig. 8e and Fig. 22e, it can be seen that both the Wake stage and the Light stage were detected frequently throughout the sleep sessions, and the detection frequency of the Deep sleep stage was almost non-existent. Therefore, in light of the fact that Fig. 8e exhibits all the features found in Fig. 8c and Fig. 8d, it can be easily seen that it is a graph analyzing the sleep of a person suffering from both sleep apnea and insomnia.

In addition, as shown in Fig. 23d, if the results of analyzing sleep for a greater number of sleep sessions than the graph shown in Fig. 22e are displayed as a stacked sleep stage graph, the frequency of occurrence of sleep stages over time during sleep can be more clearly identified.

Referring to the graph illustrated in Fig. 23d, it can be confirmed that the characteristics of a high frequency of occurrence of the Wake sleep stage throughout sleep and a characteristic of an unclear periodicity of the frequency of occurrence of the Deep sleep stage appear simultaneously in multiple sleep sessions. In addition, it can be confirmed that the frequency of occurrence of the Light sleep stage is high throughout sleep. In light of this, it can be determined that the graph illustrated in Fig. 23d is a graph analyzing the sleep of a person suffering from both sleep apnea and insomnia.

Meanwhile, according to one embodiment of the present invention, in order to more easily understand the sleep analysis content, auxiliary lines for indicating various information may be displayed together on the accumulated sleep stage graph. For example, an auxiliary line indicating the average sleep time may be displayed on the graph. Or, an auxiliary line indicating the average sleep onset delay time may be displayed on the graph. The examples of auxiliary lines are only examples for explanation and are not limited thereto, and it may be assumed that auxiliary lines indicating information belonging to at least one of various categories for interpreting sleep are possible.

In the above, a method for generating and providing a graphical user interface including a sleep stage graph based on sleep state information acquired during multiple sleep sessions has been described, but it is not limited to sleep stage information, and the frequency with which sleep event information appears may also be generated in a similar manner to the stacked sleep stage graphs illustrated in FIGS. 8a to 8e. For example, the frequency of occurrence of sleep events during multiple sessions may be analyzed, and stacked sleep event information may be acquired based on the frequency.

Stacked sleep stage information refers to information that represents the occurrence frequency of sleep events included in sleep state information acquired in each sleep session in a time-series manner when sleep state information is acquired during multiple sleep sessions. In addition, a stacked sleep event graph based on this stacked sleep event information can be generated in a form identical to or similar to that of FIGS. 8a to 8e.

Meanwhile, you can also display two graphs in parallel by setting certain conditions. Or, you can create graphs that can be compared by overlapping them, such as by adjusting the shading.

For example, the accumulated sleep stage graph and the accumulated sleep event graph can be displayed in parallel, or the two graphs can be displayed overlapping. By comparing the occurrence frequency of sleep stages and the occurrence frequency of sleep events over time in each sleep session during multiple sleep sessions, the effect of enabling multi-faceted sleep analysis can also be achieved.

In addition, the results of analyzing at least one of the stacked sleep stage graph and the stacked sleep event graph according to the present invention can be labeled and utilized as learning data for an artificial intelligence model (e.g., an artificial intelligence model based on image processing).

Alternatively, the results of analyzing at least one of the accumulated sleep stage graph and the accumulated sleep event graph may be used as learning data for an artificial intelligence model (e.g., an artificial intelligence model based on natural language processing), so that the results of analyzing the graph may be output by artificial intelligence.

Flowchart of how to create and provide a graphical user interface

According to the present invention, a graphical user interface may be provided that displays information about a user's sleep.

Hereinafter, a flowchart of a method for generating and providing a graphical user interface that displays information about a user's sleep according to embodiments of the present invention will be described using FIG. 10.

FIG. 10 is a flowchart of a method for generating and providing one or more graphical user interfaces representing information regarding a user's sleep according to one embodiment of the present invention.

As illustrated in FIG. 10, according to one embodiment of the present invention, a method for generating one or more graphical user interfaces representing information about sleep may include a step of acquiring sleep information (S120), a step of converting sleep information acquired in the time domain into information in the frequency domain (S140), a step of generating a graphical user interface (S160), and a step of providing a graphical user interface (S180).

The sleep information acquired in the step of acquiring sleep information here may include environmental sensing information or sleep sound information.

According to one embodiment of the present invention, after the step (S140) of converting sleep information acquired in the time domain into information in the frequency domain, a step (not shown) of acquiring sleep state information may be further included.

In addition, according to one embodiment of the present invention, a method for generating one or more graphical user interfaces representing information about sleep may further include a sleep log storage step of storing sleep log information related to an account assigned to a user in a memory. Here, the sleep log information may be stored in at least one of various electronic devices according to embodiments of the present invention. For example, the sleep log information may be sleep state information stored in at least one of a user terminal (10), a graphical user interface generating device (100), a graphical user interface providing device (200), and an external server (20). In a case where a device (a first electronic device) storing sleep log information according to an embodiment of the present invention and a device (a second electronic device) generating a graphical user interface are different devices, the second electronic device may receive specific sleep state information among sleep state information recorded in sleep log information stored in the first electronic device during a plurality of sleep sessions, generate a sleep state information graph based on the received sleep state information, and generate a graphical user interface including the generated sleep state information graph.

In addition, according to one embodiment of the present invention, the step (S140) of converting sleep information acquired in the time domain into information in the frequency domain may include a step of performing preprocessing on raw sound information in the time domain or information in the frequency domain.

Alternatively, according to one embodiment of the present invention, the step (S140) of converting sleep information acquired in the time domain into information in the frequency domain may include a step of converting sound information into spectrogram information in the frequency domain. In this case, the step of converting the spectrogram into a mel spectrogram by applying a mel scale may be further included.

Meanwhile, according to one embodiment of the present invention, a step of converting sleep information acquired in the time domain into information including changes in frequency components along the time axis may be included.

In addition, a step (not shown) of obtaining sleep state information according to one embodiment of the present invention may include a step of extracting sleep state information corresponding to each piece of information in the frequency domain, a spectrogram, or a mel spectrogram divided into 30-second units.

And, according to one embodiment of the present invention, the step (S160) of generating a graphical user interface may include a hypnogram graph generation step of generating a graph of sleep stages within the user's sleep period based on sleep information.

In addition, according to one embodiment of the present invention, the sleep information that serves as a basis for generating a hypnogram graph may be at least one of sleep information obtained in the time domain, information in the frequency domain, information including changes in frequency components of the sleep information along the time axis, a spectrogram, or a mel spectrogram to which a mel scale is applied.

Additionally, the step (S160) of generating a graphical user interface according to one embodiment of the present invention may include a step of generating a graph of sleep stability within the user's sleep period based on sleep information.

The step (S180) of providing a graphical user interface according to one embodiment of the present invention may further include a step of outputting the graphical user interface to a display unit equipped in various electronic devices such as a user terminal, a graphical user interface generating device (100), or a graphical user interface providing device (200).

Meanwhile, a method for providing a graphical user interface that displays information about a user's sleep according to one embodiment of the present invention may include a step of obtaining the user's sleep information, a step of generating the user's sleep state information based on the obtained user's sleep information, a step of generating a sleep state information graph over time based on the generated sleep state information, and a step of outputting a graphical user interface including the generated sleep state information graph.

Sleep state information according to one embodiment of the present invention may include at least one of sleep stage information, sleep stage probability information, accumulated sleep stage information, sleep event information, sleep event probability information, and accumulated sleep event information.

A sleep state information graph according to one embodiment of the present invention may be at least one of a graph representing sleep stage information, a graph representing sleep stage probability information, a graph representing sleep event information, and a graph representing sleep event probability information.

Meanwhile, a method for providing a graphical user interface that displays information about a user's sleep according to an embodiment of the present invention may further include a step of storing sleep log information that stores the user's sleep analysis information during a plurality of sleep sessions. At least one of the accumulated sleep stage graph or the accumulated sleep event graph according to an embodiment of the present invention may be generated based on sleep state information recorded in sleep log information stored during a plurality of sleep sessions. Here, the sleep log information may be stored in at least one of various electronic devices according to embodiments of the present invention. For example, the sleep log information may be sleep state information stored in at least one of the user terminal (10), the graphical user interface generating device (100), the graphical user interface providing device (200), and the external server (20). When the device (first electronic device) storing the sleep log information according to an embodiment of the present invention and the device (second electronic device) generating the graphical user interface are different devices, the second electronic device may receive specific sleep state information among the sleep state information recorded in the sleep log information stored in the first electronic device during a plurality of sleep sessions, generate a sleep state information graph based on the received sleep state information, and generate a graphical user interface including the generated sleep state information graph.

FIGS. 24a and 24b are conceptual diagrams illustrating a system in which various aspects of a sleep data interpretation content creation device or a sleep data interpretation content providing device based on user sleep information according to one embodiment of the present invention can be implemented.

As illustrated in FIG. 24a, a system according to embodiments of the present invention may include a computing device (100-2), a user terminal (300-2), an external server (20-2), and a network. Here, as illustrated in FIG. 24a, a sleep data interpretation content creation device (100a-2) based on user sleep information and/or a sleep data interpretation content provision device (100b-2) may be implemented as the computing device (100-2).

As illustrated in FIG. 24b, a device for generating sleep data interpretation content based on user sleep information according to an embodiment of the present invention may be implemented as a user terminal (300-2), and a device for providing sleep data interpretation content based on user sleep information may be implemented as an external server (20-2). Alternatively, as illustrated in FIG. 24b, a device for generating sleep data interpretation content based on user sleep information may be implemented as an external server (20-2), and a device for providing sleep data interpretation content based on user sleep information may be implemented as a user terminal (300-2). Alternatively, as illustrated in FIG. 24b, a device for generating sleep data interpretation content based on user sleep information and a device for providing sleep data interpretation content based on user sleep information may both be implemented as a user terminal (300-2). Alternatively, as illustrated in FIG. 24b, a device for generating sleep data interpretation content based on user sleep information and a device for providing sleep data interpretation content based on user sleep information may both be implemented as an external server (20-2).

Meanwhile, according to one embodiment of the present invention, a device for generating sleep data interpretation content based on user sleep information can generate sleep data interpretation content using generative artificial intelligence. In addition, according to one embodiment of the present invention, a device for providing sleep data interpretation content based on user sleep information can provide sleep data interpretation content generated using generative artificial intelligence.

Alternatively, according to one embodiment of the present invention, a device for generating sleep data interpretation content based on user sleep information may generate sleep data interpretation content based on a lookup table. In addition, according to one embodiment of the present invention, a device for providing sleep data interpretation content based on user sleep information may provide sleep data interpretation content generated based on a lookup table.

Figure 24c is a conceptual diagram illustrating a system in which sleep data interpretation content creation and provision based on user sleep information according to one embodiment of the present invention is implemented in a user terminal (300-2).

As illustrated in FIG. 24c, sleep data interpretation content may be created and provided based on user sleep information in a user terminal (300-2) without a separate creation device (100a-2) and/or a separate provision device (100b-2). Meanwhile, according to one embodiment of the present invention, a sleep data interpretation content creation device based on user sleep information may create sleep data interpretation content using generative artificial intelligence. In addition, according to one embodiment of the present invention, a sleep data interpretation content provision device based on user sleep information may provide sleep data interpretation content created using generative artificial intelligence.

As illustrated in FIG. 24c, even if the sleep data interpretation content creation device (100a-2) and the providing device (100b-2) based on user sleep information according to embodiments of the present invention are not separately provided, the user terminal (300-2) can perform the role of the sleep data interpretation content creation device (100a-2) and/or the providing device (100b-2) based on user sleep information through a network, thereby mutually transmitting and receiving data for the system according to embodiments of the present invention.

According to one embodiment of the present invention, a device (100a-2) for generating sleep data interpretation content based on user sleep information may include at least one of a display (not shown), a memory (not shown) storing one or more programs configured to be executed by one or more processors, and one or more processors (not shown).

In addition, according to one embodiment of the present invention, a device (100b-2) that provides sleep data interpretation content based on user sleep information may include at least one of a display (not shown), a memory (not shown) storing one or more programs configured to be executed by one or more processors, and one or more processors (not shown).

FIG. 24d is a conceptual diagram illustrating a system of a sleep data interpretation content creation device (100a-2) based on user sleep information according to one embodiment of the present invention. As illustrated in FIG. 24d, a system according to one embodiment of the present invention may include a sleep data interpretation content creation device (100a-2) based on user sleep information, a user terminal (300-2), and a network.

Figure 24e is a conceptual diagram illustrating a system of a sleep data interpretation content providing device (100b-2) based on user sleep information according to one embodiment of the present invention.

As illustrated in FIG. 24e, a system according to one embodiment of the present invention may include a sleep data interpretation content providing device (100b-2) based on user sleep information, a user terminal (300-2), and a network.

As illustrated in FIGS. 24d and 24e, in a system according to embodiments of the present invention, at least one of a sleep data interpretation content creation device (100a-2) based on sleep information and a sleep data interpretation content provision device (100b-2) based on sleep information can mutually transmit and receive data for the system according to embodiments of the present invention with a user terminal (300-2) through a network.

FIG. 24F is a conceptual diagram illustrating a system in which various aspects of various electronic devices according to embodiments of the present invention can be implemented. As illustrated in FIG. 24F, the electronic devices illustrated in FIG. 24F can perform at least one of the operations performed by various devices according to embodiments of the present invention.

For example, the operations performed by various electronic devices according to embodiments of the present invention may include operations of acquiring environmental sensing information and sleep information, operations of performing learning for sleep analysis, operations of performing inference for sleep analysis, and operations of acquiring sleep state information. Or, for example, operations of receiving information related to the user's sleep, transmitting or receiving at least one of environmental sensing information and sleep information, performing preprocessing for environmental sensing information, determining environmental sensing information and sleep information, extracting acoustic information from environmental sensing information and sleep information, processing or manipulating data, processing a service, providing a service, constructing a learning data set based on environmental sensing information or the user's sleep information, storing acquired data or a plurality of data that become inputs to a neural network, transmitting or receiving various pieces of information, mutually transmitting and receiving data for a system according to embodiments of the present invention through a network, generating or providing sleep data interpretation content based on the user's sleep information, generating or providing imagery induction information, generating or providing sleep images, generating or providing sleep data interpretation content using generative artificial intelligence based on the user's sleep information, etc.

The electronic devices illustrated in FIG. 24f may individually perform operations performed by various electronic devices according to embodiments of the present invention, but may also perform one or more operations simultaneously or in time series.

Referring to FIG. 24f, the electronic devices (1a-2 to 1d-2) illustrated in FIG. 24f may be electronic devices within the range of an area (or, sleep detection area) (11a-2) capable of acquiring environmental sensing information. Hereinafter, for convenience, the area (or, sleep detection area) (11a-2) capable of acquiring environmental sensing information will be referred to as “area (11a-2).”

Meanwhile, referring to FIG. 24f, the electronic devices (1a-2 and 1d-2) may be devices formed by a combination of two or more electronic devices.

Meanwhile, referring to FIG. 24f, the electronic devices (1a-2 and 1b-2) may be electronic devices connected to a network within the area (11a-2). In addition, the electronic devices (1c-2 and 1d-2) may be electronic devices not connected to a network within the area (11a-2). In addition, the electronic devices (2a-2 to 2b-2) may be electronic devices outside the range of the area (11a-2). In addition, there may be a network that interacts with the electronic devices within the range of the area (11a-2), and there may be a network that interacts with the electronic devices outside the range of the area (11a-2). Here, the network that interacts with the electronic devices within the range of the area (11a-2) may play a role in transmitting and receiving information for controlling smart home appliances.

In addition, the network interacting with the electronic devices within the scope of the area (11a-2) may be, for example, a short-range network or a local network. Here, the network interacting with the electronic devices within the scope of the area (11a-2) may be, for example, a long-range network or a global network. In addition, the electronic devices connected through the network outside the scope of the area (11a-2) may be one or more, and in this case, the electronic devices may perform data distribution processing or one or more operations separately. Here, the electronic devices connected through the network outside the scope of the area (11a-2) may include a server device. Alternatively, when there is one or more electronic devices connected through the network outside the scope of the area (11a-2), the electronic devices may perform various operations independently of one another.

Computing Device (100-2)

FIGS. 25A and 25B are block diagrams illustrating a computing device (100-2) according to one embodiment of the present invention.

Referring to FIG. 25b, the computing device (100-2) may include at least one of a processor (110-2), a memory (120-2), an output device (130-2), an input device (140-2), an input/output interface (150-2), a sensor module (160-2), a communication module (170-2), and a network unit (180-2).

According to the present invention, the computing device (100-2) can obtain sleep state information related to whether the user is before, during, or after sleep based on environmental sensing information. Specifically, the environmental sensing information can include sound information obtained non-invasively during the user's activity in a space or during sleep. For a specific example, the environmental sensing information can include sound generated as the user tosses and turns during sleep, sound related to muscle movement, or sound related to the user's breathing during sleep. According to an embodiment, the environmental sensing information can include sleep sound information, and the sleep sound information can mean sound information related to movement patterns and breathing patterns that occur during the user's sleep.

In one embodiment of the present invention, environmental sensing information can be obtained through a user terminal (300-2) carried by the user. For example, environmental sensing information related to the user's activity in a space can be obtained through a microphone module equipped in the user terminal (300-2).

According to the present invention, the microphone module equipped in the user terminal (300-2) carried by the user must be equipped in a relatively small-sized user terminal (300-2), and thus may be configured as a MEMS (Micro-Electro Mechanical Systems). Such a microphone module can be manufactured very small, but may have a low signal-to-noise ratio (SNR) compared to a condenser microphone or a dynamic microphone. A low signal-to-noise ratio may mean that the ratio of noise, which is a sound that is not intended to be identified, to the sound that is intended to be identified is high, and thus the sound is not easy to identify (i.e., unclear).

The environmental sensing information to be analyzed in the present invention may include sound information related to the user's breathing and movement acquired during sleep, i.e., sleep sound information. Such sleep sound information is information about very small sounds (i.e., sounds that are difficult to distinguish), such as the user's breathing and movement, and is acquired together with other sounds during a sleep environment. Therefore, if acquired through a microphone module with a low signal-to-noise ratio as described above, detection and analysis may be very difficult.

According to one embodiment of the present invention, the computing device (100-2) can acquire sleep state information based on environmental sensing information acquired through a microphone module composed of MEMS. Specifically, the computing device (100-2) can convert and/or adjust environmental sensing information acquired unclearly including a lot of noise into data that can be analyzed, and can perform learning for an artificial neural network by utilizing the converted and/or adjusted data. When pre-learning for the artificial neural network is completed, the learned neural network (e.g., an acoustic analysis model) can acquire sleep state information for the user based on data (e.g., a spectrogram) acquired (e.g., converted and/or adjusted) corresponding to sleep sound information.

According to the present invention, the sleep state information may include not only information regarding whether the user is sleeping, but also sleep stage information regarding changes in the user's sleep stage during sleep. For example, the sleep state information may include sleep stage information indicating that the user was in REM sleep at a first time point and that the user was in shallow sleep at a second time point that is different from the first time point. In this case, through the sleep state information, information may be obtained that the user was in relatively deep sleep at the first time point and was in shallow sleep at the second time point.

According to the present invention, the computing device (100-2) may be a terminal or a server, and may include any type of device. The computing device (100-2) may be a digital device, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone, which has a computing ability equipped with a processor and a memory. The computing device (100-2) may be a web server that processes a service.

According to the present invention, the computing device (100-2) can perform an operation of generating sleep data interpretation content based on user sleep information and/or an operation of providing sleep data interpretation content based on user sleep information. In this case, a sleep data interpretation content generation and provision system equipped with the computing device (100-2) can be implemented without separately providing a device (100a-2) for generating sleep data interpretation content based on user sleep information and/or a device (100b-2) for providing sleep data interpretation content based on user sleep information.

Processor (110-2)

According to the present invention, the processor (110-2) may include one or more application processors (APs), one or more communication processors (CPs), or at least one artificial intelligence processor (AI processor). The application processor, the communication processor, or the AI processor may be included in different integrated circuit (IC) packages, or may be included in one IC package.

According to the present invention, the application processor can drive an operating system or an application program to control a plurality of hardware or software components connected to the application processor and perform various data processing/operations including multimedia data. As an example, the application processor can be implemented as a SoC (system on chip). The processor (110-2) can further include a GPU (graphic processing unit, not shown).

According to the present invention, the communication processor can perform a function of managing a data link and converting a communication protocol in communication between the computing device (100-2) and other computing devices connected to a network. For example, the communication processor can be implemented as a SoC. The communication processor can perform at least a part of the multimedia control function. In addition, the communication processor can control data transmission and reception of the communication module (170-2). The communication processor can also be implemented to be included as at least a part of the application processor.

According to the present invention, the application processor or the communication processor can load and process commands or data received from at least one of the nonvolatile memory or other components connected thereto, respectively, into the volatile memory. In addition, the application processor or the communication processor can store data received from at least one of the other components or generated by at least one of the other components, into the nonvolatile memory.

According to the present invention, when the computer program is loaded into the memory (120-2), it may include one or more instructions that cause the processor (110-2) to perform the method/operation according to various embodiments of the present invention. That is, the processor (110-2) may perform the method/operation according to various embodiments of the present invention by executing the one or more instructions.

In one embodiment, the computer program may include one or more instructions for performing a method for creating a sleep environment based on sleep state information, the method including the steps of obtaining sleep state information of a user, generating environment creation information based on the sleep state information, and transmitting the environment creation information to an environment creation device.

According to one embodiment of the present invention, the processor (110-2) may be composed of one or more cores and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device.

According to one embodiment of the present invention, the processor (110-2) can read a computer program stored in the memory (120-2) and perform data processing for machine learning according to one embodiment of the present invention. In addition, the processor (110-2) can perform operations for learning a neural network. The processor (110-2) can perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating weights of a neural network using backpropagation.

In addition, at least one of the CPU, GPGPU, and TPU of the processor (110-2) can process learning of a network function. For example, the CPU and the GPGPU can process learning of a network function and data classification using the network function together. In addition, in one embodiment of the present invention, processors of a plurality of computing devices can be used together to process learning of a network function and data classification using the network function. In addition, a computer program executed in a computing device according to one embodiment of the present invention can be a CPU, GPGPU, or TPU executable program.

In this specification, the network function may be used interchangeably with artificial neural network, neural network. In this specification, the network function may include one or more neural networks, in which case the output of the network function may be an ensemble of outputs of one or more neural networks.

In this specification, a model may include a network function. The model may include one or more network functions, in which case the output of the model may be an ensemble of outputs of one or more network functions.

According to the present invention, the processor (110-2) can read a computer program stored in the memory (120-2) to provide a sleep analysis model according to one embodiment of the present invention. According to one embodiment of the present invention, the processor (110-2) can perform calculations to derive environment composition information based on sleep state information. According to one embodiment of the present invention, the processor (110-2) can perform calculations to train the sleep analysis model. The sleep analysis model will be described in more detail below. Sleep information related to the quality of the user's sleep can be inferred based on the sleep analysis model. Environmental sensing information acquired from the user in real time or periodically is input as an input value to the sleep analysis model to output data related to the user's sleep.

The learning of such a sleep analysis model and the inference based thereon can be performed by the computing device (100-2). That is, it can be designed that both learning and inference are performed by the computing device (100-2). However, in another embodiment, learning may be performed in the computing device (100-2), but inference may be performed in the user terminal (300-2). In addition, learning may be performed in the computing device (100-2), but inference may be performed in the external terminal (200-2). On the other hand, learning may be performed in the user terminal (300-2), but inference may be performed in the computing device (100-2). In addition, learning may be performed in the external terminal (200-2), but inference may be performed in the computing device (100-2). Alternatively, both learning and inference may be performed in the external terminal (200-2), or both learning and inference may be performed in the user terminal (300-2).

According to one embodiment of the present invention, the processor (110-2) can typically process the overall operation of the computing device (100-2). In addition, the processor (110-2) can process signals, data, information, etc. input or output through the components described above, or can provide or process appropriate information or functions to the user terminal by running an application program stored in the memory (120-2).

According to one embodiment of the present invention, the processor (110-2) can obtain the user's sleep state information. According to one embodiment of the present invention, obtaining the sleep state information may be obtaining or loading the sleep state information stored in the memory (120-2). In addition, obtaining the sleep sound information may be receiving or loading data from another computing device, a separate processing module within the same computing device, or another storage medium based on a wired/wireless communication means.

FIG. 25a is a block diagram illustrating a computing device (100-2) according to one embodiment of the present invention.

As illustrated in FIG. 25a, the computing device (100-2) may include a network unit (180-2), a memory (120-2), and a processor (110-2). It is not limited to the components included in the computing device (100-2) described above. That is, additional components may be included or some of the components described above may be omitted depending on the implementation aspect of the embodiments of the present invention.

Memory (120-2)

According to the present invention, the memory (120-2) may include built-in memory or external memory. The built-in memory may include at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), SDRAM (synchronous dynamic RAM), etc.) or nonvolatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, NAND flash memory, NOR flash memory, etc.). According to one embodiment, the built-in memory may take the form of an SSD (solid state drive). The external memory may further include a flash drive, for example, a CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), or a memory stick.

According to one embodiment of the present invention, the memory (120-2) can store a computer program for performing a method for creating a sleep environment according to sleep state information according to one embodiment of the present invention, and the stored computer program can be read and operated by the processor (110-2). In addition, the memory (120-2) can store any form of information generated or determined by the processor (110-2) and any form of information received by the network unit (180-2). In addition, the memory (120-2) can store data related to the user's sleep. For example, the memory (120-2) can temporarily or permanently store input/output data (e.g., environmental sensing information related to the user's sleep environment, sleep state information corresponding to the environmental sensing information, or environment creation information according to the sleep state information, etc.).

Output device (130-2)

The output device (130-2) may include at least one of a display module and/or a speaker. Specifically, the output device (130-2) may display or output various data including multimedia data, text data, voice data, etc. to the user as sound.

Input Device (140-2)

According to the present invention, the input device (140-2) may include a touch panel, a digital pen sensor, a key, or an ultrasonic input device. For example, the input device (140) may be an input/output interface (150-2).

According to the present invention, the touch panel can recognize touch input in at least one of electrostatic, pressure-sensitive, infrared, or ultrasonic methods. In addition, the touch panel may further include a controller (not shown). In the case of electrostatic, not only direct touch but also proximity recognition is possible. The touch panel may further include a tactile layer. In this case, the touch panel can provide a tactile response to the user. The digital pen sensor can be implemented using the same or similar method for receiving the user's touch input or a separate recognition layer. The key can be a keypad or a touch key. The ultrasonic input device is a device that can detect micro sound waves in a terminal and confirm data through a pen that generates an ultrasonic signal, and wireless recognition is possible. The computing device (100-2) can also receive user input from an external device (e.g., a network, a computer, or a server) connected thereto using the communication module (170-2).

According to the present invention, the input device (140-2) may further include a camera module and/or a microphone. The camera module is a device capable of capturing images and moving images, and may include one or more image sensors, an image signal processor (ISP), or a flash LED. The microphone may receive a voice signal and convert it into an electrical signal.

Input/Output Interface (150-2)

According to the present invention, the input/output interface (150-2) can transmit a command or data input by a user through an input device (140-2) or an output device (130-2) to a processor (110-2), a memory (120-2), a communication module (170-2), etc. through a bus (not shown). For example, the input/output interface (150-2) can provide data on a user's touch input input through a touch panel to the processor (110-2). For example, the input/output interface (150-2) can output a command or data received from a processor (110-2), a memory (120-2), a communication module (170-2), etc. through a bus through the output device (130-2). For example, the input/output interface (150-2) can output voice data processed by the processor (110-2) to the user through a speaker.

Sensor Module (160-2)

According to the present invention, the sensor module (160-2) may include at least one of a gesture sensor, a gyro sensor, a pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, an RGB (red, green, blue) sensor, a biometric sensor, a temperature/humidity sensor, an illuminance sensor, or an UV (ultra violet) sensor. The sensor module (160) may measure a physical quantity or detect an operating state of the computing device (100-2) and convert the measured or detected information into an electric signal. In addition, the sensor module (160-2) may include an olfactory sensor (E-nose sensor), an electromyography sensor (EMG sensor), an electroencephalogram sensor (EEG sensor, not shown), an electrocardiogram sensor (ECG sensor), a photoplethysmography sensor (PPG sensor), a heart rate monitor sensor (HRM sensor), a perspiration sensor, or a fingerprint sensor. The sensor module (160) may further include a control circuit for controlling at least one sensor included therein.

Communication Module (170-2)

According to the present invention, the communication module (170-2) may include a wireless communication module or an RF module. The wireless communication module may include, for example, Wi-Fi, BT, GPS or NFC. For example, the wireless communication module may provide a wireless communication function using a radio frequency. Additionally or alternatively, the wireless communication module may include a network interface or modem, etc. for connecting the computing device (100-2) to a network (e.g., Internet, LAN, WAN, telecommunication network, cellular network, satellite network, POTS or 5G network, etc.).

According to the present invention, the RF module can be responsible for transmitting and receiving data, for example, transmitting and receiving RF signals or called electronic signals. For example, the RF module can include a transceiver, a power amp module (PAM), a frequency filter, or a low noise amplifier (LNA). In addition, the RF module can further include a component for transmitting and receiving electromagnetic waves in free space in wireless communication, for example, a conductor or a wire.

According to the present invention, the computing device (100-2) may include at least one of a server, a TV, a smart TV, a refrigerator, an oven, a clothing styler, a robot cleaner, a drone, an air conditioner, an air purifier, a PC, a speaker, a home CCTV, a light, a washing machine, and a smart plug. Since the components of the computing device (100-2) described in FIG. 25d are examples of components generally provided in a computing device, the computing device (100-2) is not limited to the aforementioned components, and certain components may be omitted and/or added as needed.

Network Department (180-2)

The network unit (180-2) according to one embodiment of the present invention can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

In addition, the network unit (180-2) presented in this specification can use various wireless communication systems that are currently and may be realized in the future, such as mobile communication systems such as 4G and 5G (LTE), and satellite communication systems such as Starlink.

External terminal (200-2)

FIG. 25c is a block diagram for explaining an external terminal (200-2) according to one embodiment of the present invention.

According to the present invention, the external terminal (200-2) may include a processor (210-2), a memory (220-2), and a communication module (270-2). Specifically, the external terminal (200-2) may be an external server (20-2) or a cloud server. In addition, the external server (20-2) may be a digital device having a processor, memory, and computing capability, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone. The external server (20-2) may be a web server that processes a service. The types of servers described above are merely examples, and the present invention is not limited thereto.

According to one embodiment of the present invention, the external terminal (200-2) may be a server providing a cloud computing service. More specifically, the external terminal (200-2) may be a server providing a cloud computing service, which is a type of Internet-based computing that processes information with another computer connected to the Internet rather than the user's computer. The cloud computing service may be a service that stores data on the Internet and allows the user to use the service anytime and anywhere through an Internet connection without having to install necessary data or programs on his or her computer, and allows the data stored on the Internet to be easily shared and transmitted with simple operations and clicks.

According to one embodiment of the present invention, the external server (20-2) may be a server that stores information on a plurality of learning data for learning a neural network. The plurality of learning data may include, for example, health checkup information or sleep checkup information. For example, the external server (20-2) may be at least one of a hospital server and an information server, and may be a server that stores information on a plurality of polysomnography records, electronic health records, electronic medical records, and the like. For example, the polysomnography records may include information on breathing and movement during sleep of a sleep examination subject and information on sleep diagnosis results (for example, sleep stages, etc.) corresponding to the information. The information stored in the external server (20-2) may be utilized as learning data, verification data, and test data for learning a neural network in the present invention.

Processor (210)

According to the present invention, the processor (210-2) can control the external terminal (200-2). In addition, the processor (210-2) can include an AI processor (215-2). Specifically, the AI processor (215-2) can learn a neural network using a program stored in the memory (220-2). In particular, the AI processor (215-2) can learn a neural network for recognizing data related to the operation of the user terminal (300-2). Here, the neural network can be designed to simulate the human brain structure (e.g., the neuron structure of the human neural network) on a computer. The neural network can include an input layer, an output layer, and at least one hidden layer. Each layer includes at least one neuron having a weight, and the neural network can include a synapse connecting neurons to neurons. In a neural network, each neuron can output an input signal received through a synapse as a function value of an activation function for weights and/or biases.

According to the present invention, a plurality of network modes can transmit and receive data according to a connection relationship, respectively, so as to simulate the synaptic activity of neurons in which neurons transmit and receive signals through synapses. Here, the neural network may include a deep learning model developed from a neural network model. In the deep learning model, a plurality of network nodes may be located in different layers and may transmit and receive data according to a convolution connection relationship. Examples of the neural network model include various deep learning techniques such as a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network, a restricted Boltzmann machine, a deep belief network, and a deep Q-network, and can be applied in fields such as vision recognition, speech recognition, natural language processing, and speech/signal processing.

Meanwhile, the processor (210-2) performing the function as described above may be a general-purpose processor (e.g., CPU), but may be an AI-only processor for artificial intelligence learning (e.g., GPU, TPU).

Memory (220-2)

According to the present invention, the memory (220-2) can store various programs and data required for the operation of the user terminal (300-2) and/or the external terminal (200-2). The memory (220-2) is accessed by the AI processor (215-2), and data reading/recording/modifying/deleting/updating, etc. can be performed by the AI processor (215-2). In addition, the memory (220-2) can store a neural network model (e.g., a deep learning model) generated through a learning algorithm for data classification/recognition. In addition, the memory (220-2) can store not only the learning model (221-2), but also input data, learning data, learning history, etc.

Meanwhile, the AI processor (215-2) may include a data learning unit (215a-2) that learns a neural network for data classification/recognition. The data learning unit (215a-2) may learn criteria regarding which learning data to use to determine data classification/recognition and how to classify and recognize data using the learning data. The data learning unit (215a) may acquire learning data to be used for learning and learn a deep learning model by applying the acquired learning data to the deep learning model.

According to the present invention, the data learning unit (215a-2) may be manufactured in the form of at least one hardware chip and mounted on the external terminal (200-2). For example, the data learning unit (215a-2) may be manufactured in the form of a dedicated hardware chip for artificial intelligence and may be manufactured as a part of a general-purpose processor (CPU) or a graphics processor (GPU) and mounted on the external terminal (200-2). In addition, the data learning unit (215a-2) may be implemented as a software module. When implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer readable media that can be read by a computer. In this case, at least one software module may be provided to an operating system (OS) or provided by an application.

According to the present invention, the data learning unit (215a-2) can learn to have a judgment criterion on how to classify/recognize a given data by using the acquired learning data so that the neural network model can have a judgment criterion on how to classify/recognize a given data. At this time, the learning method by the model learning unit can be classified into supervised learning, unsupervised learning, and reinforcement learning. Here, supervised learning refers to a method of learning an artificial neural network in a state where a label for learning data is given, and the label can mean the correct answer (or result value) that the artificial neural network should infer when learning data is input to the artificial neural network.

According to the present invention, unsupervised learning may mean a method of learning an artificial neural network in a state where a label for learning data is not given. Reinforcement learning may mean a method of learning an agent defined in a specific environment to select an action or an action sequence that maximizes a cumulative reward in each state. In addition, the model learning unit may learn a neural network model using a learning algorithm including error backpropagation or gradient descent. When the neural network model is learned, the learned neural network model may be called a learning model (221-2). The learning model (221-2) may be stored in the memory (220-2) and used to infer results for new input data that is not learning data.

According to the present invention, the AI processor (215-2) may further include a data preprocessing unit (215b-2) and/or a data selection unit (215c-2) to improve the analysis results using the learning model (221-2) or to save resources or time required for generating the learning model (221-2).

According to the present invention, the data preprocessing unit (215b-2) can preprocess the acquired data so that the acquired data can be used for learning/inference for situation judgment. For example, the data preprocessing unit (215b-2) can extract feature information as preprocessing for input data received through the communication module (270-2), and the feature information can be extracted in a format such as a feature vector, a feature point, or a feature map.

According to the present invention, the data selection unit (215c-2) can select data required for learning from among the learning data or the learning data preprocessed in the preprocessing unit. The selected learning data can be provided to the model learning unit. For example, the data selection unit (215c-2) can select only data for an object included in a specific area as learning data by detecting a specific area from an image acquired through a camera of a computing device. In addition, the data selection unit (215c-2) can select data required for inference from among the input data acquired through an input device or the input data preprocessed in the preprocessing unit.

In addition, the AI processor (215-2) may further include a model evaluation unit (215d-2) to improve the analysis results of the neural network model. The model evaluation unit (215d-2) may input evaluation data into the neural network model, and if the analysis results output from the evaluation data do not satisfy a predetermined criterion, cause the model learning unit to learn again. In this case, the evaluation data may be preset data for evaluating the learning model (221-2). For example, the model evaluation unit (215d-2) may evaluate that the predetermined criterion is not satisfied if the number or ratio of evaluation data for which the analysis results are inaccurate among the analysis results of the learned neural network model for the evaluation data exceeds a preset threshold.

Communication Module (270-2)

According to the present invention, the communication module (270-2) can transmit the AI processing result by the AI processor (215-2) to the user terminal (300-2). In addition, it can also be transmitted to the computing device (100-2) illustrated in FIG. 24i.

User terminal (300-2)

FIG. 25d is a block diagram illustrating a user terminal (300-2) according to one embodiment of the present invention.

According to one embodiment of the present invention, the user terminal (300-2) is a terminal that can receive information related to the user's sleep through information exchange with the computing device (100-2), and may refer to a terminal possessed by the user. For example, the user terminal (300-2) may be a terminal related to a user who wants to improve his/her health through information related to his/her sleep habits. The user may obtain monitoring information related to his/her sleep through the user terminal (300-2). The monitoring information related to sleep may include, for example, sleep state information related to the time when the user fell asleep, the time he/she slept, the time when he/she woke up from sleep, or sleep stage information related to changes in sleep stages during sleep. For a specific example, the sleep stage information may refer to information on changes in the user's sleep, such as light sleep, normal sleep, deep sleep, or REM sleep, at each time point during the user's 8 hours of sleep last night. The specific description of the sleep stage information described above is merely an example, and the present invention is not limited thereto.

Wireless Communications Department (310-2)

According to the present invention, the wireless communication unit (310-2) may include one or more modules that enable wireless communication between the user terminal (300-2) and the wireless communication system, between the user terminal (300-2) and the computing device (100-2), or between the user terminal (300-2) and the external terminal (200-2). In addition, the wireless communication unit (310-2) may include one or more modules that connect the user terminal (300-2) to one or more networks. Specifically, the wireless communication unit (310-2) may include at least one of a broadcast reception module (311-2), a mobile communication module (312-2), a wireless Internet module (313-2), a short-range communication module (314-2), and a location information module (315-2).

Input section (320-2)

According to the present invention, the input unit (320-2) may include a camera (321-2) or an image input unit for inputting an image signal, a microphone (322-2) or an audio input unit for inputting an audio signal, and a user input unit (323-2, for example, a touch key, a mechanical key, etc.) for receiving information from a user. Voice data or image data collected by the input unit (320-2) may be analyzed and processed into a user's control command.

Sensing section (340-2)

According to the present invention, the sensing unit (340-2) may include one or more sensors for sensing at least one of information within the user terminal, information about the surrounding environment surrounding the user terminal, and user information. For example, the sensing unit (340-2) may include at least one of a proximity sensor (341-2), an illumination sensor (342-2), a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor), a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor, a battery gauge, an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor, etc.), and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric recognition sensor, etc.). Meanwhile, the user terminal disclosed in this specification can utilize information sensed by at least two or more of these sensors in combination.

Output section (350-2)

According to the present invention, the output unit (350-2) is for generating output related to vision, hearing, or tactile sensations, and may include at least one of a display unit (351-2), an audio output unit (352-2), a haptic module (353-2), and an optical output unit (354-2).

According to the present invention, the display unit (351-2) can implement a touch screen by forming a mutual layer structure with the touch sensor or forming an integral structure. This touch screen can function as a user input unit (323-2) that provides an input interface between the user terminal (300-2) and the user (U), and at the same time, can provide an output interface between the user terminal (300-2) and the user (U).

Interface section (360)

According to the present invention, the interface unit (360-2) serves as a passageway for various types of external devices connected to the user terminal (300-2). Specifically, the interface unit (360-2) may include at least one of a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I/O (Input/Output) port, a video I/O (Input/Output) port, and an earphone port. In the user terminal (300-2), appropriate control related to the connected external device (for example, a computing device (100-2)) may be performed in response to the connection of the external device to the interface unit (360-2).

Memory (370-2)

According to the present invention, the memory (370-2) stores data supporting various functions of the user terminal (300-2). Specifically, the memory (370-2) can store a plurality of application programs (or applications) running on the user terminal (300-2), data, commands, or instructions for the operation of the user terminal (300-2). At least some of these application programs can be downloaded from an external server via wireless communication. In addition, at least some of these application programs can exist on the user terminal (300-2) from the time of shipment for the basic functions of the user terminal (300-2) (e.g., call reception, call transmission, message reception, call transmission). Meanwhile, the application program can be stored in the memory (370-2), installed on the user terminal (300-2), and driven by the control unit (380-2) to perform the operation (or function) of the user terminal. Additionally, the memory (370-2) according to one embodiment of the present invention can store instructions for the operation of the control unit (380-2).

Control Unit (380-2)

According to the present invention, the control unit (380) can control the overall operation of the user terminal (300) in addition to the operation related to the application program. Specifically, the control unit (380-2) can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above or by operating the application program stored in the memory (370-2).

According to the present invention, the control unit (380-2) can control at least some of the components examined with reference to FIG. 25f in order to drive the application program stored in the memory (370-2). In addition, the control unit (380-2) can operate at least two or more of the components included in the user terminal (300-2) in combination with each other in order to drive the application program.

Power supply unit (390-2)

According to the present invention, the power supply unit (390-2) receives external power and internal power under the control of the control unit (380-2) and supplies power to each component included in the user terminal (300-2). The power supply unit (390-2) includes a battery, and the battery may be a built-in battery or a replaceable battery.

Sleep Information

According to one embodiment of the present invention, sleep information may be acquired (or obtained) from one or more sleep information sensor devices to generate one or more graphical user interfaces representing an evaluation of the user's sleep. The sleep information may include sleep sound information of the user acquired in a non-invasive manner during the user's activity or sleep. In addition, the sleep information may include the user's lifestyle information and the user's log data.

Meanwhile, in the present invention, sleep information may include environmental sensing information and user's lifestyle information. The user's lifestyle information may include information affecting the user's sleep. Specifically, information affecting the user's sleep may include the user's age, gender, disease, occupation, bedtime, wake-up time, heart rate, electrocardiogram, and sleep time. For example, if the user's sleep time is less than the reference time, it may affect the user's sleep the next day by requiring more sleep time. On the other hand, if the user's sleep time is more than the reference time, it may affect the user's sleep the next day by requiring less sleep time.

One or more sleep information sensor devices

According to one embodiment of the present invention, one or more sleep sensor devices may include a microphone module, a camera, and a light sensor equipped in the user terminal (300-2). For example, information related to the user's activity in a space may be obtained through the microphone module equipped in the user terminal (300-2). In addition, since the microphone module must be equipped in a relatively small-sized user terminal (300-2), it may be configured as a MEMC (Micro-electro Mechanical System). A detailed description of the sensor device will be omitted since it is the same as the description of the sensor module (160-2) or the sensing unit (340-2) described above.

Environmental sensing information

According to one embodiment of the present invention, environmental sensing information of the present invention can be acquired through the user terminal (300-2). The environmental sensing information may refer to sensing information acquired in a space where the user is located. The environmental sensing information may be sensing information acquired in relation to the user's activity or sleep in a non-contact manner. According to one embodiment of the present invention, as illustrated in FIG. 24f, it may refer to sensing information acquired in a sleep detection area (11a-2), but is not limited thereto.

For example, the environmental sensing information may be sleep sound information obtained in a bedroom where the user is sleeping. According to an embodiment, the environmental sensing information obtained through the user terminal (300-2) may be information that serves as a basis for obtaining the user's sleep state information in the present invention. For a specific example, sleep state information regarding whether the user is before, during, or after sleep may be obtained through environmental sensing information obtained in relation to the user's activity.

According to one embodiment of the present invention, the environmental sensing information may include sounds generated as the user tosses and turns during sleep, sounds related to muscle movements, or sounds related to the user's breathing during sleep. The sleep sound information may refer to sound information related to movement patterns and breathing patterns generated during the user's sleep. The environmental sensing information related to the user's activities in a space may be acquired through a microphone equipped in the user terminal (300-2).

According to the present invention, the environmental sensing information may include the user's breathing and movement information. Specifically, the user terminal (300-2) may include a radar sensor as a motion sensor. In addition, the user terminal (300-2) may perform signal processing on the user's movement and distance measured through the radar sensor to generate a discrete waveform (breathing information) corresponding to the user's breathing. The quantitative index may be obtained based on the discrete waveform and movement. In addition, the environmental sensing information may include measurement values obtained through a sensor that measures the temperature, humidity, and lighting level of the space where the user is sleeping. To this end, the user terminal (300-2) may be equipped with a sensor that measures the temperature, humidity, and lighting level of the bedroom.

According to one embodiment of the present invention, a device (100a-2) for generating sleep data interpretation content based on user sleep information and/or a device (100b-2) for providing sleep data interpretation content based on user sleep information can obtain sleep state information based on environmental sensing information obtained through a microphone module configured with MEMS.

Sleep sound information

Meanwhile, in the present invention, sleep information may include environmental sensing information and user's lifestyle information. The environmental sensing information may be sound information about the user's sleep. One or more sleep information sensor devices may collect raw data about sounds generated during sleep for analysis of sleep. The raw data about sounds generated during sleep may be in the time domain. Specifically, the sleep sound information may be related to breathing and movement patterns related to the user's sleep. For example, in a wakeful state, since all nervous systems are activated, breathing patterns may be irregular and there may be a lot of body movement. In addition, since the neck muscles are not relaxed, breathing sounds may be very small. On the other hand, when the user is sleeping, the autonomic nervous system is stabilized, breathing changes regularly, body movement may also be small, and breathing sounds may also be loud. In addition, when apnea occurs during sleep, loud breathing sounds may occur immediately after apnea as a compensatory mechanism. In other words, by collecting raw data about sleep, analysis of sleep can be performed.

Convert information in the time domain to the frequency-time domain

In one embodiment of the present invention, sound information in the time domain can be converted into information in the frequency domain or the frequency-time domain, and analysis can be performed. According to one embodiment of the present invention, the information in the frequency-time domain can be information that represents both information in the frequency domain and information in the time domain. According to one embodiment of the present invention, sound information in the time domain can be converted into information in the frequency domain or the frequency-time domain by performing at least one of a fast Fourier transform and a short-time Fourier transform (STFT), and the sound information can be analyzed based on the converted information. Meanwhile, according to one embodiment of the present invention, the information in the frequency-time domain can be a spectrogram corresponding to the sound information, but is not limited thereto.

Fig. 29

FIG. 29a is an exemplary diagram for explaining a method for obtaining a spectrogram corresponding to sleep sound information according to one embodiment of the present invention. As shown in FIG. 29a, sleep sound information can be converted to obtain a spectrogram.

At this time, a method of converting into a spectrogram based only on the amplitude excluding the phase from the raw data can be used, which not only protects privacy but also reduces the data capacity and improves the processing speed. However, in another embodiment, it is also possible to generate a spectrogram using both the phase and the amplitude.

According to one embodiment of the present invention, a processor can generate a sleep analysis model using a spectrogram generated based on sleep sound information. If sleep sound information expressed as audio data is directly used, the amount of information is very large, so the amount of computation and computation time increase significantly, and since unwanted signals are included, not only is the computation precision reduced, but there is also a concern about privacy infringement if all of the user's audio signals are transmitted to the server. The present invention removes noise from sleep sound information, converts it into a spectrogram (Mel spectrogram), and generates a sleep analysis model by learning the spectrogram, so the amount of computation and computation time can be reduced, and even an individual's privacy can be protected.

At least one of the processors (or control units) disclosed in the present invention can generate a spectrogram (SP) corresponding to sleep sound information (SS), as illustrated in FIG. 29a. Raw data (sleep sound information) that is the basis for generating the spectrogram (SP) can be input, and the raw data can be acquired through a user terminal, etc. from a start time input by the user to an end time, or can be acquired from a time when the user operates the terminal (e.g., setting an alarm) to a time corresponding to the terminal operation (e.g., alarm setting time), or can be acquired by automatically selecting a time point based on the user's sleep pattern, or can be acquired by automatically determining a time point based on a sound (e.g., user's speech, breathing, sound of peripheral devices (TV, washing machine)) or a change in illumination, as the user's intended sleep time.

Although not shown in FIG. 29a, a preprocessing process of the input raw data may be further included. The preprocessing process includes a noise reduction process of the raw data. In the noise reduction process, noise (e.g., white noise) included in the raw data is removed. The noise reduction process may be performed using an algorithm such as spectral gating or spectral subtraction to remove background noise. Furthermore, in the present invention, a noise removal process may be performed using a noise reduction algorithm based on deep learning. That is, a noise reduction algorithm specialized for the user's breathing sound or respiration sound may be used through deep learning. In particular, the present invention may generate a spectrogram based only on the amplitude excluding the phase from the raw data, but is not limited thereto. This not only protects privacy, but also reduces the data capacity to improve the processing speed.

At least one of the processors (or control units) disclosed in the present invention can perform a fast Fourier transform or a short-time Fourier transform (STFT) on sleep sound information (SS) to generate a spectrogram (SP) corresponding to the sleep sound information (SS). The spectrogram (SP) is intended to visualize and understand sound or waves, and may be a combination of waveform and spectrum features. The spectrogram (SP) may represent the difference in amplitude according to changes in the time axis and the frequency axis as a difference in printing density or display color.

According to the present invention, preprocessed sound-related raw data is cut into 30-second units and converted into a mel spectrogram. Accordingly, a 30-second mel spectrogram can have a dimension of 20 frequency bins x 1201 time steps. In the present invention, the split-cat method is used to change a rectangular mel spectrogram into a square shape, thereby preserving the amount of information.

Meanwhile, the present invention can use a method to simulate breathing sounds measured in various home environments by adding various noises generated in a home environment to clean breathing sounds. Since sounds have an additive property, they can be added to each other. However, adding original audio signals such as mp3 or pcm and converting them into mel spectrograms consumes a lot of computing resources. Therefore, the present invention proposes a method to convert breathing sounds and noises into mel spectrograms respectively and add them. Through this, it is possible to secure robustness in various home environments by simulating breathing sounds measured in various home environments and utilizing them for deep learning model learning.

In the present invention, the sleep sound information (SS) is related to sounds related to breathing and body movements acquired during the user's sleep time, and thus may be very small sounds. Accordingly, the processor (110-2) may convert the sleep sound information into a spectrogram (SP) to perform analysis on the sounds. In this case, as described above, the spectrogram (SP) includes information showing how the frequency spectrum of the sound changes over time, so that breathing or movement patterns related to relatively small sounds can be easily identified, and thus the efficiency of the analysis can be improved.

According to one embodiment, each spectrogram can be configured to have a frequency spectrum of different concentrations according to various sleep stages. That is, it may be difficult to predict whether the state is at least one of an awake state, a REM sleep state, a light sleep state, and a deep sleep state based only on changes in the energy level of sleep sound information, but by converting sleep sound information into a spectrogram, changes in the spectrum of each frequency can be easily detected, thereby enabling analysis corresponding to small sounds (e.g., breathing and body movements).

In addition, the processor (110-2) can process the spectrogram (SP) as input to the sleep analysis model to obtain sleep stage information. Here, the sleep analysis model is a model for obtaining sleep stage information related to changes in the user's sleep stage, and can output sleep stage information by inputting sleep sound information obtained during the user's sleep. In an embodiment, the sleep analysis model can include a neural network model configured through one or more network functions.

A processor according to one embodiment of the present invention can perform a fast Fourier transform or a short-time Fourier transform (STFT) on sleep sound information (SS) to convert it into information including changes in frequency components along the time axis. Learning and inference of a sleep analysis model can be performed based on the converted information.

Neural network

FIG. 30a is a schematic diagram illustrating one or more network functions according to one embodiment of the present invention.

Sleep analysis model

FIG. 30b is a diagram for explaining the structure of a sleep analysis model utilizing deep learning to analyze a user's sleep according to one embodiment of the present invention. According to one embodiment of the present invention, the sleep analysis model may include a feature extraction model that extracts one or more features for each predetermined epoch, and a feature classification model that classifies each of the features extracted through the feature extraction model into one or more sleep stages to generate sleep stage information.

According to one embodiment of the present invention, a sleep analysis model (symbol A) utilizing deep learning for analyzing a user's sleep disclosed in FIG. 30b can perform sleep information inference (E) through a feature extraction model (symbol B), an intermediate layer (symbol C), and a feature classification model (symbol D). The sleep analysis model (symbol A) utilizing deep learning configured through the feature extraction model (symbol B), the intermediate layer (symbol C), and the feature classification model (symbol D) performs learning of both time-series features and features for multiple images, and the sleep analysis model (symbol A) utilizing deep learning learned through this can infer sleep stages for the entire sleep time and infer sleep events occurring in real time.

Feature extraction model

According to one embodiment of the present invention, a feature extraction model (symbol B) can extract features of the input information by inputting sleep sound information or a converted spectrogram into a sleep analysis model (symbol A) utilizing deep learning for analyzing sleep.

The feature extraction model (symbol B) can be constructed by a unique deep learning model (preferably, MobileVITV2, Transformer, etc.) trained through a training data set. The feature extraction model (symbol B) can be trained through supervised learning or unsupervised learning. The feature extraction model can be trained to output output data similar to the input data through a training data set.

Meanwhile, according to an embodiment of the present invention, the feature extraction model (symbol B) can also perform learning by a one-to-one proxy task. In addition, in the process of learning to extract sleep state information for one spectrogram, the feature extraction model can be learned to extract features by combining another NN (Neural Network).

According to one embodiment of the present invention, information input to a sleep analysis model (symbol A) utilizing deep learning may be a spectrogram. Or, it may be information converted from acoustic information including changes in frequency components along the time axis.

FIG. 29a is an exemplary diagram illustrating a method for obtaining a spectrogram corresponding to sleep sound information according to one embodiment of the present invention.

As illustrated in FIG. 29a, the feature extraction model can extract features related to breathing sounds, breathing patterns, and movement patterns by analyzing the time-series frequency pattern of the spectrogram (SP). In one embodiment, the feature extraction model can be configured through a part of a neural network model (e.g., an autoencoder) that has been pre-trained through a learning data set. Here, the learning data set can be configured with a plurality of spectrograms and a plurality of sleep stage information corresponding to each spectrogram.

In one embodiment, the feature extraction model may be constructed from an autoencoder learned through a training data set via an encoder. The autoencoder may be learned through an unsupervised learning method. The autoencoder may be learned to output output data similar to the input data through the training data set. In detail, the encoder may learn only the core feature data (or features) of the input spectrogram through a hidden layer during the encoding process, and the remaining information may be lost. In this case, the output data of the hidden layer during the decoding process via the decoder may not be a perfect copy value but an approximation of the input data (i.e., the spectrogram). That is, the autoencoder may be learned to adjust the weights so that the output data and the input data are as similar as possible.

In an embodiment, each of a plurality of spectrograms included in a learning data set may be tagged with sleep stage information. Each of the plurality of spectrograms may be input to an encoder, and an output corresponding to each spectrogram may be stored in a manner matching the tagged sleep stage information. Specifically, when the encoder is used to input first learning data sets (i.e., a plurality of spectrograms) tagged with first sleep stage information (e.g., shallow sleep), features related to the output of the encoder for the corresponding input may be stored in a manner matching the first sleep stage information.

In an embodiment, one or more features related to the output of the encoder may be represented in a vector space. In this case, since the feature data output corresponding to each of the first learning data sets is an output through a spectrogram related to the first sleep stage, they may be located at a relatively close distance in the vector space. That is, learning of the encoder may be performed so that a plurality of spectrograms output similar features corresponding to each sleep stage.

In the case of the encoder, the decoder can be trained to extract features that enable it to restore input data well. Therefore, the feature extraction model can extract features (i.e., multiple features) that enable it to restore input data (i.e., spectrogram) well by implementing the encoder among the trained autoencoders.

The encoder that constructs the feature extraction model through the aforementioned learning process can extract features corresponding to a spectrogram when a spectrogram (e.g., a spectrogram converted in response to sleep sound information) is input.

As illustrated in FIG. 29a, a processor according to an embodiment of the present invention may process information including changes in the time axis of a spectrogram (SP) or frequency components generated in response to sleep sound information (SS) as input to a feature extraction model to extract features. Here, since the sleep sound information (SS) is time-series data acquired in time series during the user's sleep, the processor may divide the spectrogram (SP) into predetermined epochs. For example, the processor may divide the spectrogram (SP) corresponding to the sleep sound information (SS) into 30-second units to obtain a plurality of spectrograms. For example, if sleep sound information is acquired during the user's 7-hour (i.e., 420 minutes) sleep, the processor may divide the spectrogram into 30-second units to obtain 840 spectrograms. The specific numerical descriptions of the sleep time, the spectrogram division time unit, and the number of divisions described above are merely examples, and the present invention is not limited thereto.

According to one embodiment of the present invention, the processor may process each of the divided plurality of spectrograms as an input of a feature extraction model to extract a plurality of features corresponding to each of the plurality of spectrograms. For example, when the number of the plurality of spectrograms is 840, the number of the plurality of features extracted by the feature extraction model may also be 840. The specific numerical descriptions related to the above-described number of spectrograms and the plurality of features are only examples, and the present invention is not limited thereto.

In addition, the processor can process a plurality of features output through the feature extraction model as inputs of a feature classification model to obtain sleep stage information. In an embodiment, the feature classification model can be a neural network model that is pre-learned to predict a sleep stage corresponding to a feature. For example, the feature classification model can be a model that includes a fully connected layer and classifies a feature into at least one of the sleep stages. For example, when the feature classification model inputs a first feature corresponding to a first spectrogram, it can classify the first feature as shallow sleep.

Feature classification model

Referring to Figure 30b, a feature classification model (symbol D) can process multiple features obtained through a feature extraction model (symbol B) and an intermediate layer (symbol C) as inputs to the feature classification model (symbol D) to perform sleep information inference (symbol E).

In one embodiment of the present invention, the feature classification model (symbol D) may be a neural network model modeled to infer sleep information corresponding to a feature. For example, the feature classification model (symbol D) may be configured to include a fully connected layer, and may be a model that classifies a feature into at least one of sleep information. For example, when the feature classification model (symbol D) takes as input a feature corresponding to a spectrogram, it may classify the feature as REM sleep. For example, when the feature classification model (symbol D) takes as input a feature corresponding to a spectrogram, it may classify the feature as apnea during sleep.

In addition, according to an embodiment of the present invention, a processor may obtain sleep state information by processing a plurality of features output through a feature extraction model as inputs of a feature classification model. In an embodiment, the feature classification model may be a neural network model modeled to predict a sleep stage corresponding to a feature. For example, the feature classification model may be a model configured to include a fully connected layer and classify a feature into at least one of the sleep stages. For example, when the feature classification model inputs a first feature corresponding to a first spectrogram, it may classify the first feature as shallow sleep.

According to the present invention, a feature classification model can perform multi-epoch classification that predicts sleep stages of multiple epochs by inputting spectrograms related to multiple epochs. Specifically, multi-epoch classification may be intended to estimate multiple sleep stages (e.g., changes in sleep stages according to time) at once by inputting spectrograms corresponding to multiple epochs (i.e., combinations of spectrograms each corresponding to 30 seconds), rather than providing one piece of sleep stage analysis information corresponding to a single spectrogram of an epoch (i.e., one spectrogram corresponding to 30 seconds).

For example, since breathing patterns or movement patterns change slowly compared to brain wave signals or other biosignals, it may be necessary to observe how the patterns change over time to enable accurate estimation of sleep stages. For example, a feature classification model can input 40 spectrograms (e.g., 40 spectrograms each corresponding to 30 seconds), and perform prediction on 20 spectrograms located in the middle. That is, all spectrograms from 1 to 40 are examined, but sleep stages can be predicted through classification corresponding to spectrograms corresponding to 10 to 20. The specific numerical description of the number of spectrograms described above is only an example, and the present invention is not limited thereto.

That is, in the process of estimating sleep stages, rather than performing sleep stage prediction in response to each single spectrogram, the accuracy of the output can be improved by utilizing spectrograms corresponding to multiple epochs as inputs so that both past and future information can be considered.

According to one embodiment of the present invention, after the primary sleep analysis based on actigraphy and/or HRV, the secondary analysis based on sleep sound information uses the sleep analysis model as described above, and as illustrated in FIG. 29b, when the user's sleep sound information is input, the corresponding sleep stage (Wake, REM, Light, Deep) can be immediately inferred. In addition, the secondary analysis based on sleep sound information can extract the time point at which a sleep event (sleep apnea, hyperventilation) or snoring occurs through the singularity of the Mel spectrum corresponding to the sleep stage.

Figure 29c is a drawing for explaining sleep event determination using a spectrogram in a sleep analysis method according to the present invention.

As illustrated in Fig. 29c, a breathing pattern is analyzed in a single Mel spectrogram, and when a characteristic corresponding to an apnea or hyperpnea event is detected, the corresponding point in time can be determined as the point in time when the sleep event occurred. At this time, a process of classifying it as snoring rather than apnea or hyperpnea through frequency analysis may be further included.

As illustrated in FIG. 27, the user's sleep image and sleep sound are acquired in real time, and the acquired environmental sensing information or sleep sound information can be converted into information in the frequency domain, information in the frequency-time domain, or information including changes in frequency components of the acquired information along the time axis. According to one embodiment of the present invention, the user's sleep sound information can also be converted into a spectrogram. At this time, a preprocessing process of the environmental sensing information or sleep sound information can be performed.

According to the present invention, at least one of data converted from environmental sensing information or sleep sound information into information including changes in frequency components along a time axis, information in the converted frequency domain, information in the frequency-time domain, or a spectrogram is input into a sleep analysis model so that sleep stages can be analyzed. The conversion of information according to one embodiment of the present invention may be performed in real time. In addition, the operation in the case of employing a CNN or Transformer-based deep learning model in a feature classification model may be performed as follows.

According to one embodiment of the present invention, transformed information or spectrogram containing time series information can be input to a CNN-based deep learning model to output a vector with a reduced dimension. The vector with the reduced dimension can be input to a Transformer-based deep learning model to output a vector containing time series information.

According to an embodiment of the present invention, the average pooling technique can be applied to the output vector of a Transformer-based deep learning model by inputting it into a 1D CNN (1D Convolutional Neural Network), and performing an averaging operation on the time series information to convert it into an N-dimensional vector containing time series information. In this case, the N-dimensional vector containing time series information corresponds to data that still contains time series information, although it has a different resolution from the input data.

According to an embodiment of the present invention, multi-epoch classification can be performed on a combination of N-dimensional vectors containing output time series information, thereby making predictions on multiple sleep stages. In this case, predictions on continuous sleep state information can be made by using output vectors of Transformer-based deep learning models as inputs to multiple FCs (Fully Connected Layers). In addition, when a ViT or Mobile ViT-based deep learning model is employed in a feature classification model according to an embodiment of the present invention, the operation can be performed as follows.

A processor according to one embodiment of the present invention can output a vector with a reduced dimension by using information or a spectrogram containing time series information as input to a deep learning model based on Mobile ViT. In addition, according to an embodiment of the present invention, features can be extracted from each spectrogram as an output of the deep learning model based on Mobile ViT.

According to an embodiment of the present invention, a vector with a reduced dimension can be input to an Intermediate Layer, and a vector containing time series information can be output. The Intermediate Layer model can include at least one of a linearization step containing vector information, a layer normalization step for inputting mean and variance, or a dropout step for deactivating some nodes.

According to an embodiment of the present invention, by performing a process of inputting a vector with a reduced dimension to an intermediate layer and outputting a vector containing time series information, overfitting can be prevented.

According to an embodiment of the present invention, the output vector of the Intermediate Layer can be used as an input of a ViT-based deep learning model to output sleep state information. In this case, sleep state information corresponding to at least one of information including changes in frequency components along the time axis, information in the frequency domain including time series information, information in the frequency-time domain, spectrogram, or Mel spectrogram can be output from the environmental sensing information.

In addition, at least one of information including changes in frequency components along the time axis, information in the frequency domain including time series information, information in the frequency-time domain, spectrogram, or mel spectrogram converted from environmental sensing information or sleep sound information according to an embodiment of the present invention is configured in a series, and sleep state information corresponding to the information configured in the series can be output.

Meanwhile, in addition to the AI model mentioned above, various deep learning models may be employed in the feature extraction model or feature classification model according to the embodiment of the present invention to perform learning or inference, and the specific description related to the types of the deep learning models mentioned above is merely an example, and the present invention is not limited thereto.

Sleep state/sleep stage inference by inference model obtained through deep learning of environmental sensing information

As described above, an inference model for extracting a user's sleep state and sleep stage can be generated or learned through deep learning of environmental sensing information. Briefly again, environmental sensing information including acoustic information, etc., is converted into at least one of a spectrogram, information in the frequency domain, information in the frequency-time domain, or information including changes in frequency components along the time axis, and an inference model can be generated or learned based on the converted information.

The inference model can be built in at least one of various electronic devices according to embodiments of the present invention. Thereafter, environmental sensing information including user sound information obtained through the user terminal (300) is input to the inference model, and sleep state information and/or sleep stage information is output as a result value. In addition, the environmental sensing information may go through a preprocessing process before being input to the inference model. Meanwhile, learning and inference may be performed in the same device, but learning and inference may be performed in separate entities.

Identify outliers based on predefined pattern detection

According to one embodiment, a processor according to one embodiment of the present invention can obtain sleep state information based on environmental sensing information. Specifically, the processor can identify a singular point at which information of a preset pattern is detected in the environmental sensing information. Here, the information of the preset pattern may be related to breathing and movement patterns related to sleep. For example, in a wakeful state, since all nervous systems are activated, breathing patterns may be irregular and there may be a lot of body movement. In addition, since the neck muscles are not relaxed, breathing sounds may be very small. On the other hand, when a user is sleeping, the autonomic nervous system is stabilized, breathing changes regularly, body movements may also be small, and breathing sounds may also be large. That is, the processor (110-2) can identify a point in time at which acoustic information of a preset pattern related to regular breathing, small body movements, or small breathing sounds is detected in the environmental sensing information as a singular point. In addition, the processor can obtain sleep sound information based on the environmental sensing information acquired based on the identified singular point. The processor can identify singular points related to the user's sleep time from environmental sensing information acquired in a time-series manner, and acquire sleep sound information based on the singular points.

For a specific example, referring to FIG. 28, the processor may identify a singular point (P) related to a point in time when a preset pattern is identified from the environmental sensing information (E). The processor (110-2) may obtain sleep sound information (SS) based on sound information obtained after the identified singular point. The waveform and singular point related to sound in FIG. 28 are merely examples for understanding the present invention, and the present invention is not limited thereto.

That is, the processor can extract and acquire only sleep sound information from a large amount of sound information (i.e., environmental sensing information) based on the singularity by identifying the singularity related to the user's sleep from the environmental sensing information. This can provide convenience by automating the process of the user recording his/her sleep time, and can contribute to improving the accuracy of the acquired sleep sound information.

In addition, in the embodiment, the processor can obtain sleep state information related to whether the user is before or during sleep based on the singular point (P) identified from the environmental sensing information (E). Specifically, if the singular point (P) is not identified, the processor can determine that the user is before sleep, and if the singular point (P) is identified, the processor can determine that the user is during sleep after the singular point (P). In addition, the processor (110) can identify a time point (e.g., a time point of waking up) at which a preset pattern is not observed after the singular point (P) is identified, and if the time point is identified, the processor can determine that the user is after sleep, that is, has woken up.

That is, the processor may obtain sleep state information regarding whether the user is before, during, or after sleep based on whether a singularity (P) is identified in the environmental sensing information (E) and whether a preset pattern is continuously detected after the singularity is identified.

Analysis of sleep status information

Sleep status information

In one embodiment, the sleep state information may include information related to whether the user is sleeping. Specifically, the sleep state information may include at least one of first sleep state information indicating that the user is sleeping, second sleep state information indicating that the user is sleeping, and third sleep state information indicating that the user is sleeping. In other words, when the first sleep state information is inferred with respect to the user, the processor according to one embodiment of the present invention may determine that the user is in a state before sleeping (i.e., before going to bed), when the second sleep state information is inferred, the processor may determine that the user is in a state during sleeping, and when the third sleep state information is obtained, the processor may determine that the user is in a state after sleeping (i.e., waking up).

Additionally, the sleep state information may include, in addition to information related to the user's sleep stage, information about at least one of sleep apnea, snoring, tossing and turning, coughing, sneezing, or teeth grinding (e.g., sleep event information).

The sleep state information according to one embodiment of the present invention may include information on at least one of the user's motion (or movement) information or posture information. For example, the sleep state information may include information on whether the user is sleeping in a supine position or a non-supine position. The user's motion (or movement) information or posture information may be generated based on information acquired by at least one of a pressure sensor, an electroencephalogram sensor, a safety sensor, an electromyogram sensor, or an acceleration sensor. Alternatively, the user's motion (or movement) information or posture information may be generated based on information acquired by at least one of a respiratory flow measurement sensor or a microphone sensor.

One or more sleep state information may affect other sleep state information. For example, the user's posture information may affect the user's snoring information. Or, the user's sleep stage information (e.g., REM sleep stage) may affect the user's sleep event information. Therefore, predicting the user's sleep state information may affect predicting another sleep state information.

In order to learn or infer sleep stage information according to embodiments of the present invention, acoustic information acquired over a long time interval may be required.

On the other hand, in order to learn or predict sleep state information (e.g., snoring or apnea information, etc.) other than sleep stage information according to embodiments of the present invention, acoustic information acquired during a relatively short time interval (e.g., 1 minute) before and after the time when the corresponding sleep state occurs may be required.

Such sleep state information may be characterized as being acquired based on environmental sensing information. The environmental sensing information may include sensing information acquired in a non-contact manner in a space where the user is located.

According to one embodiment of the present invention, the processor can obtain sleep state information based on at least one of acoustic information, actigraphy, biometric information, and environmental sensing information obtained from the user terminal (300-2). Specifically, the processor can identify a singularity in the acoustic information. Here, the singularity of the acoustic information may be related to breathing and movement patterns related to sleep. For example, in a wakeful state, since all nervous systems are activated, breathing patterns may be irregular and there may be a lot of body movement. In addition, since the neck muscles are not relaxed, breathing sounds may be very small. On the other hand, when the user is sleeping, the autonomic nervous system is stabilized, breathing changes regularly, body movement may also be small, and breathing sounds may also be large. That is, the processor can identify a point in time when acoustic information of a pattern related to regular breathing, small body movement, or small breathing sounds is detected in the acoustic information as a singularity. In addition, the processor can obtain sleep sound information based on acoustic information obtained based on the identified singularity. The processor can identify singular points related to the user's sleep time from audio information acquired in a time-series manner, and acquire sleep audio information based on the singular points.

According to one embodiment, the processor can obtain environmental sensing information. Alternatively, the environmental sensing information can be obtained through a user terminal (300-2) carried by the user. For example, environmental sensing information related to a space where the user is active can be obtained through the user terminal (300-2) carried by the user, and the processor can receive the corresponding environmental sensing information from the user terminal (300-2). The processor according to one embodiment of the present invention can obtain sleep sound information based on the environmental sensing information.

According to the present invention, environmental sensing information may be acoustic information acquired in a non-contact manner in the user's daily life. For example, the environmental sensing information may include various acoustic information acquired according to the user's life, such as acoustic information related to cleaning, acoustic information related to cooking, acoustic information related to watching TV, and sleep acoustic information acquired during sleep. In an embodiment, sleep acoustic information acquired during the user's sleep may include sounds generated as the user tosses and turns during sleep, sounds related to muscle movements, or sounds related to the user's breathing during sleep. That is, the sleep acoustic information in the present invention may mean acoustic information related to movement patterns and breathing patterns related to the user's sleep.

Sleep Stage Information

According to one embodiment, the processor can extract sleep stage information. The sleep stage information can be extracted based on the user's environmental sensing information. The sleep stage can be divided into NREM (non-REM) sleep and REM (Rapid eye movement) sleep, and NREM sleep can be further divided into multiple stages (e.g., 2 stages of Light and Deep, 4 stages of N1 to N4). The sleep stage setting can be defined as a general sleep stage, but can also be arbitrarily set to various sleep stages depending on the designer. Through sleep stage analysis, not only the quality of sleep related to sleep but also sleep diseases (e.g., sleep apnea) and their fundamental causes (e.g., snoring) can be predicted. The processor can generate product recommendation information and verification information related to sleep based on the sleep stage information.

In addition, a processor according to one embodiment of the present invention can generate environment creation information based on sleep stage information. For example, when the sleep stage is in the Light stage or the N1 stage, environment creation information for controlling an environment creation device (light, air purifier, etc.) to induce deep sleep can be generated.

Hypnogram

FIGS. 26a and 26b are graphs verifying the performance of the sleep analysis method according to the present invention, and are drawings comparing the results of polysomnography (PSG) and the results of analysis using the AI algorithm according to the present invention (AI results). As shown in FIGS. 26a and 26b, the sleep stage information obtained according to the present invention is not only very consistent with the polysomnography, but also includes more precise and meaningful information related to the sleep stages (Wake, Light, Deep, REM).

The hypnodensity graph illustrated at the bottom of Fig. 26a is a graph representing sleep stage probability information, which represents the probability of which sleep stage among the four sleep stage classes it belongs to. Specifically, when predicting sleep stage information through the hypnodensity graph, the probability (i.e., sleep stage probability information) of which sleep stage class it belongs to among the four classes (Wake, Light, Deep, REM) in a periodic unit according to one or more epochs can be represented, and further, the probability of which sleep stage class it belongs to among the five classes (Wake, N1, N2, N3, REM) can be represented, the probability of which sleep stage class it belongs to among the three classes (Wake, Non-REM, REM) can be represented, and the probability of which sleep stage class it belongs to among the two classes (Wake, Sleep) can be represented. Here, the sleep stage probability information may mean a numerical representation of the proportion of a given sleep stage in a given epoch when classifying sleep stages.

According to one embodiment of the present invention, a hypnogram, which is a graph illustrated above a hypnodensity graph, can be obtained by determining a sleep stage with the highest probability from the hypnodensity graph. As illustrated in FIG. 26b, the sleep analysis results obtained according to the present invention showed very consistent performance when compared with labeling data obtained through a polysomnography test.

Sleep intention information

According to one embodiment of the present invention, the processor can obtain sleep intention information based on environmental sensing information. According to one embodiment, the processor can identify the type of sound included in the environmental sensing information.

In addition, a processor according to one embodiment of the present invention can calculate sleep intention information based on the number of types of identified sounds. The processor can calculate lower sleep intention information as the number of types of sounds increases, and can calculate higher sleep intention information as the number of types of sounds decreases.

For example, if there are three types of sounds included in the environmental sensing information (e.g., vacuum cleaner sound, TV sound, and user voice), the processor can calculate sleep intention information as two points. Also, if there is one type of sound included in the environmental sensing information (e.g., washing machine), the processor can calculate sleep intention information as six points.

That is, the processor can obtain sleep intention information related to how much the user intends to sleep based on the number of types of sounds included in the environmental sensing information. For example, as more types of sounds are identified, sleep intention information indicating a lower sleep intention of the user (i.e., sleep intention information with a low score) can be output. Meanwhile, the specific numerical descriptions of the types of sounds included in the aforementioned environmental sensing information and the sleep intention information are only examples, and the present invention is not limited thereto.

Obtaining sleep intention information using the intention score table method

According to one embodiment of the present invention, a processor may pre-match different intention scores to each of a plurality of pieces of sound information to generate or record an intention score table. For example, an intention score of 2 points may be pre-matched to first sound information related to a washing machine, an intention score of 5 points may be pre-matched to second sound information related to a humidifier sound, and an intention score of 1 point may be pre-matched to third sound information related to a voice. The processor may pre-match a relatively high intention score to sound information related to a user's sleep (e.g., sounds generated by the user's activities, such as vacuum cleaners, dishwashing, and voice sounds) and may pre-match a relatively low intention score to sound information unrelated to the user's sleep (e.g., sounds unrelated to the user's activities, such as vehicle noises and rain sounds) to generate an intention score table. The specific numerical description of the intention scores matched to each piece of sound information described above is merely an example, and the present invention is not limited thereto.

The processor can acquire sleep intention information based on the environmental sensing information and the intention score table. Specifically, the processor can record an intention score matched to the identified sound in response to a time point when at least one of a plurality of sounds included in the intention score table is identified in the environmental sensing information. For a specific example, if a vacuum cleaner sound is identified in response to a first time point in a process of acquiring environmental sensing information in real time, the processor can record two intention scores matched to the vacuum cleaner sound by matching them to the first time point. In the process of acquiring environmental sensing information, the processor can record an intention score matched to the identified sound by matching it to the corresponding time point whenever each of various sounds is identified.

In an embodiment, the processor may obtain sleep intention information based on the sum of intention scores obtained during a predetermined time period (e.g., 10 minutes). For example, the higher the intention score obtained during 10 minutes, the higher the sleep intention information may be obtained, and the lower the intention score obtained during 10 minutes, the lower the sleep intention information may be obtained. The specific numerical description of the above-mentioned predetermined time period is only an example, and the present invention is not limited thereto.

That is, according to one embodiment of the present invention, the processor can obtain sleep intention information related to how much the user intends to sleep based on the characteristics of the sound included in the environmental sensing information. For example, as the sound related to the user's activity is identified, sleep intention information indicating a lower user's sleep intention (i.e., sleep intention information with a low score) can be output.

Sleep event information

Sleep events according to one embodiment of the present invention include various events that may occur during sleep, such as snoring, sleep breathing (e.g., including information related to sleep apnea), and teeth grinding.

According to one embodiment of the present invention, it is possible to generate sleep event information indicating that a predetermined sleep event has occurred, or sleep event probability information indicating a probability of determining that a predetermined sleep event has occurred. Hereinafter, sleep respiration information, which is an example of sleep event information, will be described.

Fig. 26c is a graph verifying the performance of the sleep analysis method according to the present invention, and is a diagram comparing the polysomnography (PSG) result (PSG result) with the analysis result (AI result) using the AI algorithm according to the present invention in relation to sleep apnea and hypopnea. The probability graph illustrated at the bottom of Fig. 26c indicates the probability of which of the two diseases (sleep apnea, hypopnea) it belongs to for each 30 seconds when predicting a sleep disease by inputting user sleep sound information.

The graph shown in the middle of the three graphs shown in Fig. 26c can be obtained by determining the disease with the highest probability from the probability graph shown below it. Using the sleep analysis according to the present invention, as shown in Fig. 26c, the sleep state information obtained according to the present invention showed performance very consistent with the polysomnography. In addition, it showed performance that included more precise analysis information related to apnea and hypopnea.

According to the present invention, the point where a sleep event (sleep apnea, sleep hyperventilation, sleep hypoventilation) occurs can be identified by analyzing the user's sleep analysis in real time. If a stimulus (tactile, auditory, olfactory stimulus, etc.) is provided to the user at the moment when the sleep event occurs, the sleep event can be temporarily alleviated. That is, according to the present invention, the user's sleep event can be stopped and the frequency of the sleep event can be reduced based on accurate event detection related to the sleep event.

According to one embodiment of the present invention, a probability graph may indicate the probability of which of two diseases (sleep apnea, hypopnea) a user belongs to in units of 30 seconds when predicting a sleep disorder by inputting user sleep sound information, but is not limited to 30 seconds.

The graph shown in the middle of the three graphs illustrated in Fig. 26c can be obtained by determining the disease with the highest probability from the probability graph illustrated below it.

According to the present invention, the point where a sleep event (sleep apnea, sleep hyperventilation, sleep hypoventilation) occurs can be identified by analyzing the user's sleep analysis in real time. If a stimulus (tactile, auditory, olfactory stimulus, etc.) is provided to the user at the moment when the sleep event occurs, the sleep event can be temporarily alleviated. That is, according to the present invention, the user's sleep event can be stopped and the frequency of the sleep event can be reduced based on accurate event detection related to the sleep event.

Method for analyzing sleep status information using multimodal sleep information

An embodiment of a multimodal sleep state information analysis method (CONCEPT-A)

FIG. 38 is a flowchart illustrating a method for analyzing sleep state information, including a process of combining sleep sound information and sleep environment information into multimodal data according to one embodiment of the present invention.

In order to achieve the object of the present invention, according to one embodiment, a method for analyzing sleep state information using sleep sound information and sleep environment information in a multimodal manner may include a first information acquisition step (S600) of acquiring sound information in a time domain related to a user's sleep, a step of performing preprocessing of the first information (S602-2), a second information acquisition step (S610-2) of acquiring user sleep environment information related to the user's sleep, a step of performing preprocessing of the second information (S612), a combining step (S620-2) of combining data in a multimodal manner, a step (S630-2) of inputting multimodal data into a deep learning model, and a step (S640-2) of acquiring sleep state information as an output of the deep learning model.

According to an embodiment of the present invention, the first information acquisition step (S600-2) can acquire time domain sound information related to the user's sleep from the user terminal (300-2). The time domain sound information related to the user's sleep can include sound source information obtained from the sound source detection unit of the user terminal (300-2).

According to an embodiment of the present invention, in the step (S602-2) of performing data preprocessing of the first information, sleep sound information in the time domain can be converted into information including a change in the time axis of the frequency component, information in the frequency domain, or information in the frequency-time domain. In addition, the information in the frequency-time domain can be expressed as a spectrogram, and can be a mel spectrogram to which a mel scale is applied. By converting into a spectrogram, the user's privacy can be protected and the amount of data processing can be reduced. For example, by setting the bin of the information in the frequency domain to a predetermined value or less, data identification can be made difficult. In addition, the information converted from sleep sound information in the time domain is visualized, and in this case, by using it as an input for an artificial intelligence model based on image processing, sleep state information can be obtained through image analysis.

According to an embodiment of the present invention, the step (S602-2) of performing data preprocessing of the first information may further include a step of extracting features based on acoustic information. For example, a user's sleep breathing pattern may be extracted based on the acquired time domain acoustic information. For example, the acquired time domain acoustic information may be converted into information including a change in a time axis of a frequency component, and the user's breathing pattern may be extracted based on the converted information. Alternatively, the time domain acoustic information may be converted into information in a frequency domain or a frequency-time domain, and the user's sleep breathing pattern may be extracted based on the acoustic information in the frequency domain or the frequency-time domain.

In this case, the converted information is visualized and can be used as input to an image processing-based artificial intelligence model to output information such as the user's breathing pattern.

According to an embodiment of the present invention, the step (S602) of performing data preprocessing of the first information may include a data augmentation process for obtaining a sufficient amount of meaningful data for inputting sleep sound information into a deep learning model. The data augmentation technique may include pitch shifting augmentation, Tile UnTile (TUT) augmentation augmentation, and noise-added augmentation. The above-described augmentation technique is merely an example, and the present invention is not limited thereto.

According to an embodiment of the present invention, a method of adding data at a mel scale can shorten the time required for hardware to process data.

Meanwhile, the specific description of the types of noise described above is merely a simple example for explaining the noise addition augmentation of the present invention, and the present invention is not limited thereto.

According to an embodiment of the present invention, the second information acquisition step (S610-2) for acquiring user sleep environment information related to the user's sleep may acquire the user sleep environment information through the user terminal (300-2), an external server, or a network. The user's sleep environment information may refer to information related to sleep acquired in a space where the user is located. The sleep environment information may be sensing information (e.g., environmental sensing information) acquired in a space where the user is located by a non-contact method. The sleep environment information may be respiratory movement and body movement information measured through radar. The sleep environment information may be information related to the user's sleep acquired from a smart watch, smart home appliances, etc. The sleep environment information may be a photoplethysmography signal (PhotoPlethysmoGraphy). The sleep environment information may be heart rate variability (HRV) and heart rate acquired through a photoplethysmography signal (PhotoPlethysmoGraphy, PPG), and the photoplethysmography signal may be measured by a smart watch and a smart ring. Sleep environment information can be an Electro Encephalo Graphy (EEG) signal. Sleep environment information can be an Actigraphy signal measured during sleep.

According to one embodiment of the present invention, the second information may be information calculated based on a mathematical model rather than information obtained directly from a sensor. Specifically, the second information may be user's biometric information calculated based on a mathematical model, and may include at least one of sleep pressure information and biorhythm information.

According to one embodiment of the present invention, the time value elapsed from the time point of starting to measure sleep can be processed as input to a mathematical model to output the user's bio-information. Here, the output bio-information can be at least one of the user's sleep pressure information or bio-rhythm information. When the user's bio-information is used as second information to generate or acquire sleep state information multi-modally, the accuracy can be higher than when the sleep state information is generated or acquired based on a single piece of information.

According to an embodiment of the present invention, the step (S612-2) of performing preprocessing of the second information may include a data augmentation process for obtaining a sufficient amount of meaningful data for inputting data of the user's sleep environment information into a deep learning model.

According to one embodiment of the present invention, the step (S612-2) of performing preprocessing of the second information may include a step of processing data of the user's sleep environment information to extract features. For example, if the second information is a photoplethysmography signal (PPG), heart rate variability (HRV) and heart rate can be extracted from the photoplethysmography signal.

According to one embodiment of the present invention, the step (S612-2) of performing preprocessing of the second information may include TUT (Tile UnTile) augmentation and noise-added augmentation of the image information when the data of the user's sleep environment information is obtained as image information. The augmentation technique described above is merely an example of an augmentation technique of image information, and the present invention is not limited thereto. The user's sleep environment information may be information in various storage formats. The augmentation method of the user's sleep environment information may employ various methods.

According to an embodiment of the present invention, the step (S620-2) of combining first information and second information that have undergone a data preprocessing process into multimodal data combines data to input multimodal data into a deep learning model.

According to one embodiment of the present invention, a method of combining into multimodal data may be to combine preprocessed first information and preprocessed second information into data of the same format. Specifically, the first information may be acoustic image information in a frequency domain or a frequency-time domain, and the second information may be heartbeat image information in a time domain obtained from a smart watch. In this case, since the first information and the second information have different domains, they may be converted into the same domain and combined.

According to one embodiment of the present invention, a method of combining into multimodal data may be combining preprocessed first information and preprocessed second information into data of the same format. Specifically, the first information may be acoustic image information in a frequency domain or a frequency-time domain, and the second information may be heartbeat image information in a time domain obtained from a smart watch. In this case, since the first information and the second information do not have the same domain for use as inputs of a deep learning model, each data may be labeled as being related to the first information and the second information.

According to one embodiment of the present invention, the step (S620-2) of combining into multimodal data may perform first information augmentation, perform second information augmentation, and then combine. For example, the first information may be acoustic information of the user in the time domain, and the second information may be a photoplethysmography signal (PPG), and these may be combined into multimodal data. For example, the first information may be a spectrogram that converts acoustic information of the user in the time domain or acoustic information in the time domain into acoustic information in the frequency domain or frequency-time domain, and the second information may be a photoplethysmography signal (PPG), and these may be combined into multimodal data.

According to one embodiment of the present invention, the step (S620-2) of combining into multimodal data may perform first information augmentation, and perform second information augmentation and feature extraction to combine them. For example, the first information may be a spectrogram that converts time domain acoustic information of the user or time domain acoustic information into frequency domain or frequency-time domain acoustic information, and the second information may be heart rate variability (HRV) or heart rate obtained from a photoplethysmography signal (PPG), and these may be combined into multimodal data. According to one embodiment of the present invention, the step (S620-2) of combining into multimodal data may perform first information augmentation and feature extraction, and perform second information augmentation to combine them. For example, the first information may be a user breathing pattern extracted based on the user's acoustic information, and the second information may be heart rate variability (HRV) or heart rate obtained from a photoplethysmography signal (PPG), and these may be combined into multimodal data.

According to one embodiment of the present invention, the step (S620-2) of combining into multimodal data may perform first information augmentation and feature extraction, and then perform second information augmentation and feature extraction to combine. For example, the first information may be a user breathing pattern extracted based on the user's acoustic information, and the second information may be heart rate variability (HRV) or heart rate obtained from a photoplethysmography signal (PPG), and these may be combined into multimodal data.

According to an embodiment of the present invention, the step (S630-2) of inputting multimodal combined data into a deep learning model may process data into a matching form required for inputting the multimodal combined data into the deep learning model.

According to an embodiment of the present invention, the step (S640-2) of obtaining sleep state information as an output of a deep learning model can infer sleep state information by using multimodal combined data as an input of a deep learning model for inferring sleep state information. The sleep state information may be information about the user's sleep state.

According to one embodiment of the present invention, the sleep state information of the user may include sleep stage information that expresses the sleep of the user as a stage. The sleep stage may be divided into NREM (non-REM) sleep and REM (Rapid eye movement) sleep, and NREM sleep may be further divided into multiple stages (e.g., two stages of Light and Deep, four stages of N1 to N4). The sleep stage setting may be defined as a general sleep stage, but may also be arbitrarily set to various sleep stages depending on the designer.

According to one embodiment of the present invention, the sleep state information of the user may include sleep event information expressing a sleep-related disease or a sleep behavior occurring during the user's sleep. Specifically, the sleep event information occurring during the user's sleep may include sleep apnea and hypopnea information caused by the user's sleep disease. In addition, specifically, the sleep event information occurring during the user's sleep may include whether the user snores, the duration of the snoring, whether the user sleeps, the duration of the sleep talking, whether the user tosses and turns, and the duration of the tossing and turning. The described sleep event information of the user is merely an example for expressing an event occurring during the user's sleep, and is not limited thereto.

An embodiment of a multimodal sleep state information analysis method (CONCEPT-B)

FIG. 39 is a flowchart illustrating a method for analyzing sleep state information, including a step of combining each of inferred sleep sound information and sleep environment information into multimodal data according to one embodiment of the present invention.

In order to achieve the object of the present invention, according to one embodiment, a method for analyzing sleep state information using sleep sound information and sleep environment information in a multimodal manner may include a first information acquisition step (S700-2) of acquiring sound information in a time domain related to a user's sleep, a step of performing preprocessing of the first information (S702-2), a step of inferring information about sleep by using the first information as an input of a deep learning model (S704-2), a second information acquisition step (S710-2) of acquiring user sleep environment information related to the user's sleep, a step of performing preprocessing of the second information (S712-2), a step of inferring information about sleep by using the second information as an input of a deep learning model (S714-2), a combining step (S720-2) of combining data in a multimodal manner, and a step of acquiring sleep state information by combining multimodal data (S730-2).

According to an embodiment of the present invention, the first information acquisition step (S700-2) can acquire time domain sound information related to the user's sleep from the user terminal (300-2). The time domain sound information related to the user's sleep can include sound source information obtained from the sound source detection unit of the user terminal (300-2).

According to an embodiment of the present invention, in the step (S702-2) of performing data preprocessing of the first information, time-domain temporal acoustic information can be converted into information including a change in the time axis of a frequency component, or information in the frequency domain or frequency-time domain. In addition, the information in the frequency domain or frequency-time domain can be expressed as a spectrogram, and can be a Mel spectrogram to which a Mel scale is applied. By converting into a spectrogram, the user's privacy can be protected and the amount of data processing can be reduced.

According to an embodiment of the present invention, the step (S702-2) of performing data preprocessing of the first information may include a data augmentation process for obtaining a sufficient amount of meaningful data for inputting sleep sound information into a deep learning model. The data augmentation technique may include pitch shifting augmentation, Tile UnTile (TUT) augmentation augmentation, and noise-added augmentation. The above-described augmentation technique is merely an example, and the present invention is not limited thereto.

According to an embodiment of the present invention, a method of adding data at a mel scale can shorten the time required for hardware to process data.

According to an embodiment of the present invention, the second information acquisition step (S710-2) for acquiring user sleep environment information related to the user's sleep may acquire the user sleep environment information through the user terminal (300-2), an external server, or a network. The user's sleep environment information may refer to information related to sleep acquired in a space where the user is located. The sleep environment information may be sensing information acquired in a space where the user is located by a non-contact method. The sleep environment information may be information related to the user's sleep acquired from a smart watch, smart home appliances, etc. The sleep environment information may be heart rate variability (HRV) and heart rate acquired through a photoplethysmography signal (PhotoplethysmoGraphy, PPG), and the photoplethysmography signal may be measured by a smart watch and a smart ring. The sleep environment information may be an electroencephalography (EEG) signal. Sleep environmental information can be actigraphy signals measured during sleep.

According to one embodiment of the present invention, the second information may be information calculated based on a mathematical model rather than information obtained directly from a sensor. Specifically, the second information may be user's biometric information calculated based on a mathematical model, and may include at least one of sleep pressure information and biorhythm information.

According to one embodiment of the present invention, the time value elapsed from the time point of starting to measure sleep can be processed as input to a mathematical model to output the user's bio-information. Here, the output bio-information can be at least one of the user's sleep pressure information or bio-rhythm information. When the user's bio-information is used as second information to generate or acquire sleep state information multi-modally, the accuracy can be higher than when the sleep state information is generated or acquired based on a single piece of information.

According to an embodiment of the present invention, the step (S712) of performing preprocessing of the second information may include a data augmentation process for obtaining a sufficient amount of meaningful data for inputting data of the user's sleep environment information into a deep learning model.

According to one embodiment of the present invention, the step (S712-2) of performing preprocessing of the second information may include TUT (Tile UnTile) augmentation augmentation and noise addition augmentation of the image information when the data of the user's sleep environment information is obtained as image information. The augmentation technique described above is merely an example of an augmentation technique of image information, and the present invention is not limited thereto. The user's sleep environment information may be information in various storage formats. The augmentation method of the user's sleep environment information may employ various methods.

According to an embodiment of the present invention, the step (S704-2) of inferring information about sleep by using preprocessed first information as input to a deep learning model can infer information about sleep by using the pre-learned deep learning model as input.

According to an embodiment of the present invention, a pre-trained deep learning model can use inferred data as input for self-learning through the inferred data.

According to an embodiment of the present invention, a deep learning sleep analysis model that infers information about sleep by inputting first information about sleep sounds may include a feature extraction model and a feature classification model.

Among the deep learning sleep analysis models according to embodiments of the present invention, the feature extraction model can be pre-trained by a one-to-one proxy task that inputs one spectrogram and learns to predict sleep state information corresponding to one spectrogram. In the case where a CNN deep learning model is adopted in the feature extraction model according to embodiments of the present invention, learning may be performed by adopting a structure of a FC (Fully Connected Layer) or a FCN (Fully Connected Neural Network). In the case where a MobileViTV2 deep learning model is adopted in the feature extraction model according to embodiments of the present invention, learning may be performed by adopting a structure of an intermediate layer.

Among the deep learning sleep analysis models according to an embodiment of the present invention, a feature classification model can be trained to predict sleep state information of each spectrogram by inputting a plurality of continuous spectrograms, and to predict or classify overall sleep state information by analyzing a sequence of the plurality of continuous spectrograms.

According to an embodiment of the present invention, the step (S714-2) of inferring information about sleep by using the preprocessed second information as input to the inference model may infer information about sleep by using the input of a previously learned inference model. The previously learned inference model may be the sleep deep learning sleep analysis model described above, but is not limited thereto, and the previously learned inference model may be an inference model in various forms to achieve the purpose. The previously learned inference model may employ various methods.

According to an embodiment of the present invention, the step (S720-2) of combining first information and second information that have undergone a data preprocessing process into multimodal data combines data to determine sleep state information by combining information.

According to one embodiment of the present invention, a method of combining multimodal data may be to combine sleep information inferred through preprocessed first information and information inferred through preprocessed second information into data of the same format.

According to an embodiment of the present invention, the step (S730-2) of obtaining sleep state information by combining multimodal data may determine sleep state information of the user by combining data obtained multimodally. The sleep state information may be information about the sleep state of the user.

According to one embodiment of the present invention, the step (S730-2) of obtaining sleep state information by combining multimodal data may combine a hypnogram about the user's sleep inferred in the step (S704-2) of inferring information about sleep by using preprocessed first information as input of a deep learning model and a hypnogram about the user's sleep inferred in the step (S714-2) of inferring information about sleep by using preprocessed second information as input of an inference model. For example, sleep state information may be obtained by overlapping each hypnogram and employing information about sleep stages for matching portions and determining whether to employ information about sleep stages for mismatched portions by assigning weights to them.

According to one embodiment of the present invention, the step (S730-2) of obtaining sleep state information by combining multimodal data may combine the hypnodensity graph of the user's sleep inferred in the step (S704-2) of inferring information about sleep by using the preprocessed first information as input of a deep learning model and the hypnodensity graph of the user's sleep inferred in the step (S714-2) of inferring information about sleep by using the preprocessed second information as input of an inference model. For example, by substituting the probability of each hypnodensity graph into a formula, the sleep stage with the highest reliability at each time point may be obtained as the user's sleep stage information. For example, in each hypnodensity graph, if the reliability over time exceeds a preset reliability threshold, it is used as the user's sleep stage information, and if there is no sleep stage information whose reliability over time exceeds the preset reliability threshold, it is used as sleep stage information through weighting, thereby obtaining sleep state information.

According to one embodiment of the present invention, the step (S730-2) of obtaining sleep state information by combining multimodal data may combine a hypnogram regarding the user's sleep inferred in the step (S704-2) of inferring information regarding sleep by using the preprocessed first information as input to a deep learning model and a hypnodensity graph regarding the user's sleep inferred in the step (S714-2) of inferring information regarding sleep by using the preprocessed second information as input to an inference model. For example, when the reliability of the sleep stage displayed in the hypnogram and the hypnodensity graph exceeds a preset threshold, the user's sleep state information may be obtained by employing it as the user's sleep stage. For example, when the reliability of the sleep stage displayed in the hypnogram and the hypnodensity graph does not exceed a preset threshold, the user's sleep state information may be obtained with high reliability by applying a weight and employing it as the user's sleep stage.

According to one embodiment of the present invention, the sleep state information of the user may include sleep stage information that expresses the sleep of the user as a stage. The sleep stage may be divided into NREM (non-REM) sleep and REM (Rapid eye movement) sleep, and NREM sleep may be further divided into multiple stages (e.g., two stages of Light and Deep, four stages of N1 to N4). The sleep stage setting may be defined as a general sleep stage, but may also be arbitrarily set to various sleep stages depending on the designer.

According to one embodiment of the present invention, the user's sleep state information may include sleep stage information that indicates the user's sleep as stages. The method for indicating the stages of sleep may include a hypnogram that indicates the sleep stages on a graph and a hypnodensity graph that indicates the probability of each sleep stage on a graph, but the method of indicating is not limited thereto.

According to one embodiment of the present invention, the sleep state information of the user may include sleep event information expressing a sleep-related disease or a sleep behavior occurring during the user's sleep. Specifically, the sleep event information occurring during the user's sleep may include sleep apnea and hypopnea information caused by the user's sleep disease. In addition, specifically, the sleep event information occurring during the user's sleep may include whether the user snores, the duration of the snoring, whether the user sleeps, the duration of the sleep talking, whether the user tosses and turns, and the duration of the tossing and turning. The described sleep event information of the user is merely an example for expressing an event occurring during the user's sleep, and is not limited thereto.

An embodiment of a multimodal sleep state information analysis method (CONCEPT-C)

FIG. 40 is a flowchart illustrating a method for analyzing sleep state information, including a step of combining inferred sleep sound information with sleep environment information and multimodal data according to one embodiment of the present invention.

In order to achieve the object of the present invention, according to one embodiment, a method for analyzing sleep state information using sleep sound information and sleep environment information in a multimodal manner may include a first information acquisition step (S800-2) of acquiring sound information in a time domain related to a user's sleep, a step (S802-2) of performing preprocessing of the first information, a step (S804-2) of inferring information about sleep using the first information as an input of a deep learning model, a second information acquisition step (S810-2) of acquiring user sleep environment information related to the user's sleep, a combining step (S820-2) of combining data in a multimodal manner, and a step (S830-2) of acquiring sleep state information by combining multimodal data.

According to an embodiment of the present invention, the first information acquisition step (S800-2) can acquire time domain sound information related to the user's sleep from the user terminal (300-2). The time domain sound information related to the user's sleep can include sound source information obtained from the sound source detection unit of the user terminal (300-2).

According to an embodiment of the present invention, in the step (S802-2) of performing data preprocessing of the first information, time acoustic information in the time domain can be converted into information in the frequency domain or frequency-time domain. In addition, the information in the frequency domain or frequency-time domain can be expressed as a spectrogram, and can be a Mel spectrogram to which a Mel scale is applied. By converting into a spectrogram, the user's privacy can be protected and the amount of data processing can be reduced.

According to an embodiment of the present invention, the step (S802-2) of performing data preprocessing of the first information may include a data augmentation process for obtaining a sufficient amount of meaningful data for inputting sleep sound information into a deep learning model. The data augmentation technique may include pitch shifting augmentation, Tile UnTile (TUT) augmentation augmentation, and noise-added augmentation. The above-described augmentation technique is merely an example, and the present invention is not limited thereto.

According to an embodiment of the present invention, a method of adding data at a mel scale can shorten the time required for hardware to process data.

According to an embodiment of the present invention, the second information acquisition step (S810-2) for acquiring user sleep environment information related to the user's sleep may acquire the user sleep environment information through the user terminal (300-2), an external server, or a network. The user's sleep environment information may refer to information related to sleep acquired in a space where the user is located. The sleep environment information may be sensing information acquired in a space where the user is located by a non-contact method. The sleep environment information may be information related to the user's sleep acquired from a smart watch, smart home appliances, etc. The sleep environment information may be heart rate variability (HRV) and heart rate acquired through a photoplethysmography signal (PhotoplethysmoGraphy, PPG), and the photoplethysmography signal may be measured by a smart watch and a smart ring. The sleep environment information may be an electroencephalography (EEG) signal. The sleep environment information may be an actigraphy signal measured during sleep. The sleep environment information may be labeling data representing user information. Specifically, the labeling data may include the user's age, disease, physical condition, race, height, weight, and body mass index, and these are only examples of labeling data representing user information, but are not limited thereto. The above-described sleep environment information is only an example of information that may affect the user's sleep, but is not limited thereto.

According to one embodiment of the present invention, the second information may be information calculated based on a mathematical model rather than information obtained directly from a sensor. Specifically, the second information may be user's biometric information calculated based on a mathematical model, and may include at least one of sleep pressure information and biorhythm information.

According to one embodiment of the present invention, the time value elapsed from the time point of starting to measure sleep can be processed as input to a mathematical model to output the user's bio-information. Here, the output bio-information can be at least one of the user's sleep pressure information or bio-rhythm information. When the user's bio-information is used as second information to generate or acquire sleep state information multi-modally, the accuracy can be higher than when the sleep state information is generated or acquired based on a single piece of information.

According to an embodiment of the present invention, the step (S804-2) of inferring information about sleep by using preprocessed first information as input to a deep learning model can infer information about sleep by using the pre-learned deep learning model as input.

According to an embodiment of the present invention, a deep learning sleep analysis model that infers information about sleep by inputting first information about sleep sounds may include a feature extraction model and a feature classification model.

Among the deep learning sleep analysis models according to embodiments of the present invention, the feature extraction model can be pre-trained by a one-to-one proxy task that inputs one spectrogram and learns to predict sleep state information corresponding to one spectrogram. In the case where a CNN deep learning model is adopted in the feature extraction model according to embodiments of the present invention, learning may be performed by adopting a structure of a FC (Fully Connected Layer) or a FCN (Fully Connected Neural Network). In the case where a MobileViTV2 deep learning model is adopted in the feature extraction model according to embodiments of the present invention, learning may be performed by adopting a structure of an intermediate layer.

Among the deep learning sleep analysis models according to an embodiment of the present invention, a feature classification model can be trained to predict sleep state information of each spectrogram by inputting a plurality of continuous spectrograms, and to predict or classify time-series sleep state information by analyzing a sequence of the plurality of continuous spectrograms.

According to an embodiment of the present invention, the step (S820-2) of combining first information and second information that have undergone a data preprocessing process into multimodal data combines data to input multimodal data into a deep learning model.

According to one embodiment of the present invention, a method of combining multimodal data may be to combine sleep information inferred through preprocessed first information and information inferred through preprocessed second information into data of the same format.

According to an embodiment of the present invention, the step (S830-2) of obtaining sleep state information by combining multimodal data may combine data obtained multimodally and thereby determine sleep state information of the user. The sleep state information may be information about the sleep state of the user.

According to one embodiment of the present invention, the sleep state information of the user may include sleep stage information that expresses the sleep of the user as a stage. The sleep stage may be divided into NREM (non-REM) sleep and REM (Rapid eye movement) sleep, and NREM sleep may be further divided into multiple stages (e.g., two stages of Light and Deep, four stages of N1 to N4). The sleep stage setting may be defined as a general sleep stage, but may also be arbitrarily set to various sleep stages depending on the designer.

Real-time sleep analysis

According to an embodiment of the present invention, analysis of sleep state information based on acoustic information may include a detection step for sleep events (e.g., apnea, hypopnea, snoring, sleep talking, etc.). However, since the characteristics of sleep sound patterns are reflected over time, it may be difficult to identify them with only short acoustic data at a specific point in time. Therefore, in order to model acoustic information, analysis should be performed based on the time series characteristics of acoustic information.

In addition, sleep events that occur during sleep (e.g., apnea, hypopnea, snoring, sleep talking, etc.) have various characteristics related to sleep events. For example, there is no sound during an apnea event, but when the apnea event ends, a loud sound may occur as air passes through again, and the characteristics of the apnea event can be learned in time series to detect sleep events.

Meanwhile, according to one embodiment of the present invention, when analyzing acoustic information in the time domain, only a portion cut off by the length of a predetermined section on the time axis can be analyzed. A portion corresponding to the length of a predetermined section can be called a window (time window), and according to one embodiment of the present invention, the length of a predetermined section on the time axis of the window can be 20 minutes, but is not limited thereto.

According to one embodiment of the present invention, sleep for 20 minutes can be analyzed based on each piece of sound information corresponding to a 20-minute window. For the measured sound window, analysis can be performed at predetermined time intervals shorter than the length of the window section. For example, sleep analysis can be performed by generating sleep state information every 5 minutes for a window corresponding to 20 minutes. By generating sleep state information for a relatively short period of time in this way, there is an effect that sleep can be analyzed in real time. In the case of analyzing sleep by window, the size of data input to the sleep analysis model can be relatively small, so there is also an effect that the sleep analysis time can be shortened.

In addition, according to one embodiment of the present invention, by generating sleep state information by multimodally adding other information (e.g., user's biometric information) to each sound information corresponding to a window, more accurate sleep analysis can be achieved.

Differences in deep neural networks for real-time sleep event detection

In order to detect a sleep event occurring during sleep according to an embodiment of the present invention, the deep neural network structure for analyzing the sleep stage described above can be modified and used. Specifically, sleep stage analysis requires time-series learning of sleep sounds, but sleep event detection occurs on average between 10 and 60 seconds, so it is sufficient to accurately detect 1 or 2 epochs of 30 seconds. Therefore, the deep neural network structure for analyzing sleep stages according to an embodiment of the present invention can reduce the input and output amounts of the deep neural network structure for analyzing sleep stages. For example, if the deep neural network structure for analyzing sleep stages processes 40 Mel Spectrograms and outputs a sleep stage of 20 epochs, the deep neural network structure for detecting sleep events can process 14 Mel Spectrograms and output a sleep event label of 10 epochs. Here, sleep event labels may include, but are not limited to, no event, apnea, hypopnea, snoring, tossing and turning, etc.

In addition, according to an embodiment of the present invention, a deep neural network structure for detecting sleep events occurring during sleep may include a feature extraction model and a feature classification model. Specifically, the feature extraction model extracts features of sleep events found in each Mel Spectrogram, and the feature classification model detects multiple epochs to find epochs containing sleep events, analyzes neighboring features, and predicts and classifies the type of sleep event in a time series manner.

Class Weights for Real-Time Sleep Event Detection

According to one embodiment of the present invention, a method for detecting sleep events occurring during sleep may assign class weights to solve the class imbalance problem of each sleep event. Specifically, among sleep events occurring during sleep, "no event" may have a dominant influence on the entire sleep length, which may cause a decrease in sleep event learning efficiency. Therefore, by assigning a higher weight than "no event" to other sleep events, learning efficiency and accuracy may be improved. For example, if sleep event classes are classified into three types, "no event", "apnea", and "hypopnea", a weight of 1.0 may be assigned to "no event", 1.3 to "apnea", and 2.1 to "hypopnea" in order to reduce the influence of "no event" on learning. The specific numerical values related to the weights are merely examples and are not limited thereto.

Consistency training for real-time sleep event detection

FIG. 37 is a diagram for explaining consistency training according to an embodiment of the present invention. According to an embodiment of the present invention, the step of detecting a sleep event occurring during sleep may utilize consistency training, as illustrated in FIG. 14, in order to detect a sleep event occurring during sleep in a home environment and a noisy environment as described above. Consistency training is a type of semi-supervised learning model, and consistency training according to an embodiment of the present invention may be a method of performing learning using data to which noise is intentionally added and data to which noise is not intentionally added.

In addition, Consistency Training according to an embodiment of the present invention may be a method of performing learning by generating data of a virtual sleep environment using noise of a target environment.

According to an embodiment of the present invention, the noise intentionally added may be noise of the target environment, wherein the noise of the target environment may be noise obtained in an environment other than a polysomnography test, for example. Specifically, in detecting sleep events, various noises may be added by adjusting the SNR and the type of noise in order to make it similar to the environment of an actual user. Through this, it is possible to collect and learn about the types of noise obtained in various laboratories and noise occurring in an actual home environment.

According to embodiments of the present invention, for convenience, data to which noise has been intentionally added is referred to as corrupted data. Corrupted data may preferably mean data to which noise in a target environment has been intentionally added.

Also, for convenience, we refer to data to which noise has not been intentionally added as Clean data. Here, Clean data may actually contain noise, although noise has not been intentionally added.

Clean data used in Consistency Training according to one embodiment of the present invention may be data acquired in a specific environment (preferably, a polysomnography environment), and corrupted data may be data acquired in another environment or target environment (preferably, an environment other than a polysomnography environment).

Corrupted data according to one embodiment of the present invention may be data in which noise acquired in a different environment or target environment (preferably, an environment other than a polysomnography test) is intentionally added to Clean data.

According to the present invention, in Consistency Training, learning can be performed to achieve consistent prediction by defining a loss function or consistency loss so that the outputs of each are the same when Clean data and Corrupted data are input to the same deep learning model.

Home Noise Consistency Training

According to one embodiment of the present invention, detection of sleep events (e.g., apnea, hypopnea, snoring, sleep talking, etc.) occurring during sleep may include home noise consistency training. Home noise consistency training may enable the model to operate robustly even under noise at home. Home noise consistency training may enable the model to output similar predictions regardless of whether there is noise, thereby making the model robust to noise.

According to one embodiment of the present invention, detection of sleep events occurring during sleep can proceed with consistency learning in a home environment. Consistency learning in a home environment can include a consistency loss function. For example, consistency loss can be defined as the mean square error (MSE) between the prediction of a clear sleep breathing sound and the prediction of a corrupted version of that sound.

According to one embodiment of the present invention, coherence learning in a home environment can randomly sample data from training noise and add noise to clean sleep breathing sounds with a random SNR between -20 and 5 to generate corrupted sounds.

According to one embodiment of the present invention, consistency learning in a home environment can be learned so that the length of the input sequence is 14 epochs and the length of the total sampled noise is 7 minutes or more. Through this, the detection of sleep events according to the present invention detects information within a shorter period of time compared to the sleep stage analysis according to the present invention, so that the accuracy of sleep event detection can be increased.

Regression analysis for estimating AHI values from event detection

FIG. 41 is a diagram for explaining a linear regression analysis function utilized to analyze AHI, an index of sleep apnea occurrence, through sleep events occurring during sleep according to one embodiment of the present invention.

According to one embodiment of the present invention, the AHI index, which means the number of respiratory events occurring per unit time (e.g., 1 hour), can be analyzed independently of the length of an epoch for one sleep stage analysis, separately from the sleep stage analysis. Specifically, two or three short sleep events may be included during one epoch, and one long sleep event may be included during multiple epochs. According to one embodiment of the present invention, a regression analysis function may be used to estimate the number of times an actual event has occurred from the number of epochs in which sleep events occurring during sleep have occurred. For example, a RANSAC (Random Sample Consensus) regression analysis model may be used. The RANSAC regression analysis model is one of the methods for estimating the parameters of an approximate model (Fitting Model), and is a method of randomly extracting sample data and then selecting a model with the greatest match.

Multi-task analysis via multi-head

According to one embodiment of the present invention, a method for analyzing a sleep state may include analysis using a deep learning model. The deep learning model according to one embodiment of the present invention may perform multi-task learning and/or multi-task analysis. Specifically, multi-task learning and multi-task analysis may simultaneously learn tasks according to the embodiments of the present invention described above (e.g., multimodal learning, real-time sleep event analysis, sleep stage analysis, etc.).

According to one embodiment of the present invention, a deep learning model for analyzing sleep states can perform multi-task learning and multi-task analysis. Specifically, for multi-task learning and analysis, the deep learning model can adopt a structure having multiple heads. Each of the multiple heads can be in charge of a specific task or task (e.g., multimodal learning, real-time sleep event analysis, sleep stage analysis, etc.). For example, the deep learning model can have a structure having a total of three heads, a first head, a second head, and a third head, where the first head performs inference and/or classification of sleep stage information, the second head performs detection and/or classification of sleep apnea and hypopnea among sleep events, and the third head performs detection and classification of snoring among sleep events. The specific description of the specific tasks or tasks of the above-described heads is only an example for explaining the present invention and is not limited thereto. The deep learning model according to the present invention can perform multi-task learning and analysis through a structure having multiple heads, and can optimize multiple tasks or specific tasks by increasing data efficiency.

Effectiveness of sleep analysis methods

According to the present invention, it was confirmed that the results of a sleep analysis model using sleep sound information as input were very accurate when compared with the results of polysomnography (PSG).

Existing sleep analysis models predict sleep stages using ECG (Electrocardiogram) or HRV (Heart Rate Variability) as input, but the present invention can analyze and infer sleep stages by converting sleep sound information into frequency domain or frequency-time domain information, spectrogram, or mell spectrogram as input. Therefore, since sleep sound information is converted into frequency domain or frequency-time domain information, spectrogram, or mell spectrogram as input, unlike existing sleep analysis models, sleep stages can be sensed or acquired in real time by analyzing the specificity of sleep patterns.

FIG. 26a and FIG. 26b are graphs verifying the performance of a sleep analysis method according to the present invention, and are drawings comparing the results of polysomnography (PSG) and the results of analysis using an AI algorithm according to the present invention (AI result).

As illustrated in FIG. 26a, the sleep analysis results obtained according to the present invention are not only highly consistent with polysomnography, but also include more precise and meaningful information related to sleep stages (Wake, Light, Deep, REM).

The hypnogram shown at the bottom of Figure 27 shows the probability of belonging to one of the four classes (Wake, Light, Deep, REM) for each 30 seconds when predicting sleep stages by inputting user sleep sound information. Here, the four classes represent the awake state, light sleep state, deep sleep state, and REM sleep state, respectively.

Fig. 26c is a graph verifying the performance of the sleep analysis method according to the present invention, and is a diagram comparing the polysomnography (PSG) result (PSG result) with the analysis result (AI result) using the AI algorithm according to the present invention in relation to sleep apnea and hypopnea. The hypnogram illustrated at the very bottom of Fig. 26c indicates the probability of which of the two diseases (sleep apnea, hypopnea) it belongs to every 30 seconds when predicting a sleep disease by receiving user sleep sound information. When using the sleep analysis according to the present invention, as illustrated in Fig. 26c, the sleep state information obtained according to the present invention is not only highly consistent with the polysomnography, but also includes more precise analysis information related to apnea and hypopnea.

The present invention can analyze a user's sleep analysis in real time and identify the point where a sleep event (sleep apnea, sleep hyperapnea, sleep hypoapnea) occurs. If a stimulus (tactile, auditory, olfactory stimulus, etc.) is provided to the user at the moment when the sleep event occurs, the sleep event can be temporarily alleviated. That is, the present invention can stop the user's sleep event and reduce the frequency of the sleep event based on accurate event detection related to the sleep event. In addition, according to the present invention, there is also an effect that highly accurate sleep analysis is possible by performing sleep analysis in a multimodal manner.

How to provide sleep data interpretation content by category of sleep information

Categories of sleep interpretation and sleep information

According to embodiments of the present invention, the user's sleep state information may include at least one of sleep stage information, sleep stage probability information, sleep event information, and sleep event probability information. According to one embodiment of the present invention, content may be provided that interprets (or evaluates) the user's sleep data based on the user's sleep state information obtained during one or more sleep sessions.

The user's sleep data may be sleep data belonging to at least one of the following categories: Sleep Apnea category, Sleep Onset Latency category, First Cycle Sleep Quality category, REM Latency category, REM Ratio category, Deep Ratio category, Wake Time after Sleep Onset (WASO) category, Number of Awakening category, and Total Sleep Time category.

Create and provide sleep data interpretation content based on lookup tables

FIG. 31A is a diagram for explaining a method for generating interpretation content of sleep data based on a lookup table according to one embodiment of the present invention.

According to one embodiment of the present invention, the step of generating interpretation content of sleep data based on the lookup table disclosed in FIG. 31A includes the step of classifying the user's sleep characteristics (800-2) and the step of extracting the sleep data interpretation content (840-2) based on the lookup table corresponding to the user's sleep characteristics. For example, the user's sleep characteristics (800-2) may be classified into at least one of the first sleep characteristics (822-2), the second sleep characteristics (824-2), the third sleep characteristics (826-2), and the fourth sleep characteristics (828-2), and the interpretation content of sleep data may be extracted based on at least one of the first lookup table (842-2), the second lookup table (844-2), the third lookup table (846-2), and the fourth lookup table (848-2) each corresponding to the classified sleep characteristics.

Generating and providing sleep data interpretation content based on large-scale language models

FIG. 31b is a diagram for explaining a method for generating interpretation content of sleep data based on a large-scale language model according to one embodiment of the present invention.

According to one embodiment of the present invention, the step of generating interpretation content of sleep data based on the large-scale language model disclosed in FIG. 31B includes the step of classifying the user's sleep characteristics (900-2) and the step of extracting the sleep data interpretation content (940-2) based on the large-scale language model by using the classified user sleep characteristics (920-2) as input to the large-scale language model (930-2). For example, the user's sleep characteristics (900-2) may be classified into at least one of the first sleep characteristics (922-2), the second sleep characteristics (924-2), the third sleep characteristics (926-2), and the fourth sleep characteristics (928-2), and at least one of the classified sleep characteristics may be input to the large-scale language model (930-2). The sleep data interpretation content may be extracted by inputting data output from the large-scale language model (930-2) into at least one of the first large-scale language model, the second large-scale language model, the third large-scale language model, and the fourth large-scale language model (940-2).

Description of the flow chart of Figure 31c

FIG. 31c is a flowchart of a method for providing non-numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

As illustrated in FIG. 31c, a method for providing non-numerical interpretation content of sleep data may include a sleep information acquisition step (S100-2), a non-numerical interpretation content generation step of sleep data (S120-2), and a step of displaying the generated sleep data interpretation content (S140-2). The interpretation content of sleep data includes at least two words, and the generated interpretation content may not be expressed in a numerical manner.

According to one embodiment of the present invention, a method for providing non-numerical interpretation content of sleep data may include a sleep log information storage step of storing sleep log information related to an account assigned to a user in a memory.

According to one embodiment of the present invention, in a method for providing non-numerical interpretation content of sleep data, the sleep information acquisition step (S100-2) may include a sleep information conversion step for converting sleep sound information into information in a frequency domain or a frequency-time domain.

According to one embodiment of the present invention, in a method for providing non-numerical interpretation content of sleep data, the sleep information acquisition step (S100-2) may further include a sleep information inference step for inferring information about sleep by using sleep sound information as input to a deep learning model.

According to one embodiment of the present invention, in a method for providing non-numerical interpretation content of sleep data, the non-numerical interpretation content generation step (S120-2) of sleep data may include a first sleep data interpretation content generation step of generating a text having high visibility among the non-numerical interpretation content of sleep data, and a second sleep data interpretation content generation step of generating a text constituting the non-numerical interpretation content of the user's sleep data.

According to one embodiment of the present invention, in a method for providing non-numerical interpretation content of sleep data, the non-numerical interpretation content generation step (S120-2) of sleep data may include a third sleep data interpretation content generation step of generating advice text based on an evaluation of sleep.

According to one embodiment of the present invention, the non-numerical interpretation content generation step (S120-2) of sleep data may further include a step of generating interpretation content of sleep data based on a lookup table. The step of generating interpretation content of sleep data based on the lookup table may include a lookup table user sleep characteristic classification step of classifying sleep characteristics (800-2) of a user based on sleep information, and a sleep data interpretation content extraction step based on a lookup table (840-2) corresponding to the classified sleep characteristics (820-2).

According to one embodiment of the present invention, the non-numerical interpretation content generation step (S120-2) of sleep data may further include a step of generating interpretation content of sleep data based on a large-scale language model. The step of generating interpretation content of sleep data based on a large-scale language model may include a step of classifying characteristics (900-2) of a user's sleep based on sleep information, and a step of extracting interpretation content (940-2) of sleep data based on a large-scale language model that is output by using the classified sleep characteristics (920-2) as input to a large-scale language model (930-2).

According to one embodiment of the present invention, the step (S140-2) of displaying the generated sleep data interpretation content may further include a step of generating a graphical user interface. Or, the step may further include a user sleep graph generation step of generating a graph of sleep stages within the user's sleep period based on sleep information.

Description of the flow chart of Figure 31d

FIG. 31d is a flowchart of a method for providing numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

As illustrated in FIG. 31d, a method for providing numerical interpretation content based on sleep data of a user according to an embodiment of the present invention may include a sleep information acquisition step (S200-2), a non-numerical interpretation content generation step of sleep data (S220-2), a numerical interpretation content generation step of sleep data (S240-2), and a step of displaying the generated sleep data interpretation content (S260-2). The sleep data interpretation content according to an embodiment of the present invention includes at least one of sleep data interpretation content (non-numerical method) including at least two words, and sleep numerical evaluation (numerical method).

According to one embodiment of the present invention, a method of generating one or more graphical user interfaces representing an evaluation regarding sleep may include a sleep log information storing step of storing sleep log information related to an account assigned to a user in memory.

According to one embodiment of the present invention, in a method for generating one or more graphical user interfaces representing an evaluation regarding sleep, the sleep information acquisition step (S200-2) may include a sleep information conversion step for converting sleep sound information into information in a frequency domain or a frequency-time domain.

According to one embodiment of the present invention, in a method for generating one or more graphical user interfaces representing an evaluation regarding sleep, the sleep information acquisition step (S200-2) may further include a sleep information inference step for inferring information regarding sleep by using sleep sound information as input to a deep learning model.

According to one embodiment of the present invention, in a method for generating one or more graphical user interfaces representing an evaluation regarding sleep, a non-numerical interpretation content generation step (S220-2) of sleep data may include a first sleep data interpretation content generation step for generating text having high visibility among the evaluation regarding sleep, and a second sleep data interpretation content generation step for generating text constituting the user's evaluation regarding sleep.

According to one embodiment of the present invention, in a method for generating one or more graphical user interfaces representing an evaluation regarding sleep, the non-numerical interpretation content generation step (S220-2) of sleep data may include a third interpretation content generation step of generating advice text based on the evaluation regarding sleep.

According to one embodiment of the present invention, the non-numerical interpretation content generation step (S220-2) of sleep data and the numerical interpretation content generation step (S240-2) of sleep data may further include a step of generating interpretation content of sleep data based on a lookup table, as disclosed in FIG. 31A. The step of generating interpretation content of sleep data based on the lookup table may include a lookup table user sleep characteristic classification step of classifying sleep characteristics (800-2) of a user based on sleep information, and a sleep data interpretation content extraction step based on a lookup table (840-2) corresponding to the classified sleep characteristics (820-2).

According to one embodiment of the present invention, the non-numerical interpretation content generation step (S220-2) of sleep data and the numerical interpretation content generation step (S240-2) of sleep data may further include a step of generating interpretation content of sleep data based on a large-scale language model, as disclosed in FIG. 31b. The step of generating interpretation content of sleep data based on the large-scale language model may include a large-scale language model user sleep characteristic classification step of classifying sleep characteristics (900-2) of a user based on sleep information, and a step of extracting interpretation content (940-2) of sleep data based on the large-scale language model as an output using the classified sleep characteristics (920-2) as an input of the large-scale language model (930-2).

According to one embodiment of the present invention, the graphical user interface display step (S260-2) may include a user sleep graph generation step that generates a graph of sleep stages within a user's sleep period based on sleep information.

Example of providing sleep data interpretation content

Non-numerical sleep data interpretation content

FIGS. 33A through 33C are exemplary diagrams illustrating one or more graphical user interfaces representing non-numerical interpretation content generated based on a user's sleep data according to embodiments of the present invention.

FIG. 33A is an exemplary diagram illustrating one or more graphical user interfaces representing non-numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention. In one embodiment of the present invention illustrated in FIG. 33A, one or more graphical user interfaces representing a numerical evaluation of the user's sleep may include interpretation content (400-2) of first sleep data, which is a text having high visibility among the evaluations of the user's sleep, interpretation content (420-2) of second sleep data, which is a text constituting the evaluation of the user's sleep, and an image related to a dream representing the user's sleep.

According to one embodiment of the present invention, for example, the interpretation content (400-2) of the first sleep data may be a text with high visibility, such as “I wish I had slept more” or “I feel sleepy”, when an evaluation of the user’s sleep is required regarding the user’s sleep. The interpretation content (400-2) of the first sleep data may be a text generated by the lookup table disclosed in FIG. 31a, but is not limited thereto. The interpretation content (400-2) of the first sleep data may be a text generated by the large-scale language model disclosed in FIG. 31b, but is not limited thereto. For example, the interpretation content (420-2) of the second sleep data may be a text that constitutes an evaluation of the user’s sleep, such as “You seem very busy these days. How about getting some sleep tonight?” or “I don’t have enough sleep. How about getting some sleep while resting today?” when the user’s sleep is insufficient. The interpretation content (420-2) of the second sleep data may be text generated by the lookup table disclosed in Fig. 31a, but is not limited thereto. The interpretation content (420-2) of the second sleep data may be text generated by the large-scale language model disclosed in Fig. 31b, but is not limited thereto.

FIG. 33b is an exemplary diagram illustrating one or more graphical user interfaces representing non-numerical interpretation content generated based on the user's sleep data according to one embodiment of the present invention. In one embodiment of the present invention illustrated in FIG. 33b, one or more graphical user interfaces representing a numerical evaluation of the user's sleep may include interpretation content (402-2) of first sleep data, which is a highly noticeable text among the evaluations of the user's sleep, and interpretation content (422-2) of second sleep data, which is a text constituting the evaluation of the user's sleep. For example, the interpretation content (402-2) of the first sleep data may be highly noticeable text such as "stress-relieving sleep" or "sweet sleep" when the user's sleep needs to be evaluated as good. The interpretation content (402-2) of the first sleep data may be text generated by the lookup table disclosed in FIG. 31a, but is not limited thereto. The interpretation content (402-2) of the first sleep data may be text generated by the large-scale language model disclosed in FIG. 31b, but is not limited thereto.

According to one embodiment of the present invention, for example, the interpretation content (422-2) of the second sleep data may be text constituting an evaluation of the user's sleep, such as "The REM sleep ratio was high! It is sleep that helps relieve stress and develop creativity", "The REM sleep ratio was high! Good sleep makes the day enjoyable." The interpretation content (422-2) of the second sleep data may be text generated by the lookup table disclosed in FIG. 31a, but is not limited thereto. The interpretation content (422-2) of the second sleep data may be text generated by the large-scale language model disclosed in FIG. 31b, but is not limited thereto.

FIG. 33C is an exemplary diagram illustrating one or more graphical user interfaces representing non-numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention. In one embodiment of the present invention illustrated in FIG. 33C, one or more graphical user interfaces representing a numerical evaluation of the user's sleep may include interpretation content (440-2) of third sleep data, which is advice text based on the evaluation of the user's sleep, and a user sleep graph (460-2), which is a graph of sleep stages within the user's sleep period based on the user's sleep information. For example, the interpretation content (440-2) of the third sleep data may be advice text related to sleep, such as "There is a saying that you should utilize REM sleep to develop creativity," to advise the user about the ratio of REM sleep in relation to his/her sleep. The interpretation content (440-2) of the third sleep data may be text generated by the lookup table disclosed in FIG. 31A, but is not limited thereto. The interpretation content (440-2) of the first sleep data may be text generated through a large-scale language model disclosed in Fig. 31b, but is not limited thereto.

According to one embodiment of the present invention, for example, the user sleep graph (460-2) may be in the form of a hypnogram that represents the sleep stages during the user's sleep based on the user's sleep information. As disclosed in Fig. 33c, the sleep stages may be expressed as four stages of REM sleep, wakefulness, general sleep, and deep sleep, but are not limited thereto and may be arbitrarily set by the user.

Content on numerical sleep data interpretation

FIGS. 33d and 33e are exemplary diagrams illustrating one or more graphical user interfaces representing numerical interpretation content generated based on a user's sleep data according to embodiments of the present invention.

FIG. 33d is an exemplary diagram illustrating one or more graphical user interfaces representing numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

As illustrated in FIG. 33d, one or more graphical user interfaces (500-2) representing numerical sleep data interpretation content regarding the user's sleep may include at least one of interpretation content (404-2) of first sleep data, which is a highly noticeable text among evaluations regarding the user's sleep, a sleep score (520-2) which is an evaluation regarding the user's sleep, a sleep time (540-2) of the user, and a sleep time (560-2) of the user.

FIG. 33e is an exemplary diagram illustrating one or more graphical user interfaces representing numerical interpretation content generated based on a user's sleep data according to one embodiment of the present invention.

As illustrated in FIG. 33e, one or more graphical user interfaces (502-2) representing numerical sleep data interpretation content regarding a user's sleep according to one embodiment of the present invention may include a sleep score (522-2), which is an evaluation of the user's sleep, information regarding the user's sleep and waking up time (542-2), and information regarding the user's sleep time (562-2).

Lookup table for providing sleep interpretation by category of sleep information

An electronic device according to one embodiment of the present invention can provide content for interpreting sleep data based on sleep data by category of sleep information. In an embodiment, the content for interpreting sleep data may be text information composed of strings. The content for interpreting sleep data by category of sleep information may be provided based on a lookup table prepared in advance.

According to one embodiment of the present invention, content for interpreting a user's sleep data may be provided based on at least one of information comparing the user's sleep data generated over a predetermined period of time in the past, information comparing the sleep data with medical standards, and information comparing the sleep data with other users generated over a predetermined period of time in the past.

According to one embodiment of the present invention, interpreting sleep data based on information compared to user's sleep data generated over a predetermined period of time in the past means interpreting sleep data in a past sleep session based on how different it is from the average of sleep data generated over a predetermined period of time in the past (e.g., 30 days). In this case, the more it deviates from the average, the more important it may be considered.

Interpreting sleep data based on information compared to a medical standard according to one embodiment of the present invention means that, if the sleep data is a category of sleep information that can define a standard (or normal) category in the medical field related to sleep, the sleep data belonging to the category of sleep information is interpreted based on whether it is within or outside the standard (or normal) category in the medical field. In this case, if there is little difference from the range of the medical average, this standard may be considered relatively unimportant. Alternatively, if the difference from the range of the medical average is sufficiently large, this standard may be considered relatively important.

Interpreting sleep data based on information compared to sleep data of other users generated over a predetermined period of time in the past according to one embodiment of the present invention means interpreting sleep data based on how sleep data in a past sleep session differs from the average of sleep data of other users generated over a predetermined period of time in the past (e.g., 30 days). In this case, the more it deviates from the average, the more important it may be considered.

According to one embodiment of the present invention, content for interpreting the user's sleep data may be provided as a string text composed of one or more words. In addition, according to one embodiment of the present invention, it may be provided as a summary text that summarizes the string text composed of one or more words. Alternatively, the string text composed of one or more words and the summary text may be provided together.

Below, specific examples of content for interpreting sleep data by category of each sleep information according to an embodiment of the present invention are described.

Sleep data interpretation content in the Sleep Apnea category

According to the present invention, sleep apnea refers to a symptom of stopping breathing during sleep. The degree of sleep apnea can be expressed based on the Apnea-Hypopnea Index (AHI). AHI is an index showing the average number of apneas or hypopneas that occur per hour during sleep. If the respiratory flow signal decreases to less than 10% of the reference value and continues for more than 10 seconds, it is classified as apnea, and if the respiratory flow signal decreases to less than 30% of the reference value and continues for more than 10 seconds and the oxygen saturation decreases by more than 3%, it is classified as hypopnea. If the AHI is less than 5, sleep breathing can be judged as stable. If the AHI is 5 or more but less than 15, sleep breathing can be judged as normal or slightly unstable. If the AHI is 15 or more, sleep breathing can be judged as unstable. If the AHI value is lower than in the past, it can be judged as a more improved (better) index. If the AHI value is higher than in the past, it can be judged as a more deteriorated (or worse) indicator.

[Table 1] is a lookup table in which various cases of sleep data of the Sleep Apnea category that can be measured according to embodiments of the present invention are recorded. [Table 2] is a lookup table in which sleep data interpretation contents of the Sleep Apnea category that can be provided according to each case shown in [Table 1] are recorded. Hereinafter, with reference to [Table 1] and [Table 2], various sleep data interpretation contents that can be provided based on AHI data measured according to embodiments of the present invention are described.

[Table 1]

[Table 2]

Sleep Data Interpretation Content Example 1

According to one embodiment of the present invention, when the user's sleep apnea data is compared with a medical standard, an AHI index of stable sleep breathing is shown (AHI value less than 5), and when compared with the user's sleep apnea data generated over a predetermined period of time in the past, if the AHI index was normal in the past (e.g., AHI value 5 or more and less than 15) but currently shows an improved index, it can be determined that it is "calm breathing", and therefore interpretation content that "the stability of sleep breathing was normal but improved" can be provided. Specifically, interpretation content that "the breathing pattern while sleeping has improved compared to the past n days" can be provided. Here, n is a natural number.

According to one embodiment of the present invention, information such as "If your breathing pattern is regular while you sleep, sufficient oxygen is supplied, helping your body recover" or "If your breathing is easy while you sleep, your sleep quality is better." Information related to the topic "The effect of maintaining regular breathing on our body" can also be provided.

Sleep Data Interpretation Content Example 2

According to one embodiment of the present invention, when the user's sleep apnea data is compared with a medical standard, an AHI index of stable sleep breathing is shown (AHI value less than 5), and when compared with the user's sleep apnea data generated over a predetermined period of time in the past, if an AHI index of stable sleep breathing is shown both in the past and present, it can be determined that the user is "calmly breathing", and therefore, interpretation content indicating that "the stability of sleep breathing is consistently good" can be provided. Specifically, interpretation content indicating that "breathing while sleeping has been consistently stable for the past n days" can be provided. Here, n is a natural number.

According to one embodiment of the present invention, information such as "A regular sleep breathing pattern is evidence of healthy sleep" may also be provided. Information related to the topic "The effect of breathing during sleep on sleep" may also be provided.

Sleep Data Interpretation Content Example 3

According to one embodiment of the present invention, when the user's sleep apnea data is compared with a medical standard and the AHI index is normal (AHI value of 5 or more and less than 15), and when compared with the user's sleep apnea data generated over a predetermined period of time in the past, if the index is normal both in the past and present, it can be determined that the user has "slightly unstable breathing", and therefore, interpretation content such as "the breathing pattern of the last sleep session was slightly unstable" can be provided. Specifically, interpretation content such as "the breathing pattern of the last n days of sleep sessions was slightly unstable" can be provided. Here, n is a natural number.

According to one embodiment of the present invention, It may also provide interpretation content that says, "If it doesn't interfere with your daily life or cause you to experience insomnia, you don't have to worry too much."

According to one embodiment of the present invention, information relating to the topic “steps that can be taken for sleep breathing to improve the quality of sleep” may be provided.

Sleep Data Interpretation Content Example 4

According to one embodiment of the present invention, when the user's sleep apnea data is compared with a medical standard, if the AHI index is normal (AHI value 5 or more and less than 15), and when compared with the user's sleep apnea data generated over a predetermined period of time in the past, if the AHI index was good in the past (e.g., AHI value less than 5) but currently shows a lower index compared to that, it can be determined that the user has "slightly unstable breathing", and therefore, interpretation content such as "the breathing pattern of the last sleep session was slightly unstable" can be provided. Specifically, interpretation content such as "the breathing was slightly unstable in the last sleep session compared to the last n days" can be provided. Here, n is a natural number.

According to one embodiment of the present invention, it is possible to provide interpretation content such as, "If it does not interfere with your daily life or you do not experience insomnia, you do not need to worry too much."

According to one embodiment of the present invention, information such as "Your breathing pattern while sleeping is slightly unstable, but it is within a relatively normal range" or "For most people, it does not interfere with their daily activities." Information related to topics such as "What it means to have unstable breathing while sleeping" or "Daily practices to improve your breathing while sleeping" may also be provided.

Sleep Data Interpretation Content Example 5

According to one embodiment of the present invention, if the user's sleep apnea data shows a normal AHI index (AHI value of 5 or more and less than 15) when compared with the medical standard, and if the user's sleep apnea data generated over a predetermined period of time shows an AHI index of unstable sleep breathing in the past (AHI value of 15 or more) but currently shows an improved index compared to before, it can be determined that "unstable breathing is gradually improving", and therefore interpretation content such as "breathing during sleep is still a little unstable, but is gradually improving" can be provided. Specifically, interpretation content such as "breathing was a little unstable in the last sleep session, but sleep breathing is gradually improving compared to the past n days" can be provided. Here, n is a natural number.

According to one embodiment of the present invention, information may be provided relating to the topic of "ways to practice in daily life for better breathing during sleep."

Sleep Data Interpretation Content Example 6

According to one embodiment of the present invention, if the user's sleep apnea data exhibits an AHI index of unstable sleep breathing (AHI value of 15 or higher) when compared with a medical standard, and if the user's sleep apnea data generated over a predetermined period of time in the past exhibits an AHI index of stable sleep breathing (AHI value of less than 5) or an AHI index of slightly unstable sleep breathing (AHI value of 5 or higher and less than 15) in the past, but currently exhibits a lower index than before, it can be determined that "sleep breathing is gradually worsening", and therefore interpretation content such as "breathing was unstable in the last sleep session" can be provided. Specifically, interpretation content such as "breathing was unstable in the last sleep session compared to the past n days" can be provided. Here, n is a natural number.

According to one embodiment of the present invention, information such as "If breathing is unstable during sleep, it can interfere with sleep and make you feel uncomfortable when waking up" or "A temporary sleep breathing instability pattern can appear due to environmental factors such as environmental changes, stress, drinking, and smoking." In addition, information related to the topic of "Lifestyle habits or eating habits that can be practiced in daily life for better breathing during sleep" or "How to deal with sleep breathing instability that persists for a certain period of time" can be provided.

Sleep Data Interpretation Content Example 7

According to one embodiment of the present invention, if the user's sleep apnea data exhibits an AHI index of unstable sleep breathing (AHI value of 15 or higher) when compared with a medical standard, and the AHI index of unstable sleep breathing is maintained both in the past and present when compared with the user's sleep apnea data generated over a predetermined period of time in the past, it can be determined as "unstable sleep breathing", and therefore interpretation content of "breathing was unstable in the last sleep session" can be provided. Specifically, interpretation content of "sleep breathing has been continuously unstable for the past n days" can be provided. Here, n is a natural number.

According to one embodiment of the present invention, information such as "If you feel tired immediately after waking up, it may be because your breathing was unstable during sleep." In addition, information related to the topic "How to deal with unstable breathing during sleep that continues for a certain period of time" may be provided.

Sleep Data Interpretation Content in the Sleep Onset Latency Category

According to the present invention, sleep onset latency (SOL) or sleep onset latency is the time taken to gradually induce sleep from a fully awake state and enter sleep, and can generally be defined as the time taken to transition to stage 1 sleep (or shallow sleep), which is the lightest stage among non-REM sleep stages.

According to embodiments of the present invention, sleep onset latency or sleep onset latency (SOL) may mean the time it takes to transition from a fully awake state to at least one of a light sleep stage, a deep sleep stage, and a REM sleep stage.

The sleep onset delay time can be an indicator for the diagnosis and evaluation of hypersomnia or sleep deprivation through the Multiple Sleep Latency Test (MSLT). The MSLT is known as a representative test method with high reliability among the techniques used to quantify the degree of sleep deprivation.

On the MSLT, a sleep onset delay of 0 to 5 minutes may be assessed as severe sleep deprivation. A sleep onset delay of 5 to 10 minutes may be assessed as troublesome sleep deprivation. A sleep onset delay of 10 to 15 minutes may be assessed as manageable sleep deprivation. A sleep onset delay of 15 to 20 minutes may be assessed as little or no sleep deprivation.

According to embodiments of the present invention, if the measured sleep onset delay time is less than 30 minutes, the level of arousal is not high, but if it is 30 minutes or more, the level of arousal can be evaluated as high. Therefore, maintaining the sleep onset delay time within an appropriate range is a method for improving the quality of sleep. According to embodiments of the present invention, when the standard sleep onset delay time is set to 30 minutes, it can be determined as a medically good indicator when the measured sleep onset delay time is less than 30 minutes, and it can be determined as a medically bad (or bad) indicator when it is 30 minutes or more.

[Table 3] is a lookup table in which various cases of sleep data of the Sleep Onset Latency category that can be measured according to embodiments of the present invention are recorded. [Table 4] is a lookup table in which sleep data interpretation contents of the Sleep Onset Latency category that can be provided according to each case shown in [Table 3] are recorded. Hereinafter, various sleep data interpretation contents that can be provided based on SOL data measured according to embodiments of the present invention will be described with reference to [Table 3] and [Table 4].

[Table 3]

[Table 4]

Sleep Data Interpretation Content Example 1

According to one embodiment of the present invention, if the user's sleep delay time data shows a good indicator (SOL is less than 30 minutes) and, when compared with the user's sleep delay time data generated over a predetermined period of time in the past, the SOL is shortened by 20 minutes or more compared to the past, it can be determined that the "sleep delay time is shorter than the standard", and interpretation content such as "a night when I fell asleep quickly", "a night when I fell asleep as soon as I laid down", and "a night when I fell asleep in an instant" can be provided. Specifically, interpretation content such as "the time to fall asleep was shorter in the last sleep session compared to the past n days" can be provided. Here, n is a natural number.

According to one embodiment of the present invention, information such as "If you fall asleep quickly, you may feel less fatigue in your daily life and your sleep efficiency may also improve" or "The last sleep session may be evaluated as a sleep with good sleep efficiency." may also be provided.

According to one embodiment of the present invention, in light of the content that "it is also desirable to determine whether the environment before falling asleep of the last sleep session was different from usual," information related to "the optimal environment for maintaining the current sleep delay time" or "the method for maintaining the current sleep delay time" may be provided.

Sleep Data Interpretation Content Example 2

According to one embodiment of the present invention, if the user's sleep delay time data shows a good indicator (SOL is less than 30 minutes) and, when compared with the user's sleep delay time data generated over a predetermined period of time in the past, the SOL is shortened by less than 20 minutes compared to the past, it can be determined that "the sleep delay time is continuously shorter than the standard", and interpretation content such as "a night when I fell asleep quickly", "a night when I fell asleep as soon as I laid down", and "a night when I fell asleep in an instant" can be provided. Specifically, interpretation content such as "compared to the past n days, there was no significant change in the time to fall asleep in the last sleep session" can be provided. Here, n is a natural number.

According to one embodiment of the present invention, information such as "If you fall asleep quickly, you can feel less fatigue in daily life and improve sleep efficiency" or "If you fall asleep quickly, you can feel less fatigue in daily life and improve sleep efficiency" may also be provided.

According to one embodiment of the present invention, in light of the content that "the last sleep session can be evaluated as a sleep with good sleep efficiency," information related to "an optimal environment for maintaining the sleep onset delay time as it is now" or "a method for maintaining the sleep onset delay time as it is now" may be provided.

Sleep Data Interpretation Content Example 3

According to one embodiment of the present invention, if the user's sleep delay time data shows a poor (or bad) indicator (SOL of 30 minutes or more) and the SOL is shortened by 20 minutes or more compared to the user's sleep delay time data generated over a predetermined period of time in the past, it can be determined that "the sleep delay time is longer than the standard, but shorter than before," and interpretation content such as "a night when it was difficult to fall asleep," "a night when I couldn't fall asleep," or "a night when I couldn't fall asleep" can be provided. Specifically, interpretation content such as "it took a long time to fall asleep in the last sleep session, but compared to the past n days, the time it took to fall asleep has become shorter." Here, n is a natural number.

According to one embodiment of the present invention, in light of the content, "It still takes me a long time to fall asleep, but I fell asleep faster than the average for the past n days," information related to "sleep habits that can further shorten the time so that I can fall asleep within 30 minutes" can be provided.

Sleep Data Interpretation Content Example 4

According to one embodiment of the present invention, if the user's sleep delay time data shows a poor (or bad) indicator (SOL of 30 minutes or more) and the SOL is shortened by less than 20 minutes compared to the user's sleep delay time data generated over a predetermined period of time in the past, it can be determined that "the sleep delay time is still longer than the standard", and interpretation content such as "a night when it was difficult to fall asleep", "a night when I couldn't fall asleep", or "a night when I couldn't fall asleep" can be provided. Specifically, interpretation content such as "similar to the last sleep session, it took a long time to fall asleep for the past n days" can be provided. Here, n is a natural number.

According to one embodiment of the present invention, information may be provided regarding “ways to help fall asleep quickly” or “ways to deal with various situations/behaviors in which the time to fall asleep is measured long.”

Sleep Data Interpretation Content Example 5

According to one embodiment of the present invention, if the user's sleep delay time data shows a poor (or bad) indicator (SOL of 30 minutes or more) and, when compared with the user's sleep delay time data generated over a predetermined period of time in the past, the SOL has been extended by 20 minutes or more compared to the past, it can be determined that "the sleep delay time is longer than the standard and has become longer than before," and interpretation content such as "a night when it was difficult to fall asleep," "a night when I couldn't fall asleep," or "a night when I couldn't fall asleep" can be provided. Specifically, interpretation content such as "compared to the past n days, it was more difficult to fall asleep in the last sleep session" can be provided. Here, n is a natural number.

According to one embodiment of the present invention, based on the content that "if it takes more than 30 minutes to fall asleep for a certain period of time and you feel psychologically uneasy, you may suspect insomnia," information related to content such as "content that helps with sleep" or "how to deal with various situations/behaviors in which the time delay in falling asleep is measured long" may be provided.

Sleep data interpretation content in the Wake time After Sleep Onset (WASO) category

Wake time After Sleep Onset (WASO) is the total time that the user is in the wake stage after entering sleep and before waking up completely. WASO does not refer to the frequency of occurrence of wake stages during sleep, but rather the duration of the wake stage. Wake time After Sleep Onset (WASO) can be calculated by adding up the times of the wake stages during sleep that occur after falling asleep. For example, if a user wakes up 5 times in 1 minute after falling asleep and before waking up, WASO can be calculated as 5 minutes. If a user wakes up 40 times in 30 seconds after falling asleep and before waking up, WASO can be calculated as 20 minutes. The quality of a user's sleep can be evaluated through WASO, which is a quantitative indicator.

[Table 5] is a lookup table in which various cases of sleep data of the Wake time After Sleep Onset category that can be measured according to embodiments of the present invention are recorded. [Table 6] is a lookup table in which sleep data interpretation contents of the Wake time After Sleep Onset category that can be provided according to each case shown in [Table 5] are recorded. Hereinafter, with reference to [Table 5] and [Table 6], various sleep data interpretation contents that can be provided based on the Wake time After Sleep Onset data measured according to embodiments of the present invention are described.

[Table 5]

[Table 6]

Sleep Data Interpretation Content Example 1

According to one embodiment of the present invention, if the user's wake-up time data after falling asleep shows a good indicator (WASO is less than 5 minutes), and when compared with the user's wake-up time data generated over a predetermined period of time in the past, if the WASO was 5 minutes or more in the past but is now shorter, it can be determined that "the wake-up time is short" or "the wake-up time has become shorter", and interpretation content such as "the wake-up time last night has decreased" can be provided. Specifically, interpretation content such as "the wake-up time in the last sleep session has decreased compared to the last n days" can be provided. Here, n is a natural number. In addition, content such as "the longest wake-up time in the last sleep session is x minutes" can be provided. Here, x is a real number less than 5.

According to one embodiment of the present invention, information such as “it can be evaluated that sleep efficiency was good while sleeping” can also be provided.

According to one embodiment of the present invention, information related to “daily practice methods for maintaining high sleep efficiency” or “the effect of wake up time after sleep onset (WASO) on sleep quality” may be provided.

Sleep Data Interpretation Content Example 2

According to one embodiment of the present invention, if the user's wake-up time data after falling asleep shows a good indicator (WASO is less than 5 minutes), and when compared with the user's wake-up time data generated over a predetermined period of time in the past, if the WASO was less than 5 minutes in the past, it can be determined that "the wake-up time is short" or "the wake-up time was short in the past and is short now."

According to one embodiment of the present invention, interpretation content such as "The time you were awake last night was short" can be provided. Specifically, interpretation content such as "The time you were awake during sleep sessions was short during the last n days" can be provided. Here, n is a natural number. In addition, content such as "The longest time you were awake during the last sleep session was x minutes" can be provided. Here, x is a real number less than 5.

According to one embodiment of the present invention, interpretation content such as "consistently sleeping peacefully (or sleeping with high sleep efficiency)" or "sleeping can be evaluated as having good sleep efficiency" can be provided. In addition, information related to "the effect of wake time after sleep onset (WASO) on sleep quality" can also be provided.

Sleep Data Interpretation Content Example 3

According to one embodiment of the present invention, if the user's wake-up time data after sleep indicates an average indicator (WASO of 5 minutes or more and less than 15 minutes), and if compared with the user's wake-up time data generated over a predetermined period of time in the past, if the WASO was 15 minutes or more in the past but is now shorter, it can be determined that "the wake-up time is somewhat long" or "the wake-up time was long in the past but is now average."

According to one embodiment of the present invention, interpretation content such as "The time I woke up last night was a little long, but WASO is still good" can be provided. Specifically, interpretation content such as "The time I woke up in the last sleep session decreased compared to the last n days." Here, n is a natural number. In addition, content such as "The longest time I woke up in the last sleep session was x minutes." Here, x is a real number less than 15 can be provided.

According to one embodiment of the present invention, interpretation content such as "It can be evaluated that the efficiency of sleep during sleep is better than in the past" or "You can expect better condition than usual" can be provided. In addition, information related to "How to reduce the longest time spent awake" or "The effect of wake time after sleep onset (WASO) on sleep quality" can be provided.

Sleep Data Interpretation Content Example 4

According to one embodiment of the present invention, if the user's wake-up time data after sleep has shown a normal indicator (WASO of 5 minutes or more and less than 15 minutes), and if compared with the user's wake-up time data generated over a predetermined period of time in the past, if the WASO was also 5 minutes or more and less than 15 minutes in the past, it can be determined that "the wake-up time is somewhat long" or "the wake-up time was normal in the past, but is normal now."

According to one embodiment of the present invention, interpretation content such as "The time I woke up last night was a little long, but WASO is still good" or "For the past n days, the time I woke up during sleep sessions has remained similar." Here, n is a natural number. In addition, content such as "For the last sleep session, the longest time I stayed awake was x minutes." Here, x is a real number less than 15 may be provided.

According to one embodiment of the present invention, content may be provided relating to "information that although WASO was good in the last sleep session, it can be utilized to further reduce post-sleep arousals."

Sleep Data Interpretation Content Example 5

According to one embodiment of the present invention, if the user's wake-up time data after sleep has shown a normal indicator (WASO of 5 minutes or more and less than 15 minutes), and if the user's wake-up time data after sleep has been generated over a predetermined period of time in the past, and if the WASO was less than 5 minutes in the past but is now longer, it can be determined that "the wake-up time is long" or "the wake-up time was short in the past but is now normal."

According to one embodiment of the present invention, interpretation content such as "The time spent awake last night was short" or "The time spent awake in the last sleep session increased compared to the last n days." Here, n is a natural number. In addition, content such as "The longest time spent awake in the last sleep session was x minutes." Here, x is a real number less than 15.

According to one embodiment of the present invention, interpretation content such as "It can be evaluated that the efficiency of sleep during sleep was not as good as in the past" can be provided. Content related to such content as "A method to reduce the longest time spent awake" or "Although WASO was good in the last sleep session, information that can be utilized to further reduce arousals after falling asleep" can be provided.

Sleep Data Interpretation Content Example 6

According to one embodiment of the present invention, if the user's wake-up time data after sleep has shown a poor (or bad) indicator (WASO of 15 minutes or more), and if the user's wake-up time data after sleep generated over a predetermined period of time in the past is compared to the user's wake-up time data generated over a predetermined period of time in the past, and the WASO was less than 15 minutes in the past but is now longer, it can be determined that "the wake-up time has become longer" or "the wake-up time was short in the past but is now longer."

According to one embodiment of the present invention, interpretation content such as "I was awake for a long time last night" or "Compared to the last n days, the time I was awake during the last sleep session increased." Here, n is a natural number. In addition, content such as "In the last sleep session, the longest time I was awake was x minutes." Here, x is a non-negative real number.

According to one embodiment of the present invention, interpretation content such as "It can be evaluated that the efficiency of sleep while sleeping was not as good as in the past" can be provided. Content related to such content as "A method to reduce the longest time spent awake" or "Information that can be utilized to reduce awakening after falling asleep" can be provided.

Sleep Data Interpretation Content Example 7

According to one embodiment of the present invention, if the user's wake-up time data after sleep has shown a poor (or bad) indicator (WASO of 15 minutes or more), and if compared with the user's wake-up time data generated over a predetermined period of time in the past, and if the WASO was 15 minutes or more in the past as well, it can be determined that "the wake-up time is long" or "the wake-up time was long in the past, but is long now as well."

According to one embodiment of the present invention, interpretation content such as "I was awake for a long time last night" or "Compared to the last n days, the time I was awake during the last sleep session is still long." Here, n is a natural number. In addition, content such as "In the last sleep session, the longest time I was awake was x minutes." Here, x is a non-negative real number.

According to one embodiment of the present invention, interpretation content such as "It can be evaluated that the efficiency of sleep while sleeping was not as good as in the past" can be provided. Content related to such content as "A method to reduce the longest time spent awake" or "Information that can be utilized to reduce awakening after falling asleep" can be provided.

Sleep data interpretation content in the REM Latency category

REM Latency refers to the time it takes for a user to enter the first REM sleep stage after entering sleep. After entering sleep, non-REM sleep stages and REM sleep stages may appear periodically, and the time it takes from the start of sleep to the occurrence of the first REM sleep can be defined as REM Latency.

If REM Latency is not properly maintained and is excessively short or long, problems such as mental illness and sleep disorders may occur, so REM Latency can be an important sleep indicator not only for sleep quality but also medically. According to one embodiment of the present invention, if REM Latency is less than 45 minutes, it can be evaluated as excessively short, and if REM Latency is 45 minutes or more and less than 2 hours, it can be evaluated as appropriate. If REM Latency is 2 hours or more, it can be evaluated as excessively long.

[Table 7] is a lookup table in which various cases of sleep data of the REM Latency category that can be measured according to embodiments of the present invention are recorded. [Table 8] is a lookup table in which sleep data interpretation contents of the REM Latency category that can be provided according to each case shown in [Table 7] are recorded. Hereinafter, with reference to [Table 7] and [Table 8], various sleep data interpretation contents that can be provided based on REM Latency data measured according to embodiments of the present invention will be described.

[Table 7]

[Table 8]

Sleep Data Interpretation Content Example 1

According to one embodiment of the present invention, if the user's REM sleep stage latency is shown to be short (REM Latency is less than 45 minutes), it can be determined that "the onset of the first REM sleep is early."

According to one embodiment of the present invention, content such as "The first REM sleep started too quickly in the last sleep session" or "The first REM sleep occurred x minutes after falling asleep in the last sleep session" may be provided, where x is a real number less than 45.

According to one embodiment of the present invention, content such as "The early start time of the first REM sleep may be due to temporary stress or irregular sleeping habits" or "If the early start time of the first REM sleep, you may not be able to wake up feeling refreshed." may be provided.

According to one embodiment of the present invention, information related to “at what point is the first REM sleep appropriate to start” or “lifestyle habits for achieving healthy REM sleep” can be provided.

Sleep Data Interpretation Content Example 2

According to one embodiment of the present invention, if the user's REM sleep stage latency is indicative of being appropriate (REM Latency is 45 minutes or more and less than 120 minutes), it can be determined that "the start timing of the first REM sleep was appropriate."

According to one embodiment of the present invention, content such as "The first REM sleep in the last sleep session started at an appropriate time" or "The first REM sleep in the last sleep session occurred x minutes after falling asleep" may be provided, where x is a real number greater than or equal to 45 and less than 120.

According to one embodiment of the present invention, content such as "If the start time of the first REM sleep is appropriate, it can be seen as a signal that the sleep cycle is likely to be formed distinctly." can be provided.

According to one embodiment of the present invention, information related to “sleep cycles, which can be an important indicator when judging the quality of sleep,” “the relationship between REM sleep and sleep cycles,” and “the positive effect of distinct sleep cycles on sleep” can be provided.

Sleep Data Interpretation Content Example 3

According to one embodiment of the present invention, if the user's REM sleep stage latency is shown as being long (REM Latency is 120 minutes or more), it can be determined that "the start time of the first REM sleep is late."

According to one embodiment of the present invention, content such as "The first REM sleep in the last sleep session started too late" or "The first REM sleep in the last sleep session occurred x minutes after falling asleep" may be provided, where x is a real number greater than or equal to 120.

According to one embodiment of the present invention, information related to “when is it appropriate for the first REM sleep to start” or “lifestyle habits for achieving healthy REM sleep” can be provided.

Sleep data interpretation content in the REM Ratio category

REM sleep stage percentage (REM Ratio) refers to the ratio of time spent in the REM sleep stage to the total sleep time measured in the last sleep session. If the ratio of REM sleep to the total sleep time is not properly maintained and is too short or long, it can have a negative impact on sleep quality, so REM Ratio can be used as an important sleep index to evaluate sleep quality.

According to one embodiment of the present invention, if the ratio of the REM sleep stage to the total sleep time is less than 15%, the REM sleep time may be evaluated as insufficient. If the ratio of the REM sleep stage is 15% or more but less than 30%, the REM sleep time may be evaluated as appropriate. If the ratio of the REM sleep stage is 30% or more, the REM sleep time may be evaluated as excessive.

[Table 9] is a lookup table in which various cases of sleep data of the REM sleep stage ratio (REM Ratio) category that can be measured according to embodiments of the present invention are recorded. [Table 10] is a lookup table in which sleep data interpretation contents of the REM sleep stage ratio (REM Ratio) category that can be provided according to each case shown in [Table 9] are recorded. Hereinafter, with reference to [Table 9] and [Table 10], various sleep data interpretation contents that can be provided based on REM sleep stage ratio (REM Ratio) data measured according to embodiments of the present invention are described.

[Table 9]

[Table 10]

Sleep Data Interpretation Content Example 1

According to one embodiment of the present invention, when an indicator is shown that the user's REM sleep stage ratio is insufficient (REM Ratio is less than 15%), it can be determined that "the REM sleep stage ratio is insufficient."

According to one embodiment of the present invention, content such as "REM sleep was insufficient in the last sleep session" or "The proportion of REM sleep in the last sleep session was x%." where x is a real number less than 15.

According to one embodiment of the present invention, information related to “cause of lack of REM sleep” or “appropriate ratio of REM sleep” can be provided.

Sleep Data Interpretation Content Example 2

According to one embodiment of the present invention, when an indicator is shown that the user's REM sleep stage ratio is adequate (REM Ratio is 15% or more and less than 30%), it can be determined that "the REM sleep stage ratio is sufficient."

According to one embodiment of the present invention, content such as "REM sleep was sufficient in the last sleep session" or "The proportion of REM sleep in the last sleep session was x%." Here, x is a real number greater than or equal to 15 and less than 30.

According to one embodiment of the present invention, content such as "the rate at which REM sleep increases in each sleep stage cycle was also ideal" can be provided.

According to one embodiment of the present invention, information regarding “details on REM sleep”, “positive effects of increasing REM sleep per sleep stage cycle” or “appropriate proportion and frequency of REM sleep” may be provided.

Sleep Data Interpretation Content Example 3

According to one embodiment of the present invention, if a user's REM sleep stage ratio is indicated as adequate (REM Ratio is 15% or more and less than 30%), but the REM sleep stage ratio is insufficient in the latter half of the sleep session, it may be determined that "the REM sleep stage ratio is sufficient" but the REM sleep stage ratio in the latter half is insufficient. Here, the latter half of the sleep session may mean the first cycle of sleep stages that appear during sleep.

According to one embodiment of the present invention, content such as "REM sleep was sufficient in the last sleep session" or "The proportion of REM sleep in the last sleep session was x%." Here, x is a real number greater than or equal to 15 and less than 30.

According to one embodiment of the present invention, content such as "You got enough REM sleep in the last sleep session, but the ratio of REM sleep stages was somewhat insufficient in the latter half" or "You got enough REM sleep in the last sleep session, but REM sleep occurred too early" may be provided.

According to one embodiment of the present invention, information regarding "how the condition after waking up changes depending on the time when REM sleep appears" can be provided.

Sleep Data Interpretation Content Example 4

According to one embodiment of the present invention, when an indicator is shown that the user's REM sleep stage ratio is excessive (REM Ratio is 30% or more), it can be determined that "the REM sleep stage ratio is too high."

According to one embodiment of the present invention, content such as "The percentage of REM sleep stage was too high in the last sleep session" or "The percentage of REM sleep was x% in the last sleep session" may be provided, where x is a real number greater than or equal to 30.

According to one embodiment of the present invention, content such as "When there is a lot of stress or when sleep habits are irregular, the proportion of REM sleep may increase" may be provided.

According to one embodiment of the present invention, “information related to methods for reducing stress” or “content helpful in reducing stress” may be provided.

Sleep data interpretation content in the Deep Ratio category

Deep Ratio refers to the ratio of time in the Deep sleep stage from the time of falling asleep to the time of waking up during a sleep session. For example, if sleep measurement started at exactly 9:00 PM, fell asleep at 9:30 PM, woke up at 7:30 AM, and ended at 8:00 AM, the ratio of time in the Deep sleep stage among the time from 9:30 PM to 7:30 AM (a total of 10 hours) is the Deep Ratio. It refers to the ratio of time in the Deep sleep stage to the sleep time measured in the last sleep session. If the ratio of the Deep sleep stage to the sleep time is not appropriate and is too short or long, it can have a negative impact on sleep quality, so the Deep Ratio can be used as an important sleep indicator to evaluate sleep quality.

According to one embodiment of the present invention, if the ratio of the deep sleep stage to the total sleep time is less than 15%, it can be evaluated that the time in the deep sleep stage is insufficient. If the ratio of the deep sleep stage is 15% or more, it can be evaluated that the time in the deep sleep stage is appropriate.

In addition, in terms of the distribution of deep sleep stages, if the ratio of deep stages to sleep time in the first half of sleep is more than 50%, it can be evaluated that the deep sleep time in the first half of sleep was appropriate. On the other hand, if the ratio of deep sleep stages measured in the first half of sleep is less than 50%, it can be evaluated that the deep sleep time in the first half of sleep was insufficient. Here, the first half of sleep refers to the period before half of the time from the time of falling asleep to the time of waking up. For example, if sleep measurement was started at exactly 9:00 PM, fell asleep at 9:30 PM, woke up at 7:30 AM, and ended at 8:00 AM, the first 5 hours out of the 10 hours from 9:30 PM to 7:30 AM can be considered the first half of sleep. In other words, in this case, the time from 9:30 PM to 2:30 AM corresponds to the first half of sleep.

[Table 11] is a lookup table in which various cases of sleep data of the Deep Ratio category that can be measured according to embodiments of the present invention are recorded. [Table 12] is a lookup table in which sleep data interpretation contents of the Deep Ratio category that can be provided according to each case shown in [Table 11] are recorded. Hereinafter, with reference to [Table 11] and [Table 12], various sleep data interpretation contents that can be provided based on the Deep Ratio data measured according to embodiments of the present invention will be described.

[Table 11]

[Table 12]

Sleep Data Interpretation Content Example 1

According to one embodiment of the present invention, if the overall deep sleep stage ratio of the user's sleep session is less than 15% and more than 50% of the deep sleep stage appears in the first half of the sleep, and if compared with the user's deep sleep stage ratio data generated over a predetermined period of time in the past, if the overall deep sleep stage ratio of the sleep session was also less than 15% in the past, it may be determined that there is "overall deep sleep deprivation."

Specifically, it can provide content such as "I have not had deep sleep for the past n days, but I was able to recover my body by sleeping deeply in the first half of my last sleep session", "The proportion of deep sleep stages in my last sleep session was x%." Here, n is a natural number, and x is a real number less than 15.

According to one embodiment of the present invention, information related to “the impact of deep sleep stages on the body” or “the correlation between deep sleep stages and sleep quality” may be provided.

Sleep Data Interpretation Content Example 2

According to one embodiment of the present invention, if the overall deep sleep stage ratio of the user's sleep session is less than 15% and more than 50% of the deep sleep stage appears in the latter half of the sleep session, and if compared with the user's deep sleep stage ratio data generated over a predetermined period of time in the past, if the overall deep sleep stage ratio of the sleep session was also less than 15% in the past, it may be determined that the user is "deprived of deep sleep overall."

Specifically, it can provide content such as "I have not had enough deep sleep for the past n days, and I mainly slept deeply in the second half of my last sleep session", "The proportion of deep sleep stages in my last sleep session was x%", where n is a natural number and x is a real number less than 15.

According to one embodiment of the present invention, information related to “the impact of deep sleep stages on the body” or “the correlation between deep sleep stages and sleep quality” may be provided.

Sleep Data Interpretation Content Example 3

According to one embodiment of the present invention, if the overall deep sleep stage ratio of the user's sleep session is less than 15%, and when compared with the user's deep sleep stage ratio data generated over a predetermined period of time in the past, if the overall deep sleep stage ratio of the sleep session in the past was 15% or more, it may be determined that there is "overall deep sleep deprivation."

Specifically, it can provide content such as "The last sleep session lacked deep sleep compared to the last n days", "The proportion of deep sleep stage in the last sleep session was x%", where n is a natural number and x is a real number less than 15.

According to one embodiment of the present invention, information related to “the impact of deep sleep stages on the body” or “the correlation between deep sleep stages and sleep quality” may be provided.

Sleep Data Interpretation Content Example 4

According to one embodiment of the present invention, if the overall deep sleep stage ratio of the user's sleep session is 15% or more, and when compared with the user's deep sleep stage ratio data generated over a predetermined period of time in the past, if the overall deep sleep stage ratio of the sleep session in the past was less than 15%, it can be determined that "overall deep sleep is sufficient."

Specifically, it can provide content such as "I did not get enough deep sleep for the past n days, but I got enough deep sleep in my last sleep session", "The proportion of deep sleep stages in my last sleep session was x%", where n is a natural number and x is a real number greater than or equal to 15.

According to one embodiment of the present invention, information related to “the impact of deep sleep stages on the body” or “the correlation between deep sleep stages and sleep quality” may be provided.

Sleep Data Interpretation Content Example 5

According to one embodiment of the present invention, if the overall deep sleep stage ratio of the user's sleep session was 15% or more, and 50% or more of the deep sleep stage appeared in the first half of the sleep, and if the overall deep sleep stage ratio of the user's sleep session was 15% or more in the past as compared to the deep sleep stage ratio data of the user generated over a predetermined period of time, it can be determined that "overall deep sleep was sufficient."

Specifically, it can provide content such as "I had enough deep sleep for the past n days, and I was able to recover my body by sleeping deeply in the first half of my last sleep session", "The proportion of deep sleep stages in my last sleep session was x%", where n is a natural number and x is a real number greater than or equal to 15.

According to one embodiment of the present invention, information related to “the impact of deep sleep stages on the body” or “the correlation between deep sleep stages and sleep quality” may be provided.

Sleep Data Interpretation Content Example 6

According to one embodiment of the present invention, if the overall deep sleep stage ratio of the user's sleep session is 15% or more, and 50% or more of the deep sleep stage appears in the latter half of the sleep session, and if compared with the user's deep sleep stage ratio data generated over a predetermined period of time in the past, if the overall deep sleep stage ratio of the sleep session was also less than 15% in the past, it may be determined that the user is "deprived of deep sleep overall."

Specifically, it can provide content such as "I had sufficient deep sleep for the past n days, and I slept deeply in the second half of my last sleep session", "The proportion of deep sleep stages in my last sleep session was x%", where n is a natural number and x is a real number greater than or equal to 15.

According to one embodiment of the present invention, information related to “the impact of deep sleep stages on the body” or “the correlation between deep sleep stages and sleep quality” may be provided.

Sleep data interpretation content in the First Cycle Sleep Quality category

According to the present invention, the first cycle sleep quality can be determined based on the ratio of the wake stage and the deep sleep stage that occur during the first part of sleep. The first part of sleep means the shorter time from the time of falling asleep to the time just before the first REM sleep stage is measured, or a predetermined time.

For example, if the time of falling asleep is 9:30 PM and the time of measuring the first REM sleep stage is 10:30 PM, the time from 9:30 PM to 10:30 PM can be determined as the early part of sleep. Alternatively, if the first REM sleep stage is measured later than the time of falling asleep, a predetermined time can be determined as the early part of sleep. For example, if the time of falling asleep is 9:30 PM and the first REM sleep stage is measured later (e.g., 12:30 PM), a predetermined time (e.g., 2 hours) can be determined as the early part of sleep. Here, the predetermined time is not limited to 2 hours and can be arbitrarily set to another value.

According to one embodiment of the present invention, if the awakening stage appears more than 10% during the first cycle, it can be evaluated that frequent or long-term awakening occurs in the early part of sleep. If the awakening stage appears less than 10% and the deep sleep stage appears less than 33% during the first cycle, it can be evaluated that few awakenings occur in the early part of sleep, but deep sleep is also insufficient. If the awakening stage appears less than 10% and the deep sleep stage appears more than 33% during the first cycle, it can be evaluated that few awakenings occur in the early part of sleep, but deep sleep is also sufficient.

[Table 13] is a lookup table in which various cases of sleep data of the First Cycle Sleep Quality category that can be measured according to embodiments of the present invention are recorded. [Table 14] is a lookup table in which sleep data interpretation contents of the First Cycle Sleep Quality category that can be provided according to each case shown in [Table 13] are recorded. Hereinafter, with reference to [Table 13] and [Table 14], various sleep data interpretation contents that can be provided based on the First Cycle Sleep Quality data that can be inferred according to the ratio of the Wake stage and/or the Deep stage in the early part of sleep measured according to embodiments of the present invention are described.

[Table 13]

[Table 14]

Sleep Data Interpretation Content Example 1

According to one embodiment of the present invention, if the Wake stage appears in the first half of the user's sleep at a rate of 10% or more, and the Wake stage appears in the first half more frequently compared to the average sleep data of other users, it can be determined that the user "wakes up a lot in the first half."

According to one embodiment of the present invention, content such as "I woke up a lot in the early part of the last sleep session and could not sleep deeply", "Overall, the quality of sleep can be evaluated as poor", or "The number of times the awakening stage appeared in the early part of sleep was n times" may be provided, where n is a non-negative integer.

According to one embodiment of the present invention, information related to “the impact of sleep quality in the early part of sleep on overall sleep quality” or “a method for improving sleep quality in the early part of sleep” can be provided.

Sleep Data Interpretation Content Example 2

According to one embodiment of the present invention, if the Wake stage is less than 10% and the Deep sleep stage is less than 33% in the first half of the user's sleep, and if the Wake stage is less and the Deep sleep stage is less in the first half compared to the average sleep data of other users, it can be determined that the user "wakes up less in the first half" and "does not sleep deeply in the first half."

According to one embodiment of the present invention, content such as "I did not wake up well in the early part of the last sleep session, but I did not fall into a deep sleep", "The number of awakenings was low in the first half of the sleep session", "The rate of deep sleep appeared in the first half of the sleep session was low", or "The number of awakenings in the early part of sleep appeared was n" may be provided, where n is a non-negative integer.

According to one embodiment of the present invention, information related to “the impact of sleep quality in the early part of sleep on overall sleep quality” or “a method for improving sleep quality in the early part of sleep” can be provided.

Sleep Data Interpretation Content Example 3

According to one embodiment of the present invention, if the Wake stage is less than 10% and the Deep sleep stage is more than 33% in the first half of the user's sleep, and if the Wake stage is less and the Deep sleep stage is more in the first half compared to the average sleep data of other users, it can be determined that the user "wakes up little in the first half" and "sleeps deeply in the first half."

According to one embodiment of the present invention, content such as "I slept deeply without waking up in the early part of the last sleep session", "Overall, the quality of sleep can be evaluated as good", "The number of awakenings in the first half of the sleep session was low", "The rate of deep sleep in the first half of the sleep session was high", or "The number of awakenings in the early part of sleep was n times" may be provided, where n is a non-negative integer.

According to one embodiment of the present invention, information related to “the impact of sleep quality in the early part of sleep on overall sleep quality” or “a method for improving sleep quality in the early part of sleep” can be provided.

Sleep data interpretation content in the Number of Awakening category

The number of awakenings refers to the number of awakening stages that occur during sleep before waking up completely after falling asleep. If the number of awakenings is high, the quality of sleep may be evaluated as low. According to one embodiment of the present invention, if the number of awakenings is 30 or more, the number of awakenings may be evaluated as high. If the number of awakenings is less than 30, the number of awakenings may be evaluated as low.

[Table 15] is a lookup table in which various cases of sleep data of the Number of Awakening category that can be measured according to embodiments of the present invention are recorded. [Table 16] is a lookup table in which sleep data interpretation contents of the Number of Awakening category that can be provided according to each case shown in [Table 15] are recorded. Hereinafter, with reference to [Table 15] and [Table 16], various sleep data interpretation contents that can be provided based on the Number of Awakening data measured according to embodiments of the present invention will be described.

[Table 15]

[Table 16]

Sleep Data Interpretation Content Example 1

According to one embodiment of the present invention, if the number of awakenings during sleep of a user is 30 or more, the user may be evaluated as having “a high number of awakenings during sleep” or “poor quality of sleep.”

According to one embodiment of the present invention, interpretation content such as "There were many awakenings during sleep in the last sleep session", "There were n awakenings during sleep in the last sleep session", "The quality of sleep in the last sleep session may be evaluated as poor", "The satisfaction of sleep may be affected by the number of awakenings during sleep", or "You may not have sufficiently recovered from fatigue last night" may be provided. Here, n is a non-negative integer.

According to one embodiment of the present invention, “information related to a method for reducing the number of awakenings during sleep” or “content that can be provided to reduce the number of awakenings during sleep” may be provided.

Sleep Data Interpretation Content Example 2

According to one embodiment of the present invention, if the number of awakenings during sleep of a user is less than 30, the user may be evaluated as having “a low number of awakenings during sleep” or “good quality of sleep.”

According to one embodiment of the present invention, interpretation content such as "the number of awakenings during sleep was small in the last sleep session", "the number of awakenings during sleep was n times in the last sleep session", "the quality of sleep in the last sleep session can be evaluated as good", "the satisfaction of sleep can be affected by the number of awakenings during sleep", or "you may have sufficiently recovered from fatigue last night" can be provided. Here, n is a non-negative integer.

According to one embodiment of the present invention, “information related to a method for reducing the number of awakenings during sleep” or “content that can be provided to reduce the number of awakenings during sleep” may be provided.

Sleep data interpretation content in the Total Sleep Time category

Total sleep time refers to the total time from falling asleep to waking up completely during a sleep session. Since total sleep time is also related to the number of sleep stage cycles that can occur during sleep, maintaining an appropriate sleep time can improve sleep quality. In addition, the number of sleep stage cycles that occur appropriately can improve sleep quality.

According to one embodiment of the present invention, if the total sleep time is 10 hours or more, the total sleep time can be evaluated as long. If the total sleep time is 6 hours or more but less than 8 hours, the total sleep time can be evaluated as appropriate. If the total sleep time is 4 hours or more but less than 6 hours, the total sleep time can be evaluated as somewhat short. If the total sleep time is less than 4 hours, the total sleep time can be evaluated as short.

In addition, according to one embodiment of the present invention, if the number of sleep stage cycles that appear during the entire sleep time is less than 4, the sleep cycles may be evaluated as insufficient. If the number of sleep stage cycles that appear during the entire sleep time is 4 or more but less than 8, the sleep cycles may be evaluated as adequate. If the number of sleep stage cycles that appear during the entire sleep time is 8 or more, the sleep cycles may be evaluated as abundant.

[Table 17] is a lookup table in which various cases of sleep data of the Total Sleep Time category that can be measured according to embodiments of the present invention are recorded. [Table 18] is a lookup table in which sleep data interpretation contents of the Total Sleep Time category that can be provided according to each case shown in [Table 17] are recorded. Hereinafter, with reference to [Table 17] and [Table 18], various sleep data interpretation contents that can be provided based on the Total Sleep Time data measured according to embodiments of the present invention are described.

[Table 17]

[Table 18]

Sleep Data Interpretation Content Example 1

According to one embodiment of the present invention, if the user's total sleep time was less than 4 hours, the user may be evaluated as having "insufficient sleep time."

According to one embodiment of the present invention, interpretation content such as "You did not have enough sleep time in the last sleep session", "You slept x hours in the last sleep session", "You need at least 4 hours of sleep to recover from fatigue", or "You are recommended to sleep more than 4 hours so that a sufficient sleep stage cycle can occur" can be provided. Here, x is a real number less than 4.

According to one embodiment of the present invention, information related to “a method for increasing sleep time” or “the effect of sleep time on condition” may be provided.

Sleep Data Interpretation Content Example 2

According to one embodiment of the present invention, if the user's total sleep time is 10 hours or more, the user may be evaluated as having "excessive sleep time."

According to one embodiment of the present invention, interpretation content such as “the slept time was too long in the last sleep session”, “the slept time in the last sleep session was x hours”, or “sleeping for too long can actually lower the quality of sleep” can be provided, where x is a real number greater than or equal to 10.

According to one embodiment of the present invention, information related to “the effect of sleeping too long on the quality of sleep”, “a method for finding out a suitable sleep time for a user”, or “the effect of sleep time on condition” may be provided.

Sleep Data Interpretation Content Example 3

According to one embodiment of the present invention, if the total sleep time of a user is 4 hours or more but less than 6 hours, and the number of times a sleep stage cycle appears is less than 4, the user may be evaluated as having “insufficient sleep time” and “insufficient number of sleep cycles.”

According to one embodiment of the present invention, interpretation content such as "You did not have enough sleep time in the last sleep session", "You slept x hours in the last sleep session", or "You need a sufficient amount of sleep to properly display the cycle of sleep stages" may be provided. Here, x is a real number greater than or equal to 4 and less than 6.

According to one embodiment of the present invention, information related to “sleep stage cycle”, “method for finding optimal sleep stage cycle”, “method for increasing sleep time”, or “effect of sleep time on condition” may be provided.

Sleep Data Interpretation Content Example 4

According to one embodiment of the present invention, if the total sleep time of the user is 4 hours or more but less than 6 hours, and the number of times the sleep stage cycle appears is 4 or more but less than 8, the user may be evaluated as having “insufficient sleep time” and “appropriate number of sleep cycles.”

According to one embodiment of the present invention, interpretation content such as "Although the sleep time was not long in the last sleep session, the cycle of sleep stages appeared appropriately" or "The sleep time in the last sleep session was x hours" can be provided, where x is a real number greater than or equal to 4 and less than 6.

According to one embodiment of the present invention, information related to “a method for finding out a suitable sleep time for a user,” “a cycle of sleep stages,” “a method for finding an optimal cycle of sleep stages,” or “the effect of sleep time on condition” may be provided.

Sleep Data Interpretation Content Example 5

According to one embodiment of the present invention, if the user's total sleep time is 4 hours or more and less than 6 hours, and the number of times the sleep stage cycle appears is 8 or more, the user may be evaluated as having "insufficient sleep time" and "an appropriate number of sleep cycles."

According to one embodiment of the present invention, interpretation content such as "Although the sleep time was not long in the last sleep session, the cycle of sleep stages appeared appropriately" or "The sleep time in the last sleep session was x hours" can be provided, where x is a real number greater than or equal to 4 and less than 6.

According to one embodiment of the present invention, information related to “a method for finding out a suitable sleep time for a user,” “a cycle of sleep stages,” “a method for finding an optimal cycle of sleep stages,” or “the effect of sleep time on condition” may be provided.

Sleep Data Interpretation Content Example 6

According to one embodiment of the present invention, if the total sleep time of the user is 6 hours or more but less than 8 hours, and the number of times the sleep stage cycle appears is less than 4, the user may be evaluated as having “sufficient sleep time” and “insufficient number of sleep cycles.”

According to one embodiment of the present invention, interpretation content such as "the sleep time in the last sleep session was adequate, but the cycle of sleep stages was not sufficiently displayed" or "the sleep time in the last sleep session was x hours" can be provided, where x is a real number greater than or equal to 6 and less than 8.

According to one embodiment of the present invention, information related to “the effect of a short sleep stage cycle on sleep quality,” “a method for finding out a sleep time suitable for a user,” “a sleep stage cycle,” “a method for finding an optimal sleep stage cycle,” or “the effect of sleep time on condition” can be provided.

Sleep Data Interpretation Content Example 7

According to one embodiment of the present invention, if the total sleep time of the user is 6 hours or more but less than 8 hours, and the number of times the sleep stage cycle appears is 4 or more but less than 8, the user may be evaluated as having “sufficient sleep time” and “appropriate number of sleep cycles.”

According to one embodiment of the present invention, interpretation content such as "the sleep time in the last sleep session was appropriate and the cycle of sleep stages was appropriate", "the sleep time in the last sleep session was x hours", or "the sleep time in the last sleep session may be the optimal sleep time" may be provided. Here, x is a real number greater than or equal to 6 and less than 8.

According to one embodiment of the present invention, information related to “sleep stage cycle”, “correlation between sleep stage cycle and sleep time”, “method for finding optimal sleep stage cycle”, or “effect of sleep time on condition” can be provided.

Sleep Data Interpretation Content Example 8

According to one embodiment of the present invention, if the total sleep time of the user is 6 hours or more and less than 8 hours, and the number of times the sleep stage cycle appears is 8 or more, the user may be evaluated as having “sufficient sleep time” and “appropriate number of sleep cycles.”

According to one embodiment of the present invention, interpretation content such as "The sleep time was appropriate in the last sleep session, but the cycle of sleep stages appeared too much", "The sleep time in the last sleep session was x hours", or "It may be better to sleep a little less than the sleep time in the last sleep session" may be provided. Here, x is a real number greater than or equal to 6 and less than 8.

According to one embodiment of the present invention, information related to “sleep stage cycle”, “correlation between sleep stage cycle and sleep time”, “method for finding optimal sleep stage cycle”, or “effect of sleep time on condition” can be provided.

Sleep Data Interpretation Content Example 9

According to one embodiment of the present invention, if the total sleep time of a user is 8 hours or more but less than 10 hours, and the number of sleep stage cycles is less than 4, the user may be evaluated as having “excessive sleep time” and “insufficient number of sleep cycles.”

According to one embodiment of the present invention, interpretation content such as "The sleep time was long in the last sleep session, but the sleep stage cycle was not sufficiently displayed", "The sleep time in the last sleep session was x hours", or "The sleep cycle was not sufficiently displayed in the last sleep session may be due to temporary stress or bad sleeping habits" may be provided. Here, x is a real number greater than or equal to 8 and less than 10.

According to one embodiment of the present invention, information related to “sleep stage cycle”, “correlation between sleep stage cycle and sleep time”, “method for finding optimal sleep stage cycle”, or “effect of sleep time on condition” can be provided.

Sleep Data Interpretation Content Example 10

According to one embodiment of the present invention, if the total sleep time of the user is 8 hours or more but less than 10 hours, and the number of times the sleep stage cycle appears is 4 or more but less than 8, the user may be evaluated as having “excessive sleep time” and “appropriate number of sleep cycles.”

According to one embodiment of the present invention, interpretation content such as "the sleep time was long in the last sleep session, but the cycle of sleep stages appeared appropriately", "the sleep time in the last sleep session was x hours", or "the sleep time in the last sleep session may be the optimal sleep time" can be provided. Here, x is a real number greater than or equal to 8 and less than 10.

According to one embodiment of the present invention, information related to “sleep stage cycle”, “correlation between sleep stage cycle and sleep time”, “method for finding optimal sleep stage cycle”, or “effect of sleep time on condition” can be provided.

Sleep Data Interpretation Content Example 11

According to one embodiment of the present invention, if the total sleep time of a user is 8 hours or more but less than 10 hours, and the number of times a sleep stage cycle appears is 8 or more, the user may be evaluated as having “excessive sleep time” and “excessive number of sleep cycles.”

According to one embodiment of the present invention, interpretation content such as "Your last sleep session had a long sleep time and the sleep stage cycle was too frequent", "Your last sleep session had x hours of sleep", "It may be better to sleep a little less than your last sleep session", or "It is more important to sleep well for an appropriate amount of time than to sleep for a long time" may be provided. Here, x is a real number greater than or equal to 8 and less than 10.

According to one embodiment of the present invention, information related to “sleep stage cycle”, “correlation between sleep stage cycle and sleep time”, “method for finding optimal sleep stage cycle”, or “effect of sleep time on condition” can be provided.

How to determine the importance of sleep information across categories

In providing a user with sleep interpretation by category of sleep information, there may be a demand to preferentially provide a category of sleep information that is judged to be most important in the current sleep session (or the previous sleep session). Accordingly, according to embodiments of the present invention, a method is disclosed for calculating importance (or importance score) between categories of sleep information and providing a user with sleep interpretation by category of sleep information based on the calculated importance.

According to one embodiment of the present invention, a method for determining importance between categories of sleep information may include a step of calculating an importance score based on at least one of information comparing sleep data (e.g., sleep state information) generated in a previous sleep session with sleep data of a user generated during a predetermined period in the past, information comparing the sleep data with a medical standard, and information comparing the sleep data of another user generated during a predetermined period in the past.

According to one embodiment of the present invention, the operation of the algorithm for determining the importance between categories of sleep information can be performed through a program written in one or more computer programming languages. Here, the computer programming language can include at least one of commercialized computer programming languages such as Python, JavaScript, Java, C, C++, C#, Ruby, PHP, Swift, Kotlin, R, Golang, Go, G-code, Perl, Haskell, HolyC, Scala, Lua, Ada, COBOL, Erlang, Rust, Lisp, F#, Prolog, Dart, Objective-C, Julia, and Nim.

In a program for performing data analysis and interpretation according to one embodiment of the present invention, at least one of a function related to an operating system, a module for numerical operations, and a module for providing type hints can be loaded or input.

The operating system-related functions according to one embodiment of the present invention refer to various functions for interacting with the operating system. When a computer program calls a function related to the operating system, it can perform operating system-related tasks such as file and directory operations, setting environment variables, checking and changing the current working directory, and process management.

A module for numerical calculation according to one embodiment of the present invention may be a module for providing various functions, such as creating and manipulating arrays and performing mathematical operations on arrays through a computer programming language. For example, the numpy module provided in the Python language may be called from the outside and numerical calculation operations may be performed using this module.

According to one embodiment of the present invention, a type module that explicitly defines a type (data type) of at least one of a function and a variable in a programming language and performs type checking can be called. The type module can help improve the readability of the programming language and prevent errors related to the type of data expressed as a value or variable in advance.

According to one embodiment of the present invention, a class of a category related to sleep interpretation can be defined. A data structure consisting of a pair of keys and values including a name of the category and parameters necessary for calculating importance can be defined. For example, this data structure can be at least one of a hash map, an associative array, a map, and a dictionary. When defining this data structure, a pair of keys and values is stored, so that it can be easy to search for, insert, modify, and delete a linked value using a specific key.

A data structure composed of pairs of keys and values according to one embodiment of the present invention may include at least one of a data structure representing learned parameters used for calculating importance, a data structure representing sleep measurement values of a target sleep session, and a data structure representing an average measurement value over a predetermined period of time. In the data structure according to one embodiment of the present invention, a name of a category may be designated as a key, and an importance and an interpretation ID may be designated as values.

According to one embodiment of the present invention, for each sleep session, parameters can be calculated for each category, and the importance of the category can be calculated based on the calculated parameters. Meanwhile, the parameters for each category can be calculated based on at least one of a sleep measurement value of a target sleep session and an average measurement value for a predetermined period. The importance (or importance score) between categories can be calculated as a numerical value, and can be expressed by approximating it using 0 and 1 in a floating point manner.

According to one embodiment of the present invention, identification information (e.g., interpretation ID) may be required to provide analysis results of sleep data by category. The identification information may be provided in the form of a string. Here, content by category of sleep information, which is indicated by a specific string corresponding to each interpretation ID, may be provided.

According to one embodiment of the present invention, an analysis result of sleep data acquired during a predetermined period may be provided based on at least one of sleep measurement values of a plurality of sleep sessions acquired during a predetermined period and average measurement values of the plurality of sleep sessions. In addition, the analysis result of sleep data may be identified based on identification information of the plurality of sleep sessions. Meanwhile, the predetermined period may be 30 days, but a specific number related to the predetermined period is merely an example and is not limited thereto. In addition, the plurality of sleep sessions may be a total of 30 sleep sessions, but a specific number related to the number of sleep sessions is merely an example and is not limited thereto.

According to one embodiment of the present invention, the importance (or importance score) between categories of sleep information can be calculated as a numerical value. The importance score between categories can be a numerical value used to estimate the importance of data by category of sleep information. For example, if a high importance score is assigned to a category to which sleep data or sleep events belong, it can be determined to be more important than other categories of sleep information. In addition, a category of sleep information to which a relatively low importance score is assigned can be determined to be less important than other categories of sleep information.

According to one embodiment of the present invention, data of a category of sleep information having a relatively high importance score can be generated or provided so that it can be easily identified than sleep data belonging to other categories of sleep information. For example, when outputting sleep data by category of sleep information, data generated so that data of a category of sleep information having a high importance score is placed at the top can be output.

According to one embodiment of the present invention, when outputting sleep data by category of sleep information, the data of the category of sleep information with a high importance score may be output so that the data is arranged at the top. Alternatively, when outputting sleep data by category of sleep information, the data of the category of sleep information with a high importance score may be output first in time series. Alternatively, when visually displaying sleep data by category of sleep information, the data may be expressed in a color or brightness that is distinct from data belonging to other categories of sleep information.

Meanwhile, even if it is a category of sleep information with a high calculated importance score, if it is a sleep category that is set not to provide sleep data interpretation, sleep data interpretation of the sleep category may not be provided. In this case, data interpretation of a category of sleep information with a lower importance ranking may be provided instead. According to one embodiment of the present invention, the minimum value of the importance (or importance score) of the category of sleep information may be set to a constant. This may represent the lowest value among the possible values representing the importance. For example, this constant may be -100, but is not limited thereto.

According to one embodiment of the present invention, a class of a category of sleep information defined may include at least one of a Sleep Apnea category, a Sleep Onset Latency category, a Wake Time after Sleep Onset (WASO) category, a REM Latency category, a REM Ratio category, a Deep Ratio category, a First Cycle Sleep Quality category, a Number of Awakening category, and a Total Sleep Time category. For each category of sleep information, a parameter may be calculated for each sleep session, and an importance (or importance score) of the category of sleep information may be calculated based on the parameter.

According to one embodiment of the present invention, first, the name of the sleep category expressed as a string and the importance parameters expressed as data of a string and/or number are initialized. The importance parameters are a dictionary including parameters required for calculating importance and may mean a data structure composed of keys and values. The value of the importance parameters may be multiplied by the sleep data as a weight. Thereafter, an importance score calculated as a floating point number may be calculated, and sleep interpretation data may be provided as a string based on an interpretation ID corresponding to the sleep data.

How to learn importance parameters

Hereinafter, a method for calculating the importance score of a sleep category and learning importance parameters according to embodiments of the present invention will be described using FIGS. 32, 34 to 36, [Table 19], and [Table 20].

FIGS. 34 and 35 are graphs for explaining a method for calculating an importance score of a category of sleep information based on medical criteria according to embodiments of the present invention. FIG. 13 is a graph for explaining a method for calculating an importance score of a category of sleep information based on information compared with sleep data of a user generated over a predetermined period of time in the past according to embodiments of the present invention. The predetermined period of time may be, for example, 30 days or 30 sleep sessions, but is not limited thereto.

FIG. 32 is a flowchart illustrating a method for learning importance parameters that serve as a basis for calculating importance scores of sleep categories according to embodiments of the present invention.

According to one embodiment of the present invention, importance parameters can be learned based on Reinforecement Learning with Human Feedback (RLHF). Reinforcement Learning with Human Feedback (RLHF) is a learning method that combines reinforcement learning and human feedback, and improves the performance of artificial intelligence by learning information output through an artificial intelligence model based on human feedback. According to one embodiment of the present invention, importance scores of categories of sleep information can be calculated based on learned importance parameters.

According to FIG. 32, a human feedback-based reinforcement learning method according to an embodiment of the present invention may include a step of obtaining sleep data based on sleep state information of a user obtained during one or more sleep sessions (S300-2), a step of generating a learning training sample set for one or more sleep categories to which the obtained sleep data belongs (S320-2), a step of calculating an importance score based on an operation of an algorithm that determines importance between categories of sleep information (S340-2), a step of performing learning feedback that compares the calculated importance score with the generated learning training sample (S360-2), and a step of updating an importance parameter by an algorithm according to the learning feedback (S380-2).

According to one embodiment of the present invention, the user's sleep data may be sleep data belonging to at least one category among the Sleep Apnea category, the Sleep Onset Latency category, the First Cycle Sleep Quality category, the REM Latency category, the REM Ratio category, the Deep Ratio category, the Wake Time after Sleep Onset (WASO) category, the Number of Awakening category, and the Total Sleep Time category.

According to one embodiment of the present invention, the step (S320-2) of generating a learning training sample set for a sleep category may further include a step of setting a ranking of categories judged to have high importance among the categories to which the acquired sleep data belongs. According to one embodiment of the present invention, the learning training sample set may be a data set composed of pairs of rankings of sleep categories judged to have high importance and names of sleep categories. In addition, according to one embodiment of the present invention, the importance of the sleep category may be determined, and the rankings of three sleep categories judged to have high importance may be determined. In this case, the learning training sample set corresponds to a data set composed of ordered pairs of rankings from 1st to 3rd and names of sleep categories corresponding to each ranking. Here, the specific numerical value such as 3rd place is only an example and is not limited thereto.

According to one embodiment of the present invention, the step (S340-2) of calculating an importance score based on an operation of an algorithm for determining importance between categories of sleep information may further include a step of calculating an importance score by adding an importance score obtained based on at least one of the graphs shown in FIGS. 34 and 35 and an importance score obtained based on the graph shown in FIG. 36.

In a case where importance scores are obtained based on both the graphs illustrated in FIG. 34 and FIG. 35 according to one embodiment of the present invention, the importance score may be calculated by adding together the importance score obtained based on the graph illustrated in FIG. 34, the importance score obtained based on the graph illustrated in FIG. 35, and the importance score obtained based on the graph illustrated in FIG. 36.

FIGS. 34 and 35 are graphs for explaining a method for calculating an importance score of a category of sleep information based on medical criteria according to embodiments of the present invention. In the graphs illustrated in FIGS. 34 and 35, the x-axis represents a value of sleep data belonging to a corresponding sleep category, and the y-axis represents an importance score based on medical criteria. The slope values (a and m) and the bias values (b and n) of the linear graph illustrated in FIG. 34 correspond to importance parameters, and the lower bound and upper bound described on the x-axis may represent the lower bound and upper bound of a standard value according to medical criteria in the corresponding sleep category, respectively. In the graph including the upward convex curve illustrated in FIG. 35, a` and m`, b`, and n` correspond to importance parameters.

According to one embodiment of the present invention, the importance score of a category of sleep information based on medical criteria may be calculated based on at least one of a linear graph as illustrated in FIG. 34 or a graph including an upward convex curve as illustrated in FIG. 35, but the form of the graph is not limited thereto. For example, although the graph including an upward convex curve as illustrated in FIG. 35 is expressed in the form of a logarithmic function, it may be expressed in the form of a graph including another upward convex curve that is not a logarithmic function.

Alternatively, according to one embodiment of the present invention, the importance may be calculated based on a straight line in which the importance increases linearly for values of sleep data lower than the lower limit of a standard value according to medical standards, and a curve in which the importance increases convexly upward for values of sleep data higher than the upper limit of the standard value. Alternatively, according to one embodiment of the present invention, the importance may be calculated based on a curve in which the importance increases convexly upward for values of sleep data lower than the lower limit of a standard value according to medical standards, and a straight line in which the importance increases linearly for values of sleep data higher than the upper limit of the standard value.

According to one embodiment of the present invention, when calculating an importance score based on medical criteria, if the value of sleep data is outside the lower or upper limit of the standard value according to the medical criteria, it may mean that it is outside the general range. At this time, considering the sensitivity of the importance for cases where it is outside the lower or upper limit of the standard value, it may be learned to determine whether to calculate the importance based on a linear graph or a graph including an upward convex curve. Meanwhile, an example of a graph including an upward convex curve may be a logarithmic function.

According to one embodiment of the present invention, if the sleep data value in the last sleep session is significantly outside the lower or upper limit of the standard value according to medical standards, and if the difference in importance is not large compared to the case where the sleep data value slightly deviates from the lower or upper limit of the standard value, it can be determined that the sensitivity of the importance according to the size of the difference from the standard value is small. In this case, it can be learned that the importance score is calculated based on a graph including an upward convex curve for an area where the sleep data value is smaller than the lower limit or larger than the upper limit of the standard value. On the other hand, if the importance should increase linearly as it deviates from the upper or lower limit of the medical standard value, it can be determined that the sensitivity of the importance according to the size of the difference from the standard value is large. In this case, it can be learned that the importance score is calculated based on a linear graph for an area where the sleep data value is smaller than the lower limit or larger than the upper limit of the standard value.

FIG. 34 is a graph for explaining a method for calculating an importance score of a category of sleep information based on information compared with sleep data of a user generated over a predetermined period of time in the past according to embodiments of the present invention. In the graph illustrated in FIG. 34, the x-axis represents a deviation between an average value of sleep data of a user generated over a predetermined period of time in the past and a value of sleep data of the user acquired in a previous sleep session. The y-axis represents an importance score based on a comparison with an average value of sleep data generated over a predetermined period of time in the past.

According to one embodiment of the present invention, if the importance is high as the difference between the sleep data value acquired in the last sleep session and the average value of sleep data generated over a predetermined period of time in the past is large, learning can be performed so that the absolute value of the importance parameter (a or m) corresponding to the slope of the linear graph illustrated in FIG. 34 increases. Alternatively, according to one embodiment of the present invention, if the importance is not high even if the sleep data value acquired in the last sleep session has a large difference between the average value of sleep data generated over a predetermined period of time in the past, learning can be performed so that the absolute value of the importance parameter (a or m) corresponding to the slope of the linear graph illustrated in FIG. 34 decreases.

For example, if a sleep data value according to one embodiment of the present invention is greater than the average, and if the importance increases as it gets farther from the average, learning can be performed so that the absolute value of the slope (importance parameter) of the linear graph in the area where x is greater than 0 increases. If a sleep data value according to one embodiment of the present invention is greater than the average, and if the importance is not great even if it gets farther from the average, learning can be performed so that the absolute value of the slope (importance parameter) of the linear graph in the area where x is greater than 0 decreases.

As another example, if a sleep data value according to an embodiment of the present invention is less than the average, and if the importance increases as it gets farther from the average, learning can be performed so that the absolute value of the slope (importance parameter) of the linear graph in the area where x is less than 0 increases. If a sleep data value according to an embodiment of the present invention is less than the average, and if the importance is not large even if it gets farther from the average, learning can be performed so that the absolute value of the slope (importance parameter) of the linear graph in the area where x is less than 0 decreases.

[Table 19] is a table for explaining a method for determining an importance score of a sleep apnea category according to embodiments of the present invention. According to one embodiment of the present invention, the importance score of a sleep apnea category may be determined according to an embodiment described in [Table 19], rather than a calculation based on the graphs of FIGS. 34 to 36.

[Table 19]

According to [Table 19], the importance score of the Sleep Apnea category can be determined based on comparing the average of the user's AHI values over a predetermined period of time in the past with the AHI values in the last sleep session. Here, the predetermined period of time can be, for example, 30 days or 30 sleep sessions, but is not limited thereto.

Additionally, according to one embodiment of the present invention, the importance score of the Wake Time after Sleep Onset (WASO) category may be determined according to the embodiment described in [Table 20], rather than calculation based on the graphs of FIGS. 34 to 36.

[Table 20]

According to [Table 20], the importance score of the Wake Time after Sleep Onset (WASO) category can be determined based on comparing the average of the WASO values for a predetermined period of time in the past of the user with the WASO values in the last sleep session. Here, the predetermined period of time can be, for example, 30 days or 30 sleep sessions, but is not limited thereto.

The step (S360) of performing learning feedback by comparing the calculated importance scores with the generated learning training samples according to one embodiment of the present invention may further include a step of comparing the importance scores calculated in the manner described above through FIGS. 34 to 36, [Table 19], and [Table 20] with the learning training samples generated through the step (S320-2) of generating a learning training sample set. According to one embodiment of the present invention, learning feedback may be performed by calculating the importance scores of one or more sleep categories and comparing the rankings of three sleep categories showing the highest values among the calculated importance scores with the learning training sample set. Here, the specific number of three is only an example and is not limited thereto.

The step (S380-2) in which the algorithm updates the importance parameter according to the learning feedback can update the importance parameter so that the ranking of the sleep category according to the calculated importance score matches the generated learning training sample through the comparison performed in the step (S360-2) in which the learning feedback is performed to compare the calculated importance score with the learning training sample. In addition, the importance parameter can be re-updated by repeating the process of calculating the importance score by the operation of the algorithm that determines the importance between the categories of sleep information based on the learned importance parameter and comparing the calculated importance score with the learning training sample again.

According to one embodiment of the present invention, the importance score of a sleep category can be calculated based on sleep data measured in the last sleep session and sleep data measured in the last 30 sleep sessions. The detailed implementation details for calculating the importance of each class of the category to which each sleep information belongs and determining the interpretation ID are described in detail below.

Calculate and interpret importance scores for Sleep Apnea category ID determination

Hereinafter, a method for calculating the importance score of the Sleep Apnea category according to one embodiment of the present invention is described in detail.

First, the name of the sleep category, which is expressed as a string, and the importance parameters, which are expressed as string and/or numeric data, are initialized. The name of the sleep category here may be expressed as Sleep Apnea. In addition, instead of Sleep Apnea, it may be expressed as an abbreviation, "A." For example, in the present disclosure, the interpretation ID of the sleep apnea category is expressed using the abbreviation, "A," but is not limited thereto.

According to one embodiment of the present invention, a judgment on a daily condition for sleep apnea can be made based on a value of an AHI index (hereinafter referred to as 'daily AHI') measured in a user's last sleep session. Alternatively, a judgment on a monthly condition for sleep apnea can be made based on an average of AHI values measured in the user's last 30 sleep sessions (hereinafter referred to as 'monthly AHI').

For example, a judgment on the sleep apnea condition can be performed by outputting a string of "good", "normal", or "severe" depending on the AHI value. For example, if the daily AHI value is less than 5, it can be judged as "good", if the daily AHI value is 5 or more but less than 15, it can be judged as "normal", and if the daily AHI value is 30 or more, it can be judged as "severe". Similarly, if the monthly AHI value is less than 5, it can be judged as "good", if the daily AHI value is 5 or more but less than 15, it can be judged as "normal", and if the daily AHI value is 30 or more, it can be judged as "severe".

According to one embodiment of the present invention, the importance of the sleep apnea category can be calculated based on the daily AHI and monthly AHI. Meanwhile, if the calculation cannot be performed due to an error in the value entered in the daily AHI, the AHI can be regarded as 0.

According to one embodiment of the present invention, an importance score calculation for a sleep apnea category can be performed based on a comparison between daily AHI values and monthly AHI values.

According to one embodiment of the present invention, the importance score of the sleep apnea category can be calculated by multiplying the daily AHI value by a value corresponding to the importance parameter, "sleep apnea weight." More specifically, the importance score of the daily AHI sleep apnea category can be calculated by applying the importance parameter according to each case, dividing the case into the case where the monthly AHI is greater than the daily AHI and the case where the monthly AHI is less than the daily AHI. Here, the importance score of the sleep apnea category can also be calculated by adding the "sleep apnea bias" value, which is an importance parameter.

According to one embodiment of the present invention, as described above through [Table 19], the importance score can be calculated by comparing the monthly AHI with the daily AHI. The importance score or the importance parameter can be updated through learning feedback that compares the calculated importance score with a set of learning training samples.

Meanwhile, according to embodiments of the present invention, an interpretation ID is determined based on at least one of the conditions of daily AHI and monthly AHI, and various sleep data interpretation contents according to embodiments of the present invention corresponding to the determined interpretation ID can be provided.

According to embodiments of the present invention, when the judgment for daily AHI is "good" and the judgment for monthly AHI is "normal", an 'A1-1' interpretation ID may be determined. When the judgment for daily AHI is "good" and the judgment for monthly AHI is "severe", an 'A1-2' interpretation ID may be determined. When the judgment for daily AHI is "good" and the judgment for monthly AHI is "good", an 'A2' interpretation ID may be determined. When the judgment for daily AHI is "normal" and the judgment for monthly AHI is "normal", an 'A3' interpretation ID may be determined. When the judgment for daily AHI is "normal" and the judgment for monthly AHI is "good", an 'A4' interpretation ID may be determined. When the judgment for daily AHI is "normal" and the judgment for monthly AHI is "severe", an 'A5' interpretation ID may be determined. If the judgment for daily AHI is "Serious" and the judgment for monthly AHI is "Good", an 'A6-1' interpretation ID can be determined. If the judgment for daily AHI is "Serious" and the judgment for monthly AHI is "Moderate", an 'A6-2' interpretation ID can be determined. If the judgment for daily AHI is "Serious" and the judgment for monthly AHI is "Serious", an 'A7' interpretation ID can be determined.

Meanwhile, at least one of the interpretation IDs of A1-1 to A7 described above is assigned with at least one of the interpretation contents of various sleep data according to the embodiments of the present invention described through [Table 1] and [Table 2], so that interpretation contents of sleep data corresponding to the interpretation ID can be provided.

Calculate importance score and interpret ID for Sleep Onset Latency category

Hereinafter, a method for calculating the importance score of the Sleep Onset Latency category according to one embodiment of the present invention is described in detail.

First, the name of the sleep category, which is expressed as a string, and the importance parameters, which are expressed as string and/or numeric data, are initialized. The name of the sleep category here may be expressed as Sleep Onset Latency. In addition, instead of Sleep Onset Latency, it may be expressed as various abbreviations, such as "SO" or "SOL". For example, in the present disclosure, the interpretation ID of the Sleep Onset Latency category is expressed using the abbreviation "SO", but is not limited thereto.

According to one embodiment of the present invention, a judgment on a daily condition for sleep onset latency may be performed based on a value of SOL measured in a user's last sleep session (hereinafter referred to as 'daily SOL'). Alternatively, an importance score calculation for a monthly condition category for sleep onset latency may be performed based on an average of SOL values measured in the user's last 30 sleep sessions (hereinafter referred to as 'monthly SOL').

According to one embodiment of the present invention, the judgment of the sleep onset latency condition is performed by outputting one of the strings of "good" or "serious" according to the SOL value. For example, if the SOL value is less than a threshold value, it may be judged as "good", and if it is greater than the threshold value, it may be judged as "serious". According to one embodiment of the present invention, the threshold value to be compared for performing the sleep onset latency condition judgment may be 30 minutes.

According to one embodiment of the present invention, daily SOL and monthly SOL values can be measured, and the difference between the daily SOL and the monthly SOL can be calculated based on the measured values. According to one embodiment of the present invention, the calculated difference values can also be rounded to a certain number of decimal places for calculation.

According to one embodiment of the present invention, a 'first importance score' can be calculated by dividing the case where the daily SOL exceeds a predetermined threshold and the case where it does not, and applying an importance parameter to each case. The 'first importance score' calculated here can be used as a value that serves as a basis for calculating the importance score of the sleep onset latency category. Here, the predetermined threshold may be 30 minutes, but is not limited thereto.

According to one embodiment of the present invention, when the daily SOL value is greater than a predetermined threshold value, a 'first importance score' can be calculated by adding a value obtained by applying a 'first importance parameter of SOL' on a linear graph for the difference between the daily SOL and the predetermined threshold value and a value obtained by applying a 'second importance parameter of SOL' on a graph including an upward convex curve for the difference between the daily SOL and the threshold value.

Alternatively, according to one embodiment of the present invention, when the daily SOL value is less than or equal to a predetermined threshold value, a value obtained by applying the 'third importance parameter of SOL' on a linear graph for the difference between the daily SOL and the predetermined threshold value and a value obtained by applying the 'fourth importance parameter of SOL' on a graph including an upward convex curve for the difference between the daily SOL and the threshold value may be added to calculate a 'first importance score'.

According to one embodiment of the present invention, by dividing the case where the difference between daily SOL and monthly SOL is positive and the case where it is not, and applying the importance parameter for each case, a 'second importance score' according to the change in SOL can be calculated. Here, the 'second importance score' calculated can be utilized as a value that serves as the basis for calculating the importance score of the sleep onset latency category.

According to one embodiment of the present invention, when the difference between daily SOL and monthly SOL is positive, the 'second importance score' can be calculated by applying the 'fifth importance parameter of SOL' on a linear graph for the difference between daily SOL and monthly SOL.

Alternatively, according to one embodiment of the present invention, if the difference between the daily SOL and the monthly SOL is less than or equal to 0, the 'second importance score' can be calculated by applying the 'sixth importance parameter of SOL' on a linear graph to the absolute value of the difference between the daily SOL and the monthly SOL.

According to one embodiment of the present invention, the importance score of the sleep onset latency category can be calculated by adding the calculated 'first importance score', 'second importance score', and the 'sleep onset latency bias' value, which is an importance parameter.

Meanwhile, according to embodiments of the present invention, an interpretation ID is determined based on at least one of the conditions of daily SOL and monthly SOL, and various sleep data interpretation contents according to embodiments of the present invention corresponding to the determined interpretation ID can be provided. Meanwhile, in case calculation is not possible due to an error in the value entered in daily SOL, SOL can be regarded as 30 minutes.

According to one embodiment of the present invention, the interpretation ID can be determined by comparing the difference between the daily SOL and the monthly SOL with a predetermined threshold value. Meanwhile, rounding to a predetermined number of digits can be applied to the difference between the daily SOL and the monthly SOL. In addition, the predetermined threshold value to be compared can be 20 minutes.

According to one embodiment of the present invention, if the judgment for daily SOL is "good" and the difference between the daily SOL and the monthly SOL is less than a negative threshold value (e.g., -20 minutes), an interpretation ID of 'SO1-1' may be determined if the judgment for monthly SOL is "severe", and an interpretation ID of 'SO1-2' may be determined if the judgment for monthly SOL is "good".

According to one embodiment of the present invention, if the judgment for daily SOL is “good” and the difference between daily SOL and monthly SOL is greater than or equal to a negative threshold value (e.g., -20 minutes), an interpretation ID of ‘SO2’ can be determined.

According to one embodiment of the present invention, if the judgment for daily SOL is "serious" and the difference between the daily SOL and the monthly SOL is less than a negative threshold value (e.g., -20 minutes), an interpretation ID of 'SO3-1' may be determined if the judgment for monthly SOL is "serious", and an interpretation ID of 'SO3-2' may be determined if the judgment for monthly SOL is "good".

According to one embodiment of the present invention, if the judgment for daily SOL is "serious" and the absolute value of the difference between the daily SOL and the monthly SOL is less than or equal to a threshold value (e.g., 20 minutes), an interpretation ID of 'SO4-1' may be determined if the judgment for monthly SOL is "serious", and an interpretation ID of 'SO4-2' may be determined if the judgment for monthly SOL is "good".

According to one embodiment of the present invention, if none of the conditions for determining the interpretation ID of SO1-1, the conditions for determining the interpretation ID of SO1-2, the conditions for determining the interpretation ID of SO2, the conditions for determining the interpretation ID of SO3-1, the conditions for determining the interpretation ID of SO3-2, the conditions for determining the interpretation ID of SO4-1, and the conditions for determining the interpretation ID of SO4-2 are satisfied, then if the judgment for monthly SOL is "serious", the interpretation ID of 'SO5-1' can be determined, and if the judgment for monthly SOL is "good", the interpretation ID of 'SO5-2' can be determined.

Meanwhile, at least one of the interpretation IDs of SO1-1 to SO5-2 described above is assigned with at least one of the interpretation contents of various sleep data according to the embodiments of the present invention described through [Table 3] and [Table 4], so that the interpretation contents of sleep data corresponding to the determined interpretation ID can be provided.

Calculating Importance Score and Determining Interpretation ID for Wake Time after Sleep Onset (WASO) Category

Hereinafter, a method for calculating the importance score of the Wake Time after Sleep Onset (WASO) category according to one embodiment of the present invention is described in detail.

First, the name of the sleep category, which is represented as a string, and the importance parameters, which are represented as string and/or numeric data, are initialized. The name of the sleep category here may be represented as Wake Time after Sleep Onset (WASO). In addition, instead of Wake Time after Sleep Onset (WASO), it may be represented by various abbreviations, such as "W" or "WASO". For example, in the present disclosure, the interpretation ID of the Wake Time after Sleep Onset (WASO) category is represented using the abbreviation "W", but is not limited thereto.

According to one embodiment of the present invention, a judgment on a daily condition for a wake-up time after sleep can be performed based on a value of WASO measured in a user's last sleep session (hereinafter referred to as 'daily WASO'). Alternatively, a judgment on a monthly condition for a wake-up time after sleep can be performed based on an average of WASO values measured in the user's last 30 sleep sessions (hereinafter referred to as 'monthly WASO'). For example, a judgment on a condition for a wake-up time after sleep can be performed by outputting one of the strings "good", "normal", or "severe" according to the WASO value. For example, if the daily WASO value is less than a lower threshold value, it can be judged as "good", if the daily WASO value is equal to or greater than the lower threshold value and less than the upper threshold value, it can be judged as "normal", and if the daily WASO value is equal to or greater than the upper threshold value, it can be judged as "severe". Similarly, if the monthly WASO value is less than the lower threshold value, it may be judged as “good”, if the monthly WASO value is greater than or equal to the lower threshold value and less than the upper threshold value, it may be judged as “normal”, and if the monthly WASO value is greater than or equal to the upper threshold value, it may be judged as “severe”. According to one embodiment of the present invention, the lower threshold value may be 5 minutes, and the upper threshold value may be 15 minutes, but is not limited thereto.

In addition, according to one embodiment of the present invention, an importance score calculation for a Wake Time after Sleep Onset (WASO) category may be performed based on comparing daily WASO and monthly WASO with predetermined thresholds, respectively. The predetermined threshold according to one embodiment of the present invention may be divided into a lower threshold value and an upper threshold value. For example, the lower threshold value may be 5 minutes, and the upper threshold value may be 15 minutes, but is not limited thereto.

According to one embodiment of the present invention, a first importance score of a post-sleep awakening time (WASO) category may be calculated by adding a value obtained by applying a 'first importance parameter of WASO' on a linear graph for daily WASO, a value obtained by applying a 'second importance parameter of WASO' on a graph including an upward convex curve for daily WASO, and a 'third importance parameter'. According to one embodiment of the present invention, the third importance parameter may be an importance score stored in a lookup table based on a value obtained by comparing monthly WASO and daily WASO, as described above through [Table 20].

Additionally, according to one embodiment of the present invention, a second importance score of the wake time after sleep (WASO) category can be calculated by dividing the cases into cases where the monthly WASO is greater than the daily WASO and cases where it is not.

According to one embodiment of the present invention, if the monthly WASO is greater than the daily WASO, a value obtained by applying the 'fourth importance parameter of WASO' on a linear graph to a value obtained by subtracting the daily WASO from the monthly WASO may be calculated as a 'second importance score'. Alternatively, according to one embodiment of the present invention, if the monthly WASO is less than the daily WASO, a value obtained by applying the 'fifth importance parameter of WASO' on a linear graph to a value obtained by subtracting the daily WASO from the monthly WASO may be calculated as a 'second importance score'.

According to one embodiment of the present invention, an importance score for the Wake Time after Sleep Onset (WASO) category can be calculated by adding the calculated first importance score and second importance score and the importance parameter 'W bias'.

According to one embodiment of the present invention, an interpretation ID of a Wake Time after Sleep Onset (WASO) category can be determined based on at least one of a user's daily WASO and monthly WASO.

According to one embodiment of the present invention, if the judgment for daily WASO is "good" and the judgment for monthly WASO is "normal", a 'W1-1' interpretation ID may be determined. Or, if the judgment for monthly WASO is "severe", a 'W1-2' interpretation ID may be determined.

According to one embodiment of the present invention, if the judgment for daily WASO is “good”, a ‘W2’ interpretation ID can be determined.

According to one embodiment of the present invention, when the judgment for daily WASO is “normal” and the judgment for monthly WASO is “severe”, a ‘W3’ interpretation ID can be determined.

According to one embodiment of the present invention, when the judgment for daily WASO is “normal” and the judgment for monthly WASO is “normal”, the ‘W4’ interpretation ID can be determined.

According to one embodiment of the present invention, if the judgment for daily WASO is “normal”, a ‘W5’ interpretation ID can be determined.

According to one embodiment of the present invention, when the judgment for daily WASO is "serious" and the judgment for monthly WASO is "good", a 'W6-1' interpretation ID may be determined. Or, when the judgment for monthly WASO is "normal", a 'W6-2' interpretation ID may be determined.

According to one embodiment of the present invention, if the judgment for daily WASO is “serious”, a ‘W7’ interpretation ID may be determined.

Meanwhile, at least one of the interpretation IDs of W1-1 to W6-2 described above is assigned with at least one of the interpretation contents of various sleep data according to the embodiments of the present invention described through [Table 5] and [Table 6], so that the interpretation contents of sleep data corresponding to the determined interpretation ID can be provided.

Calculate importance scores and interpret REM Latency category ID determination

Hereinafter, a method for calculating an importance score of a REM latency category according to one embodiment of the present invention is described in detail.

First, the name of the sleep category, which is expressed as a string, and the importance parameters, which are expressed as string and/or numeric data, are initialized. The name of the sleep category here may be expressed as REM Latency. In addition, instead of REM Latency, it may be expressed by various abbreviations, such as "RL". For example, in the present disclosure, the interpretation ID of the REM Latency category is expressed using the abbreviation "RL", but is not limited thereto.

According to one embodiment of the present invention, an importance score for a REM latency category may be calculated based on a comparison of the REM latency measured in the user's last sleep session with a predetermined threshold value. The predetermined threshold value according to one embodiment of the present invention may be, but is not limited to, 45 minutes.

Meanwhile, if the calculation cannot be performed due to an error in the value entered for REM Latency, this value may be considered as 0. In this case, the importance score for the REM Latency category may be calculated as the lowest value among the possible values. According to an embodiment of the present invention, the lowest value among the importance scores may be -100, but is not limited thereto.

According to one embodiment of the present invention, the importance score of the REM latency category can be calculated by dividing the case where the REM latency is less than a predetermined threshold value and the case where it is not, and applying an importance parameter to each case.

According to one embodiment of the present invention, when the REM sleep latency is smaller than a predetermined threshold value, the importance score of the REM sleep latency category can be calculated by adding together a value obtained by applying the 'first importance parameter of RL' on a linear graph for the difference between the REM sleep latency and the predetermined threshold value, a value obtained by applying the 'second importance parameter of RL' on a graph including an upward convex curve for the difference between the REM sleep latency and the predetermined threshold value, and a value of the 'REM sleep onset latency bias' which is an importance parameter.

Alternatively, according to one embodiment of the present invention, when the REM sleep latency is greater than or equal to a predetermined threshold value, the importance score of the REM sleep latency category may be calculated by adding together a value obtained by applying the 'third importance parameter of RL' on a linear graph for the difference between the REM sleep latency and the predetermined threshold value, a value obtained by applying the 'fourth importance parameter of RL' on a graph including an upward convex curve for the difference between the REM sleep latency and the predetermined threshold value, and a value of the 'REM sleep onset latency bias' which is an importance parameter.

According to one embodiment of the present invention, an interpretation ID of a REM sleep latency category may be determined based on a REM sleep stage latency measured in a user's last sleep session (hereinafter referred to as 'daily REM Latency') and an average of REM sleep stage latency values measured in the user's last 30 sleep sessions (hereinafter referred to as 'monthly REM Latency'). In addition, an interpretation ID of a REM sleep stage latency category may be determined based on a comparison between the measured REM Latency and a predetermined threshold value.

According to one embodiment of the present invention, if the daily REM Latency is less than a predetermined threshold, an interpretation ID of 'RL1-1' may be determined if the monthly REM Latency is less than a predetermined threshold, and if the monthly REM Latency is greater than or equal to the predetermined threshold, an interpretation ID of 'RL1-2' may be determined.

According to one embodiment of the present invention, when the daily REM Latency is greater than or equal to a predetermined threshold, an interpretation ID of 'RL2' may be determined. Meanwhile, a predetermined threshold value that is a target for comparison with REM Latency sleep data to determine an interpretation ID according to one embodiment of the present invention may be 45 minutes, but is not limited thereto.

At least one of the interpretation IDs of the above-described RL1-1, RL1-2 and RL2 is assigned with at least one of the interpretation contents of various sleep data according to the embodiments of the present invention described through [Table 7] and [Table 8], so that the interpretation contents of sleep data corresponding to the determined interpretation ID can be provided.

Calculate importance score and interpret REM Ratio category ID determination

Hereinafter, a method for calculating an importance score of a REM Ratio category according to one embodiment of the present invention is described in detail.

First, the name of the sleep category, which is expressed as a string, and the importance parameters, which are expressed as string and/or numeric data, are initialized. The name of the sleep category here may be expressed as REM Ratio. In addition, instead of REM Ratio, it may be expressed as various abbreviations, such as "R" or "REM". For example, in the present disclosure, the interpretation ID of the REM Ratio category is expressed using the abbreviation "R", but is not limited thereto.

According to one embodiment of the present invention, an importance score for a REM sleep stage ratio (REM Ratio) category may be calculated based on comparing a REM sleep stage ratio (REM Ratio) measured in a user's last sleep session with a predetermined threshold value. The predetermined threshold value according to one embodiment of the present invention may be divided into a lower threshold value and an upper threshold value. For example, the lower threshold value may be 15%, and the upper threshold value may be 30%, but is not limited thereto.

In addition, the importance score of the REM Ratio category can be further considered based on the REM Cycle Gradient measured in the user's previous sleep session. The REM Cycle Gradient refers to the average change in the length of the duration of the REM sleep stage that occurs during sleep. It is generally known that the duration of the REM sleep stage changes as sleep progresses. For example, if three REM sleep stages occur during sleep, and the duration of the first REM sleep stage lasts 20 minutes, the duration of the second REM sleep stage lasts 40 minutes, and the duration of the third REM sleep stage lasts 60 minutes, the duration of the REM sleep stage can be calculated to have increased by 20 minutes on average. In this case, the REM Cycle Gradient can be calculated to be +20 minutes. Conversely, if the average duration of REM sleep stages decreases, the REM Cycle Gradient can have a negative value. For example, if three REM stages occur during sleep, and the first REM stage lasts 50 minutes, the second REM stage lasts 30 minutes, and the third REM stage lasts 70 minutes, the first change in REM sleep duration is -20 minutes, and the change in REM sleep duration is +40 minutes, so the average of the two changes is +10 minutes. In this case, the REM Cycle Gradient can be calculated as +10 minutes.

Meanwhile, if calculations cannot be performed due to errors in the values entered for REM Ratio and REM Cycle Gradient, each can be considered as 0.

According to one embodiment of the present invention, when the REM sleep stage ratio (REM Ratio) is less than a lower threshold value, the importance score of the REM sleep stage ratio (REM Ratio) category can be calculated by applying the value of the 'first importance parameter of R' on a linear graph to the difference between the lower threshold value and the REM sleep stage ratio (REM Ratio) and adding the 'REM Ratio bias' which is an importance parameter.

According to one embodiment of the present invention, when the REM sleep stage ratio (REM Ratio) is greater than the upper threshold value, the importance score of the REM sleep stage ratio (REM Ratio) category can be calculated by applying the value of the 'second importance parameter of R' on a linear graph to the difference between the REM sleep stage ratio (REM Ratio) value and the upper threshold value and adding the 'REM Ratio bias' which is an importance parameter.

According to one embodiment of the present invention, if the REM sleep stage ratio (REM Ratio) is equal to or greater than the lower threshold value and equal to or less than the upper threshold value, and if the REM cycle gradient is greater than 0, the importance score of the REM sleep stage ratio (REM Ratio) category can be calculated by adding one or more importance parameters to a value obtained by applying the 'third importance parameter of R' on a linear graph for the REM cycle gradient.

According to one embodiment of the present invention, if the REM sleep stage ratio (REM Ratio) is equal to or greater than the lower threshold value and equal to or less than the upper threshold value, and if the REM cycle gradient is less than or equal to 0, the importance score of the REM sleep stage ratio (REM Ratio) category can be calculated by adding one or more importance parameters to a value obtained by applying the 'fourth importance parameter of R' on a linear graph for the absolute value of the REM cycle gradient.

According to one embodiment of the present invention, an interpretation ID of a REM sleep stage ratio (REM Ratio) category may be determined based on at least one of a REM sleep stage ratio in a user's last sleep session (hereinafter referred to as 'daily REM Ratio'), a REM sleep stage ratio measured in the user's last 30 sleep sessions (hereinafter referred to as 'monthly REM Ratio'), and a REM cycle gradient.

According to one embodiment of the present invention, if the daily REM Ratio is less than the lower threshold value, and if the monthly REM Ratio is less than the lower threshold value, an 'R1-1' interpretation ID may be determined. If the monthly REM Ratio is greater than or equal to the lower threshold value and less than or equal to the upper threshold value, an 'R1-2' interpretation ID may be determined. If the monthly REM Ratio is greater than the upper threshold value, an 'R1-3' interpretation ID may be determined.

According to one embodiment of the present invention, when the daily REM Ratio is greater than or equal to the lower threshold value and less than or equal to the upper threshold value, an interpretation ID may be determined based on whether the REM Cycle Gradient is positive or negative. When the REM Cycle Gradient is positive, if the monthly REM Ratio is less than the lower threshold value, an 'R2-1' interpretation ID may be determined. If the monthly REM Ratio is greater than or equal to the lower threshold value and less than or equal to the upper threshold value, an 'R2-2' interpretation ID may be determined. If the monthly REM Ratio is greater than the upper threshold value, an 'R2-3' interpretation ID may be determined. On the other hand, when the REM Cycle Gradient is negative, if the monthly REM Ratio is less than the lower threshold value, an 'R3-1' interpretation ID may be determined. If the monthly REM Ratio is greater than or equal to the lower threshold value and less than or equal to the upper threshold value, an 'R3-2' interpretation ID may be determined. If the monthly REM Ratio is greater than the upper threshold, an 'R3-3' interpretation ID can be determined.

According to one embodiment of the present invention, if the daily REM Ratio is greater than the upper threshold value and if the monthly REM Ratio is less than the lower threshold value, an 'R4-1' interpretation ID may be determined. If the monthly REM Ratio is greater than or equal to the lower threshold value and less than or equal to the upper threshold value, an 'R4-2' interpretation ID may be determined. If the monthly REM Ratio is greater than the upper threshold value, an 'R4-3' interpretation ID may be determined.

Meanwhile, in order to determine an interpretation ID according to one embodiment of the present invention, the lower threshold value to be compared with each sleep data may be 15%, and the upper threshold value may be 30%, but is not limited thereto.

At least one of the interpretation IDs of R1-1 to R4-3 described above is assigned with at least one of the interpretation contents of various sleep data according to the embodiments of the present invention described through [Table 9] and [Table 10], so that the interpretation contents of sleep data corresponding to the determined interpretation ID can be provided.

Calculate importance score and interpret ID for Deep Ratio category

Hereinafter, a method for calculating the importance score of the Deep Ratio category according to one embodiment of the present invention is described in detail.

First, the name of the sleep category, which is expressed as a string, and the importance parameters, which are expressed as string and/or numeric data, are initialized. The name of the sleep category here may be expressed as Deep Ratio. In addition, instead of Deep Ratio, it may be expressed as various abbreviations, such as "D" or "Deep". For example, in the present disclosure, the interpretation ID of the Deep Ratio category is expressed using the abbreviation "D", but is not limited thereto.

According to one embodiment of the present invention, an importance score calculation for a Deep Ratio category may be performed based on comparing a deep sleep stage ratio (hereinafter referred to as 'daily Deep Ratio') measured in a user's last sleep session with a predetermined threshold value. In addition, an importance score calculation for a Deep Ratio category may be performed based on comparing a deep sleep stage ratio (hereinafter referred to as 'monthly Deep Ratio') measured in a user's last 30 sleep sessions with a predetermined threshold value. The predetermined threshold value according to one embodiment of the present invention may be divided into a lower threshold value and an upper threshold value. For example, the lower threshold value may be 15%, and the upper threshold value may be 30%, but is not limited thereto.

In addition, the importance score calculation of the REM Ratio category may further consider the ratio of the deep sleep stage in the first half of sleep (hereinafter referred to as 'First Half Deep Ratio') measured in the user's last sleep session. Here, the first half of sleep refers to the period before half of the time from the time of falling asleep to the time of waking up. According to one embodiment of the present invention, the importance score calculation for the Deep Ratio category may be performed based on comparing the ratio of the deep sleep stage in the first half of sleep with a predetermined threshold value. For example, the predetermined threshold value to be compared with the deep sleep stage ratio in the first half of sleep may be 50%, but is not limited thereto.

Meanwhile, according to one embodiment of the present invention, if calculation is not possible due to an error in the values entered in the Daily Deep Ratio, Monthly Deep Ratio, or First Half Deep Ratio, each value may be regarded as 0.

According to one embodiment of the present invention, when the daily Deep Ratio is less than the lower threshold value and the monthly Deep Ratio is also less than the lower threshold value, the importance score of the deep sleep stage ratio (Deep Ratio) category can be calculated by adding one or more importance parameters to a value obtained by applying the 'first importance parameter of D' on a linear graph to the absolute value of the difference between the daily Deep Ratio and the lower threshold value.

According to one embodiment of the present invention, when the daily Deep Ratio is less than the lower threshold value and the monthly Deep Ratio is greater than the lower threshold value, the importance score of the deep sleep stage ratio (Deep Ratio) category can be calculated by adding one or more importance parameters to a value obtained by applying the 'second importance parameter of D' on a linear graph to the absolute value of the difference between the daily Deep Ratio and the lower threshold value.

According to one embodiment of the present invention, when the daily Deep Ratio is greater than the lower threshold value and the monthly Deep Ratio is less than the lower threshold value, the importance score of the deep sleep stage ratio (Deep Ratio) category can be calculated by adding one or more importance parameters to a value obtained by applying the 'third importance parameter of D' on a linear graph to the absolute value of the difference between the daily Deep Ratio and the lower threshold value.

According to one embodiment of the present invention, when the daily Deep Ratio is greater than the lower threshold value and the monthly Deep Ratio is also greater than the lower threshold value, the importance score of the deep sleep stage ratio (Deep Ratio) category can be calculated by adding one or more importance parameters to a value obtained by applying the 'fourth importance parameter of D' on a linear graph to the absolute value of the difference between the daily Deep Ratio and the lower threshold value.

According to one embodiment of the present invention, an interpretation ID of a Wake Time after Sleep Onset (WASO) category can be determined based on at least one of a user's Daily Deep Ratio, Monthly Deep Ratio, and First Half Deep Ratio.

According to one embodiment of the present invention, if the daily Deep Ratio is less than the lower threshold value and the monthly Deep Ratio is less than the lower threshold value, the 'D1' interpretation ID may be determined if the First Half Deep Ratio is greater than a predetermined threshold value, and the 'D2' interpretation ID may be determined if the First Half Deep Ratio is less than the predetermined threshold value.

According to one embodiment of the present invention, if the daily Deep Ratio is less than the lower threshold value and the monthly Deep Ratio is greater than the lower threshold value, the 'D3' interpretation ID can be determined.

According to one embodiment of the present invention, if the daily Deep Ratio is greater than the lower threshold value and the monthly Deep Ratio is less than the lower threshold value, the 'D4' interpretation ID can be determined.

According to one embodiment of the present invention, if the daily Deep Ratio is greater than the lower limit threshold value and the monthly Deep Ratio is greater than the lower limit threshold value, if the First Half Deep Ratio is greater than a predetermined threshold value, a 'D5' interpretation ID may be determined, and if the First Half Deep Ratio is less than the predetermined threshold value, a 'D6' interpretation ID may be determined. Meanwhile, in order to determine the interpretation ID according to one embodiment of the present invention, the lower limit threshold value to be compared with each sleep data may be 15%, the upper limit threshold value may be 30%, and the predetermined threshold value may be 50%, but the present invention is not limited thereto.

At least one of the interpretation IDs of D1 to D6 described above is assigned with at least one of the interpretation contents of various sleep data according to the embodiments of the present invention described through [Table 11] and [Table 12], so that the interpretation contents of sleep data corresponding to the determined interpretation ID can be provided.

Calculate and interpret the importance score of the First Cycle Sleep Quality category to determine ID

Hereinafter, a method for calculating the importance score of the First Cycle Sleep Quality category according to one embodiment of the present invention will be described in detail.

First, the name of the sleep category, which is represented as a string, and the importance parameters, which are represented as string and/or numeric data, are initialized. The name of the sleep category here may be represented as First Cycle Sleep Quality. In addition, instead of First Cycle Sleep Quality, it may be represented by various abbreviations, such as "F" or "FCS". For example, in the present disclosure, the interpretation ID of the First Cycle Sleep Quality category is represented using the abbreviation "F", but is not limited thereto.

According to one embodiment of the present invention, an importance score calculation for a First Cycle Sleep Quality category can be performed based on a ratio of wakefulness after sleep onset before first REM sleep stage (WASO ratio before first REM) and a ratio of deep sleep stage before first REM sleep stage (Deep ratio before first REM) measured in a user's last sleep session.

Meanwhile, if the calculation cannot be performed due to an error in the entered values for the ratio of wakefulness after sleep before the first REM stage (WASO ratio before first REM) or the ratio of deep sleep before the first REM stage (Deep ratio before first REM), each can be regarded as 0.

According to one embodiment of the present invention, an importance score can be calculated by applying an importance parameter for each case based on a comparison of the size of the wake-up time ratio before the first REM sleep stage (WASO ratio before first REM) with a first threshold value and a comparison of the size of the deep sleep stage ratio before the first REM sleep stage (Deep ratio before first REM) with a second threshold value. Here, the first threshold value and the second threshold value can be different values, but can also be the same value. For example, the first threshold value can be 10% and the second threshold value can be 33%.

If the ratio of wake time after sleep to first REM sleep stage (WASO ratio before first REM) is greater than the first threshold, the importance score can be calculated by applying the difference between the ratio of wake time after sleep to first REM sleep stage (WASO ratio before first REM) and the first threshold and the 'first importance parameter of F'.

If the ratio of wakefulness after sleep before the first REM sleep stage (WASO ratio before first REM) is less than or equal to the first threshold, the importance score can be calculated according to an additional condition. In this case, if the ratio of deep sleep before the first REM sleep stage (Deep ratio before first REM) is greater than a predetermined threshold, the importance score can be calculated by applying the difference between the ratio of wakefulness after sleep before the first REM sleep stage (WASO ratio before first REM) and the first threshold, the difference between the ratio of deep sleep before the first REM sleep stage (Deep ratio before first REM) and the second threshold, the 'second importance parameter of F', and the 'third importance parameter of F'. In addition, the importance score can be calculated by additionally adding the predetermined parameter values. If in this case, the ratio of deep sleep stages before the first REM sleep stage (Deep ratio before first REM) is less than or equal to a predetermined threshold, the importance score can be calculated by applying the difference between the ratio of wakefulness after sleep before the first REM sleep stage (WASO ratio before first REM) and the first threshold, the difference between the ratio of deep sleep stages before the first REM sleep stage (Deep ratio before first REM) and the second threshold, the '4th importance parameter of F', and the '5th importance parameter of F'. In addition, the importance score can be calculated by additionally adding the values of predetermined parameters.

According to one embodiment of the present invention, the importance score of the First Cycle Sleep Quality category can be calculated by adding the 'F bias' value, which is an importance parameter, to the calculated importance score.

Meanwhile, according to embodiments of the present invention, an interpretation ID is determined according to WASO ratio before first REM and Deep ratio before first REM conditions, and various sleep data interpretation contents according to embodiments of the present invention can be provided based on the determined interpretation ID. Meanwhile, if calculation is not possible due to an error in the value entered in WASO ratio before first REM or Deep ratio before first REM, each value can be regarded as 0.

According to one embodiment of the present invention, when the WASO ratio before first REM is greater than a first threshold value, the interpretation ID of 'F1' can be determined.

According to one embodiment of the present invention, when the WASO ratio before first REM is less than or equal to a first threshold value, an additional condition may be considered to determine a sleep interpretation ID. When the Deep ratio before first REM is less than or equal to a second threshold value, an interpretation ID of 'F2' may be determined. Alternatively, when the Deep ratio before first REM is greater than the second threshold value, an interpretation ID of 'F3' may be determined.

Meanwhile, in order to determine an interpretation ID according to one embodiment of the present invention, the first threshold value to be compared with each sleep data may be 10%, and the second threshold value may be 33%, but is not limited thereto.

Meanwhile, at least one of the interpretation IDs of F1, F2 and F3 described above is assigned with at least one of the interpretation contents of various sleep data according to the embodiments of the present invention described through [Table 13] and [Table 14], so that the interpretation contents of sleep data corresponding to the determined interpretation ID can be provided.

Calculate importance score and determine interpretation ID for Number of Awakening category

Hereinafter, a method for calculating the importance score of the Number of Awakening category according to one embodiment of the present invention is described in detail.

First, the name of the sleep category represented as a string and the importance parameters represented as string and/or numeric data are initialized. The name of the sleep category here may be represented as Number of Awakenings. In addition, instead of Number of Awakenings, it may be represented by various abbreviations such as "N", "NoA', "Num" or "M". For example, in the present disclosure, the interpretation ID of the Number of Awakenings category is represented using the abbreviation "M", but is not limited thereto.

According to one embodiment of the present invention, an importance score for a Number of Awakening category may be calculated based on comparing the Number of Awakenings measured during a user's last sleep session with a predetermined threshold value. The predetermined threshold value according to one embodiment of the present invention may be divided into a lower threshold value and an upper threshold value. For example, the lower threshold value may be 15 times, and the upper threshold value may be 30 times, but is not limited thereto.

According to one embodiment of the present invention, when the Number of Awakenings during sleep is greater than the upper threshold value, the importance score of the Number of Awakenings category can be calculated by adding a value obtained by applying the 'first importance parameter of M' on a linear graph for the difference between the Number of Awakenings during sleep and the upper threshold value and a value obtained by applying the 'second importance parameter of M' on a curved graph for the difference between the Number of Awakenings during sleep and the upper threshold value.

According to one embodiment of the present invention, when the Number of Awakenings during sleep is less than or equal to an upper threshold value and greater than a lower threshold value, the importance score for the Number of Awakenings during sleep category may be calculated as the lowest value among the possible values. According to one embodiment of the present invention, the lowest value among the importance scores may be -100, but is not limited thereto.

According to one embodiment of the present invention, when the Number of Awakenings during sleep is equal to or lower than a lower threshold value, the importance score of the Number of Awakenings category can be calculated by adding together a value obtained by applying the 'third importance parameter of M' on a linear graph to the absolute value of the difference between the Number of Awakenings during sleep and the upper threshold value, and a value obtained by applying the 'fourth importance parameter of M' on a curved graph to the difference between the Number of Awakenings during sleep and the upper threshold value.

According to one embodiment of the present invention, an importance score of the Number of Awakening category may be calculated by adding one or more importance parameters to the calculated importance score.

According to one embodiment of the present invention, a curve graph utilized for calculating the importance score of the number of awakenings during sleep category may include at least one of an upwardly convex curve and a downwardly convex curve. The upwardly convex curve may be a curve in the form of a quadratic function, but is not limited thereto.

According to one embodiment of the present invention, an interpretation ID of a Number of Awakenings category may be determined based on at least one of the number of awakenings during sleep measured in a user's last sleep session (hereinafter referred to as 'daily Number of Awakenings') and the number of awakenings during sleep measured in the user's last 30 sleep sessions (hereinafter referred to as 'monthly Number of Awakenings'). In addition, an interpretation ID of the Number of Awakenings category may be determined based on a comparison between the measured Number of Awakenings and a predetermined threshold value.

According to one embodiment of the present invention, if the daily Number of Awakening is greater than the upper threshold value, an interpretation ID of 'M1-1' may be determined if the monthly Number of Awakening is less than or equal to the upper threshold value, and if the monthly Number of Awakening is greater than the upper threshold value, an interpretation ID of 'M1-2' may be determined.

According to one embodiment of the present invention, if the daily Number of Awakening is less than or equal to an upper threshold value, an interpretation ID of 'M2-1' may be determined if the monthly Number of Awakening is less than or equal to the upper threshold value, and if the monthly Number of Awakening is greater than the upper threshold value, an interpretation ID of 'M2-2' may be determined. Meanwhile, in order to determine the interpretation ID according to one embodiment of the present invention, an upper threshold value to be compared with the Number of Awakening sleep data may be 30 times, but is not limited thereto.

At least one of the interpretation IDs of M1-1 to M2-2 described above is assigned with at least one of the interpretation contents of various sleep data according to the embodiments of the present invention described through [Table 15] and [Table 16], so that the interpretation contents of sleep data corresponding to the determined interpretation ID can be provided.

Calculate and interpret importance scores for Total Sleep Time category ID determination

Hereinafter, a method for calculating the importance score of the Total Sleep Time category according to one embodiment of the present invention is described in detail.

First, the name of the sleep category, which is expressed as a string, and the importance parameters, which are expressed as string and/or numeric data, are initialized. The name of the sleep category here may be expressed as Total Sleep Time. In addition, instead of Total Sleep Time, it may be expressed as various abbreviations, such as "T" or "TST". For example, in the present disclosure, the interpretation ID of the Total Sleep Time category is expressed using the abbreviation "TST", but is not limited thereto.

According to one embodiment of the present invention, an importance score calculation for a Total Sleep Time category may be performed based on comparing the Total Sleep Time measured in the user's last sleep session with a predetermined threshold value. The predetermined threshold value according to one embodiment of the present invention may be divided into one or more threshold values. For example, the first threshold value may be 240 minutes, the second threshold value may be 360 minutes, the third threshold value may be 480 minutes, and the fourth threshold value may be 600 minutes, but is not limited thereto.

In addition, the importance score calculation for the Total Sleep Time category may be performed based on comparing the number of cycles in which sleep stages appear, or the number of sleep cycles, measured in the user's last sleep session with a predetermined threshold value. The predetermined threshold value according to one embodiment of the present invention may be divided into a lower threshold value and an upper threshold value. For example, the lower threshold value may be 4 times, and the upper threshold value may be 8 times, but is not limited thereto.

According to one embodiment of the present invention, when the Total Sleep Time is less than a first threshold value, a value obtained by applying the 'first importance parameter of TST' on a linear graph for the absolute value of the difference between the Total Sleep Time and the first threshold value, and a value obtained by applying the 'second importance parameter of TST' on a graph including an upward convex curve for the difference between the Total Sleep Time and the first threshold value are added together to calculate the importance score of the Total Sleep Time category. Meanwhile, one or more TST importance parameters may be further added here to calculate the importance score of the Total Sleep Time category.

According to one embodiment of the present invention, when the Total Sleep Time is equal to or greater than the first threshold value and less than the second threshold value, a value obtained by applying the 'third importance parameter of TST' on a linear graph to the absolute value of the difference between the Total Sleep Time and the second threshold value can be calculated as an importance score of the Total Sleep Time category. Meanwhile, one or more TST importance parameters can be further added here to calculate the importance score of the Total Sleep Time category.

According to one embodiment of the present invention, when the Total Sleep Time is greater than or equal to the second threshold and less than the third threshold, the importance score of the Total Sleep Time category can be calculated to be a sufficiently low value. For example, the sufficiently low importance score can be 0.

According to one embodiment of the present invention, when the Total Sleep Time is equal to or greater than a third threshold value and less than a fourth threshold value, a value obtained by applying the 'fourth importance parameter of TST' on a linear graph to the absolute value of the difference between the Total Sleep Time and the third threshold value can be calculated as an importance score of the Total Sleep Time category. Meanwhile, one or more TST importance parameters can be further added here to calculate the importance score of the Total Sleep Time category.

According to one embodiment of the present invention, when the Total Sleep Time is equal to or greater than a fourth threshold value, a value obtained by applying the 'fifth importance parameter of TST' on a linear graph for the absolute value of the difference between the Total Sleep Time and the fourth threshold value, and a value obtained by applying the 'sixth importance parameter of TST' on a graph including an upward convex curve for the difference between the Total Sleep Time and the fourth threshold value are added together to calculate the importance score of the Total Sleep Time category. Meanwhile, one or more TST importance parameters may be further added here to calculate the importance score of the Total Sleep Time category.

According to one embodiment of the present invention, when calculating the importance score of the Total Sleep Time category, the number of cycles of sleep stages (or the number of sleep cycles) measured in the user's last sleep session may be further taken into consideration.

According to one embodiment of the present invention, if the Total Sleep Time is greater than the first threshold value and less than the fourth threshold value, and if the Number of Sleep Cycles is less than the lower threshold value, the importance score of the Total Sleep Time category can be calculated by adding a value obtained by applying the '6th importance parameter of TST' to the calculated importance score on a linear graph for the absolute value of the difference between the Number of Sleep Cycles and the lower threshold value. If the Number of Sleep Cycles is greater than or equal to the lower threshold value and less than the upper threshold value, the calculated importance score can be calculated as the importance score of the Total Sleep Time category as it is. If the Number of Sleep Cycles is greater than or equal to the upper threshold, the importance score of the Total Sleep Time category can be calculated by adding the value obtained by applying the '7th Importance Parameter of TST' on a linear graph to the absolute value of the difference between the Number of Sleep Cycles and the upper threshold to the calculated importance score. Meanwhile, the importance score of the Total Sleep Time category can be calculated by further adding the TST importance parameter to the calculated importance category.

According to one embodiment of the present invention, an interpretation ID of a Total Sleep Time category may be determined based on at least one of the Total Sleep Time and the Number of Sleep Cycles measured in a user's last sleep session.

According to one embodiment of the present invention, if the total sleep time is less than a first threshold value, the interpretation ID of 'TS1' can be determined.

According to one embodiment of the present invention, if the total sleep time is greater than the fourth threshold value, the interpretation ID of 'TS2' can be determined.

According to one embodiment of the present invention, if the Total Sleep Time is greater than or equal to a first threshold value and less than a second threshold value, and if the Number of Sleep Cycles is less than a lower threshold value, an interpretation ID of 'TS3' may be determined. If the Number of Sleep Cycles is greater than or equal to the lower threshold value and less than the upper threshold value, an interpretation ID of 'TS4' may be determined. If the Number of Sleep Cycles is greater than the upper threshold value, an interpretation ID of 'TS5' may be determined.

According to one embodiment of the present invention, if the Total Sleep Time is greater than or equal to the second threshold value and less than the third threshold value, and if the Number of Sleep Cycles is less than the lower threshold value, an interpretation ID of 'TS6' may be determined. If the Number of Sleep Cycles is greater than or equal to the lower threshold value and less than the upper threshold value, an interpretation ID of 'TS7' may be determined. If the Number of Sleep Cycles is greater than the upper threshold value, an interpretation ID of 'TS8' may be determined.

According to one embodiment of the present invention, if the total sleep time is equal to or greater than a third threshold value and the number of sleep cycles is less than a lower threshold value, an interpretation ID of 'TS9' may be determined. If the number of sleep cycles is equal to or greater than the lower threshold value and less than the upper threshold value, an interpretation ID of 'TS10' may be determined. If the number of sleep cycles is greater than the upper threshold value, an interpretation ID of 'TS11' may be determined.

Meanwhile, in order to determine an interpretation ID according to one embodiment of the present invention, the first threshold value to be compared with the total sleep time sleep data may be 240 minutes, the second threshold value may be 360 minutes, the third threshold value may be 480 minutes, and the fourth threshold value may be 600 minutes, but is not limited thereto. In addition, in order to determine an interpretation ID according to one embodiment of the present invention, the lower threshold value to be compared with the number of sleep cycles sleep data may be 4 times, and the upper threshold value may be 8 times, but is not limited thereto.

At least one of the interpretation IDs of the above-described TS1 to TS11 may be assigned at least one of the interpretation contents of various sleep data according to the embodiments of the present invention described through [Table 17] and [Table 18], so that the interpretation contents of sleep data corresponding to the determined interpretation ID may be provided.

Figures 42a to 42c are conceptual diagrams showing exemplary systems in which services according to the present invention can be implemented.

Referring to FIG. 42a, the system of the present invention may include at least one of a computing device (100-3), a server (200-3), a user terminal (300-3), and a network. As illustrated in FIG. 42a, at least one of the computing device (100-3), the server (200-3), and the user terminal (300-3) may transmit and receive data to and from each other via the network. Here, the system illustrated in FIG. 42a is illustrated according to one embodiment of the present invention, and its components are not limited to the embodiment illustrated in FIG. 42a, and may be added, changed, or deleted as needed. For example, at least one of the computing device (100-3), the user terminal (300-3), and the server (200-3) may be configured in multiple units.

Referring to FIG. 42b, the system of the present invention may include at least one of a user terminal (300-3), a server (200-3), and a network. Here, as illustrated in FIG. 42b, a service according to the present invention may be provided through a system configured to include at least one of a user terminal (300-3), a server (200-3), and a network, without a separate computing device (100-3). As illustrated in FIG. 42b, the server (200-3) and the user terminal (300-3) according to one embodiment of the present invention may transmit and receive data to and from each other through a network.

The present invention can provide a server (200-3) that receives environmental sensing information from a user terminal (300-3) that generates a broadcast trigger for initiating sleep measurement and/or a sleep analysis trigger including time point information for starting sleep analysis of the measured sleep, and performs sleep analysis. Specifically, the server (200-3) includes a communication module (270-3) that receives environmental sensing information from the user terminal (300-3), a processor (210-3) that performs sleep analysis based on the received environmental sensing information, and a memory (220-3) that stores a sleep analysis model for performing sleep analysis. Here, the communication module (270-3) is configured to receive environmental sensing information acquired by the user terminal (300-3) from the time point of generation of the broadcast trigger generated by the user terminal (300-3).

According to one embodiment of the present invention, the communication module (270-3) may be configured to receive information on the start time of sleep analysis of the measured sleep from the user terminal (300-3). In addition, the processor (210-3) is configured to delete environmental sensing information prior to the start time of sleep analysis of the measured sleep among the environmental sensing information when performing the sleep analysis based on the received environmental sensing information. In addition, the processor (210-3) is configured to perform sleep analysis only on environmental sensing information after the start time of sleep analysis of the measured sleep among the environmental sensing information when performing the sleep analysis based on the received environmental sensing information. In addition, the processor (210-3) is configured to acquire sleep sound information only on environmental sensing information after the start time of sleep analysis of the measured sleep among the environmental sensing information when performing the sleep analysis based on the received environmental sensing information. In addition, when the processor (210-3) performs the sleep analysis based on the received environmental sensing information, it is configured to acquire sleep state information only from the environmental sensing information after the point in time when the sleep analysis of the measured sleep starts among the environmental sensing information.

Referring to FIG. 42c, the system of the present invention may include a user terminal (300-3), a computing device (100-3), and a network. Here, as illustrated in FIG. 42c, a service according to the present invention may be provided through a system configured to include at least one of a user terminal (300-3), a computing device (100-3), and a network, without a separate server (200-3). As illustrated in FIG. 42c, the computing device (100-3) and the user terminal (300-3) of the present invention may transmit and receive data to and from each other through a network. Here, the system illustrated in FIG. 42c is illustrated according to one embodiment of the present invention, and its components are not limited to the embodiment illustrated in FIG. 42c, and may be added, changed, or deleted as needed. For example, at least one of the user terminal (300-3) and the computing device (100-3) may be configured in multiple units. In addition, as illustrated in FIG. 42c, even if a server (200-3) according to one embodiment of the present invention is not separately provided, a service according to one embodiment of the present invention can be provided by having at least one of a user terminal (300-3) and a computing device (100-3) perform at least one of the various operations of the server (200-3) on behalf of the server.

FIGS. 42d and 42e are conceptual diagrams illustrating an exemplary system in which a service for providing content related to a user's sleep state information according to the present invention can be implemented.

Referring to FIG. 42d, a system according to an embodiment of the present invention may include at least one of a sleep analysis server (200a-3), a content provision server (200b-3), a user terminal (300-3), and a network. As illustrated in FIG. 42d, at least one of the sleep analysis server (200a-3), the content provision server (200b-3), and the user terminal (300-3) according to an embodiment of the present invention may transmit and receive data to and from each other via a network. Here, the system illustrated in FIG. 42d is illustrated according to an embodiment of the present invention, and its components are not limited to the embodiment illustrated in FIG. 42d, and may be added, changed, or deleted as needed. For example, at least one of the sleep analysis server (200a-3), the content provision server (200b-3), and the user terminal (300-3) may be configured in multiple units. In addition, as illustrated in FIG. 42d, even if a computing device (100-3) according to one embodiment of the present invention is not separately provided, a service according to one embodiment of the present invention can be provided by having at least one of a sleep analysis server (200a-3), a content provision server (200b-3), and a user terminal (300-3) perform at least one of the various operations of the computing device (100-3) on behalf of the computing device (100-3).

Referring to FIG. 42e, the system of the present invention may include at least one of a sleep analysis server (200a-3), a content provision server (200b-3), a computing device (100-3), and a network. As illustrated in FIG. 42e, at least one of the sleep analysis server (200a-3), the content provision server (200b-3), and the computing device (100-3) may transmit and receive data to and from each other via a network. Here, the system illustrated in FIG. 42e is illustrated according to one embodiment of the present invention, and its components are not limited to the embodiment illustrated in FIG. 1e, and may be added, changed, or deleted as needed. For example, at least one of the sleep analysis server (200a-3), the content provision server (200b-3), and the computing device (100-3) may be configured in multiple units. In addition, as illustrated in FIG. 42e, even if a user terminal (300-3) according to one embodiment of the present invention is not separately provided, a service according to one embodiment of the present invention can be provided by having at least one of a sleep analysis server (200a-3), a content provision server (200b-3), and a computing device (100-3) perform at least one of various operations of the user terminal (300-3) on behalf of the user terminal (300-3).

FIG. 42F is a conceptual diagram illustrating a system in which various aspects of various electronic devices according to embodiments of the present invention can be implemented. As illustrated in FIG. 42F, the electronic devices illustrated in FIG. 42F can perform at least one of the operations performed by various devices (e.g., computing device (100-3), server (200-3), user terminal (300-3), etc.) according to embodiments of the present invention.

For example, operations performed by various electronic devices according to embodiments of the present invention may include an operation of acquiring environmental sensing information, an operation of acquiring sleep sound information, an operation of performing learning for a sleep analysis model, an operation of inference using a sleep analysis model, a feedback operation for improving inference of a sleep analysis model, and an operation of acquiring sleep state information.

Meanwhile, referring to FIG. 42f, according to embodiments of the present invention, there may be a network that interacts with one or more electronic devices within the range of the detection area (11a-3), and there may be a network that interacts with one or more electronic devices outside the range of the detection area (11a-3). Here, the network that interacts with the electronic devices within the range of the detection area (11a-3) may enable one or more electronic devices located within the range of the detection area (11a-3) to transmit and receive data for a system according to embodiments of the present invention.

Meanwhile, referring to FIG. 42f, there may be one or more electronic devices connected through a network outside the range of the detection area (11a-3), and in this case, the electronic devices may distribute and process data to each other or perform one or more operations according to embodiments of the present invention. Here, the electronic devices connected through a network outside the range of the detection area (11a-3) may include electronic devices implemented in the form of servers.

As illustrated in FIG. 42f, a network interacting with electronic devices within the range of the detection area (11a-3) may be a short-range network or a local network, etc. According to one embodiment of the present invention, a network interacting with electronic devices outside the range of the detection area (11a-3) may be a long-range network or a global network, etc. However, the present invention is not limited thereto, and a network interacting with electronic devices within the range of the detection area (11a-3) may also be configured as a long-range network or a global network. In addition, according to one embodiment of the present invention, a network interacting with electronic devices within the range of the detection area (11a-3) and a network interacting with electronic devices outside the range of the detection area (11a-3) may be configured as one network.

Computing Device (100-3)

FIGS. 43a and 43b are block diagrams illustrating a computing device (100-3) according to one embodiment of the present invention.

Referring to FIG. 43a, the computing device (100-3) may include at least one of a processor (110-3), a memory (120-3), and a communication module (170-3). More specifically, referring to FIG. 43b, the computing device (100-3) of the present invention may include at least one of a processor (110-3), a memory (120-3), an output unit (130-3), an input unit (140-3), an interface unit (150-3), a sensor module (160-3), and a communication module (170-3). The components of the computing device (100-3) of the present invention are not limited to the components illustrated in FIG. 43a. That is, additional components may be included or some of the components illustrated in FIG. 43a may be omitted depending on the implementation aspect of the embodiments of the present invention.

According to one embodiment of the present invention, the computing device (100-3) may be implemented as an IoT electronic device such as a TV, a smart TV, a refrigerator, an oven, a clothing styler, a robot cleaner, a drone, an air conditioner, an air purifier, a PC, a speaker, a home CCTV, a light, a washing machine, and a smart plug.

According to one embodiment of the present invention, the computing device (100-3) may be a terminal or a server, and may include any type of device. The computing device (100-3) may be a digital device, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone, which has a processor and a memory and has a computing ability. The computing device (100-3) may be a web server that processes a service. The types of servers described above are merely examples, and the present invention is not limited thereto.

According to the present invention, the computing device (100-3) can obtain the user's sleep state information based on the environmental sensing information. In addition, the computing device (100-3) according to one embodiment of the present invention can perform an operation of generating content related to the user's sleep state information and/or an operation of providing content related to the user's sleep state information.

According to one embodiment of the present invention, the environmental sensing information utilized by the computing device (100-3) for sleep state analysis may include sound information acquired non-invasively during the user's activity or sleep in a work space. For a specific example, the environmental sensing information may include sound information generated as the user tosses and turns during sleep, sound information related to muscle movement, or sound information related to the user's breathing during sleep. According to an embodiment, the environmental sensing information may include sleep sound information, and the sleep sound information may refer to sound information related to movement patterns and breathing patterns generated during the user's sleep. In addition, for example, the sleep state information may include information related to whether the user is before, during, or after sleep. In addition, the sleep state information may include sleep stage information. In addition, the sleep state information may include sleep event information. The sleep state information will be described in detail later.

Processor (110-3)

According to the present invention, the processor (110-3) may include one or more application processors (APs), one or more communication processors (CPs), or at least one artificial intelligence processor (AI processor). The application processor, the communication processor, or the AI processor may be included in different integrated circuit (IC) packages, or may be included in one IC package.

The application processor may drive an operating system or an application program to control a plurality of hardware or software components connected to the application processor and perform various data processing/operations including multimedia data. As an example, the application processor may be implemented as a SoC (system on chip). The processor (110-3) may further include a GPU (graphic processing unit, not shown).

The communication processor may perform functions of managing data links and converting communication protocols in communications between the computing device (100-3) and other computing devices connected to a network. For example, the communication processor may be implemented as a SoC. The communication processor may perform at least a portion of the multimedia control functions.

Additionally, the communication processor can control data transmission and reception of the communication module (170-3). The communication processor may be implemented to be included as at least a part of the application processor.

The application processor or the communication processor may load and process instructions or data received from at least one of the nonvolatile memory or other components connected thereto, respectively, into the volatile memory. Additionally, the application processor or the communication processor may store data received from at least one of the other components or generated by at least one of the other components, into the nonvolatile memory.

When loaded into the memory (120-3), the computer program may include one or more instructions that cause the processor (110-3) to perform a method/operation according to various embodiments of the present invention. That is, the processor (110-3) may perform the method/operation according to various embodiments of the present invention by executing one or more instructions.

In one embodiment, the computer program may include one or more instructions for performing a method for creating a sleep environment based on sleep state information, the method including the steps of obtaining sleep state information of a user, generating environment creation information based on the sleep state information, and transmitting the environment creation information to an environment creation device.

In this specification, a neural network may be used interchangeably with an artificial neural network, a neural network. In this specification, a neural network may include one or more neural networks, in which case the output of the neural network may be an ensemble of outputs of one or more neural networks.

In this specification, a model may include a neural network. The model may include one or more neural networks, in which case the output of the model may be an ensemble of outputs of one or more neural networks.

The processor (110-3) can read a computer program stored in the memory (120-3) to provide a sleep analysis model according to one embodiment of the present invention. According to one embodiment of the present invention, the processor (110-3) can perform calculations to derive environment composition information based on sleep state information. According to one embodiment of the present invention, the processor (110-3) can perform calculations to train the sleep analysis model. The sleep analysis model will be described in more detail below. Sleep information related to the quality of the user's sleep can be inferred based on the sleep analysis model. Environmental sensing information acquired from the user in real time or periodically is input as an input value to the sleep analysis model to output data related to the user's sleep.

The learning of such a sleep analysis model and the inference based thereon can be performed by the computing device (100-3). That is, it can be designed that both learning and inference are performed by the computing device (100-3). However, in another embodiment, learning may be performed in the computing device (100-3), but inference may be performed in the user terminal (300-3). In addition, learning may be performed in the computing device (100-3), but inference may be performed in the server (200-3). Conversely, learning may be performed in the user terminal (300-3), but inference may be performed in the computing device (100). In addition, learning may be performed in the server (200-3), but inference may be performed in the computing device (100). Alternatively, both learning and inference may be performed in the server (200-3), or both learning and inference may be performed in the user terminal (300-3).

According to one embodiment of the present invention, the processor (110-3) can typically process the overall operation of the computing device (100-3). The processor (110-3) can process signals, data, information, etc. input or output through the components described above, or can provide or process appropriate information or functions to the user terminal by running an application program stored in the memory (120-3).

According to one embodiment of the present invention, the processor (110-3) can obtain the user's sleep state information. According to one embodiment of the present invention, obtaining the sleep state information may be obtaining or loading the sleep state information stored in the memory (120-3). In addition, obtaining the sleep sound information may be receiving or loading data from another computing device, a separate processing module within the same computing device, or another storage medium based on a wired/wireless communication means.

Memory (120-3)

According to one embodiment of the present invention, the memory (120-3) can store a computer program for performing a method for creating a sleep environment according to sleep state information according to one embodiment of the present invention, and the stored computer program can be read and operated by the processor (110-3). In addition, the memory (120-3) can store any form of information generated or determined by the processor (110-3) and any form of information received through the communication module (170-3). In addition, the memory (120-3) can store data related to the user's sleep. For example, the memory (120-3) can temporarily or permanently store input/output data (e.g., environmental sensing information related to the user's sleep environment, sleep state information corresponding to the environmental sensing information, or environmental creating information according to the sleep state information, etc.).

Output section (130-3)

According to one embodiment of the present invention, the output unit (130-3) may include at least one module that performs an output function, such as a display module or/and a speaker, an optical output module, etc. The output unit (130-3) may perform an operation such as visually displaying various data including multimedia data, text data, voice data, etc. to a user or outputting them as sound.

Input section (140-3)

According to one embodiment of the present invention, the input unit (140-3) may include a camera module and/or a microphone. The camera module is a device capable of capturing images and moving images, and may include one or more image sensors, an image signal processor (ISP), or a flash LED. The microphone may receive a voice signal and convert it into an electrical signal.

According to one embodiment of the present invention, the input unit (140-3) may include a touch panel, a digital pen sensor, a key, or an ultrasonic input device. For example, the input unit (140-3) may be an interface unit (150-3).

The touch panel can recognize touch input in at least one of electrostatic, pressure-sensitive, infrared, or ultrasonic methods. In addition, the touch panel may further include a controller (not shown). In the case of electrostatic, not only direct touch but also proximity recognition is possible. The touch panel may further include a tactile layer. In this case, the touch panel can provide a tactile response to the user. The digital pen sensor can be implemented using the same or similar method for receiving the user's touch input or a separate recognition layer. The key can be a keypad or a touch key. The ultrasonic input device is a device that can detect micro sound waves in a terminal and confirm data through a pen that generates an ultrasonic signal, and wireless recognition is possible. The computing device (100-3) can also receive user input from an external device (e.g., a network, a computer, or a server) connected thereto using the communication module (170-3).

Interface section (150-3)

According to one embodiment of the present invention, the interface unit (150-3) can serve as a passageway with various types of external devices connected to the computing device (100).

The interface unit (150-3) according to one embodiment of the present invention can transmit a command or data input by a user through the input unit (140-3) or the output unit (130-3) to the processor (110-3), the memory (120-3), the communication module (170-3), etc. through a bus (not shown). For example, the interface unit (150-3) can provide data regarding a user's touch input input through a touch panel to the processor (110-3). For example, the interface unit (150-3) can output a command or data received from the processor (110-3), the memory (120-3), the communication module (170-3), etc. through the bus through the output unit (130-3). For example, the interface unit (150-3) can output voice data processed by the processor (110-3) to the user through a speaker. Meanwhile, the interface unit (150-3) according to one embodiment of the present invention can be driven based on user input through the input unit (140-3).

Sensor Module (160-3)

According to one embodiment of the present invention, the sensor module (160-3) may include at least one of a gesture sensor, a gyro sensor, a pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, an RGB (red, green, blue) sensor, a biometric sensor, a temperature/humidity sensor, an illuminance sensor or an UV (ultra violet) sensor, and a microphone sensor. According to one embodiment of the present invention, the sensor module (160-3) may measure a physical quantity or detect an operating state of the computing device (100-3) and convert the measured or detected information into an electrical signal. Additionally or alternatively, the sensor module (160-3) may include an olfactory sensor (E-nose sensor), an electromyography sensor (EMG sensor), an electroencephalogram sensor (EEG sensor, not shown), an electrocardiogram sensor (ECG sensor), a photoplethysmography sensor (PPG sensor), a heart rate monitor sensor (HRM sensor), a perspiration sensor, or a fingerprint sensor. The sensor module (160-3) may further include a control circuit for controlling at least one sensor included therein.

Communication Module (170-3)

According to one embodiment of the present invention, the communication module (170-3) may include a wireless communication module or an RF module. The wireless communication module may include, for example, Wi-Fi, BT, GPS, or NFC.

According to one embodiment of the present invention, the communication module (170-3) may provide a wireless communication function using a radio frequency. Additionally or alternatively, the wireless communication module may include a network interface or modem, etc. for connecting the computing device (100-3) to a network (e.g., Internet, LAN, WAN, telecommunication network, cellular network, satellite network, POTS or 5G network, etc.).

Server (200-3)

Figure 43c is a block diagram illustrating a server (200-3) according to one embodiment of the present invention.

According to the present invention, the server (200-3) may include at least one of a processor (210-3), a memory (220-3), and a communication module (270-3). Meanwhile, the components of the server (200-3) of the present invention are not limited to the components illustrated in FIG. 43c.

Meanwhile, according to embodiments of the present invention, the server (200-3) may be configured as a single server or may be configured as a plurality of servers. In addition, according to embodiments of the present invention, as shown in FIGS. 42d and 42e, the server (200-3) may be implemented as a plurality of servers including a sleep analysis server (200a-3) and a content provision server (200b).

Meanwhile, according to one embodiment of the present invention, the content providing server (200b-3) may provide content related to sleep state information by utilizing artificial intelligence based on natural language processing. According to one embodiment of the present invention, the content providing server (200b-3) may generate content related to sleep state information by utilizing generative artificial intelligence and provide the same. Alternatively, according to one embodiment of the present invention, the content providing server (200b-3) may provide content related to sleep state information selected based on a lookup table. Alternatively, according to one embodiment of the present invention, the content providing server (200b-3) may provide content related to sleep state information generated based on a lookup table.

According to one embodiment of the present invention, the server (200-3) may be a server that stores information on a plurality of learning data for learning a neural network. The plurality of learning data may include, for example, health checkup information or sleep checkup information. For example, the server (200-3) may be at least one of a hospital server and an information server, and may be a server that stores information on a plurality of polysomnography records, electronic health records, electronic medical records, and the like. For example, the polysomnography records may include information on breathing and movement during sleep of a sleep checkup subject and information on sleep diagnosis results (for example, sleep stages, etc.) corresponding to the information. The information stored in the server (200-3) may be utilized as learning data, verification data, and test data for learning a neural network in the present invention.

At least one server (200-3) described above can mutually transmit and receive data based on a data transmission and reception tool (Tool) based on API or the like with at least one of a computing device (100-3) and a user terminal (300-3), and thus, a service according to embodiments of the present invention can be provided.

Processor (210-3)

According to one embodiment of the present invention, the processor (210-3) controls the server (200-3) as a whole. According to one embodiment of the present invention, the processor (210-3) may be a general-purpose processor (e.g., CPU), but may also be an AI-only processor (e.g., GPU, TPU) for learning and/or executing an artificial intelligence model.

According to one embodiment of the present invention, the processor (210-3) may include an AI processor (215-3). According to one embodiment of the present invention, the AI processor (215-3) may learn a neural network using a program stored in the memory (220-3). In particular, the AI processor (215-3) may learn a neural network for recognizing data related to the operation of the user terminal (300-3). Here, the neural network may be designed to simulate the human brain structure (e.g., the neuron structure of the human neural network) on a computer. The neural network may include an input layer, an output layer, and at least one hidden layer. Each layer may include at least one neuron having a weight, and the neural network may include a synapse connecting neurons to neurons. In a neural network, each neuron can output an input signal received through a synapse as a function value of an activation function for weights and/or biases.

Memory (220-3)

According to the present invention, the memory (220-3) can store various programs and data required for the operation of the user terminal (300-3) and/or the server (200-3). The memory (220-3) is accessed by the processor (210-3), and data reading/recording/modifying/deleting/updating, etc. can be performed by the processor (210-3). In addition, the memory (220-3) can store a neural network model (e.g., a deep learning model) generated through a learning algorithm for data classification/recognition. Furthermore, the memory (220-3) can store not only the learning model (221-3), but also input data, learning data, learning history, etc.

According to one embodiment of the present invention, the AI processor (215-3) may include a data learning unit (215a-3) that learns a neural network for data classification/recognition. The data learning unit (215a-3) may learn criteria regarding which learning data to use for determining data classification/recognition and how to classify and recognize data using the learning data. The data learning unit (215a-3) may acquire learning data to be used for learning and learn a deep learning model by applying the acquired learning data to a deep learning model.

According to one embodiment of the present invention, the data learning unit (215a-3) may be manufactured in the form of at least one hardware chip and mounted on the server (200-3). For example, the data learning unit (215a-3) may be manufactured in the form of a dedicated hardware chip for artificial intelligence and may be manufactured as a part of a general-purpose processor (CPU) or a graphics processor (GPU) and mounted on the server (200-3). In addition, the data learning unit (215a-3) may be implemented as a software module. When implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer readable medium that can be read by a computer. In this case, at least one software module may be provided to an operating system (OS) or provided by an application.

According to one embodiment of the present invention, the data learning unit (215a-3) can learn to have a judgment criterion on how to classify/recognize a given data by using the acquired learning data. At this time, the learning method by the model learning unit can be classified into supervised learning, unsupervised learning, and reinforcement learning. Here, supervised learning refers to a method of learning an artificial neural network in a state where a label for learning data is given, and the label may mean the correct answer (or result value) that the artificial neural network should infer when learning data is input to the artificial neural network. Unsupervised learning may refer to a method of learning an artificial neural network in a state where a label for learning data is not given. Reinforcement learning may refer to a method of learning an agent defined in a specific environment to select an action or action sequence that maximizes the cumulative reward in each state. In addition, the model learning unit may learn the neural network model using a learning algorithm including error backpropagation or gradient descent. When a neural network model is trained, the trained neural network model can be called a training model (221). The training model (221-3) is stored in memory (220-3) and can be used to infer results for new input data that is not training data.

According to one embodiment of the present invention, the AI processor (215-3) may further include a data preprocessing unit (215b-3) and/or a data selection unit (215c-3) to improve analysis results using the learning model (221-3) or to save resources or time required for generating the learning model (221-3).

According to one embodiment of the present invention, the data preprocessing unit (215b-3) can preprocess the acquired data so that the acquired data can be used for learning/inference for situation judgment. For example, the data preprocessing unit (215b-3) can extract feature information as preprocessing for input data received through the communication module (270-3), and the feature information can be extracted in a format such as a feature vector, a feature point, or a feature map.

According to one embodiment of the present invention, the data selection unit (215c-3) can select data required for learning from among the learning data or the learning data preprocessed in the preprocessing unit. The selected learning data can be provided to the model learning unit. For example, the data selection unit (215c-3) can detect a specific area among the data acquired through the input unit (140) of the computing device (100-3) and/or the input unit (340-3) of the user terminal (300-3), thereby selecting only data for an object included in the specific area as learning data. In addition, the data selection unit (215c-3) can also select data required for inference from among the input data acquired through the input device or the input data preprocessed in the preprocessing unit.

In addition, according to one embodiment of the present invention, the AI processor (215-3) may further include a model evaluation unit (215d-3) to improve the analysis result of the neural network model. The model evaluation unit (215d-3) may input evaluation data into the neural network model, and if the analysis result output from the evaluation data does not satisfy a predetermined criterion, may cause the model learning unit to learn again. In this case, the evaluation data may be preset data for evaluating the learning model (221-3). For example, the model evaluation unit (215d-3) may evaluate that the predetermined criterion is not satisfied if the number or ratio of evaluation data for which the analysis result is inaccurate among the analysis results of the learned neural network model for the evaluation data exceeds a preset threshold. According to one embodiment of the present invention, the model evaluation unit (215d-3) may also improve the model's inference ability through feedback.

Communication Module (270-3)

According to one embodiment of the present invention, the communication module (270-3) may include a wireless communication module or an RF module. The wireless communication module may include, for example, Wi-Fi, BT, GPS, or NFC.

User terminal (300-3)

FIG. 2d is a block diagram illustrating a user terminal (300-3) according to one embodiment of the present invention.

According to one embodiment of the present invention, the user terminal (300-3) may be a terminal that can receive information related to the user's sleep through information exchange with the computing device (100-3) and/or the server (200-3), and may refer to a terminal possessed by the user. For example, the user terminal (300-3) may be a terminal related to a user who wants to improve his/her health through information related to his/her sleep habits. The user may obtain monitoring information related to his/her sleep through the user terminal (300-3). The monitoring information related to sleep may include, for example, information related to the time when the user fell asleep, the time he/she slept, the time when he/she woke up from sleep, sleep event information, or sleep stage information related to changes in sleep stages during sleep. For a specific example, the sleep stage information may refer to information on changes in the user's sleep, such as light sleep, normal sleep, deep sleep, or REM sleep, at each time point during the user's 8 hours of sleep last night. The specific description of the sleep stage information described above is merely an example, and the present invention is not limited thereto.

A user terminal (300-3) according to one embodiment of the present invention may include at least one of a processor (310-3), a memory (320-3), an output unit (330-3), an input unit (340-3), an interface unit (350-3), a sensor module (360-3), a communication module (370-3), and a power supply unit (380-3). Meanwhile, the components of the user terminal (300-3) of the present invention are not limited to the components illustrated in FIG. 43d. That is, additional components may be included or some of the components illustrated in FIG. 43d may be omitted depending on the implementation aspect of the embodiments of the present invention.

Processor (310-3)

According to the present invention, the processor (310-3) controls the overall operation of the user terminal (300-3) in addition to the operations related to the application program. The processor (310-3) processes signals, data, information, etc. input or output through the components described above, or operates an application program stored in the memory (320-3), thereby providing or processing appropriate information or functions to the user.

According to the present invention, the processor (310-3) may control at least some of the components examined together with FIG. 43f in order to drive an application program stored in the memory (320-3). Furthermore, the processor (310-3) may operate at least two or more of the components included in the user terminal (300-3) in combination with each other in order to drive the application program. According to one embodiment of the present invention, the processor (310-3) may be a general-purpose processor (e.g., CPU), but may also be an AI-only processor (e.g., GPU, TPU) for learning and/or executing an artificial intelligence model.

Memory (320-3)

According to the present invention, the memory (320-3) stores data supporting various functions of the user terminal (300-3). The memory (320-3) can store a plurality of application programs (or applications) running on the user terminal (300-3), data, commands, or instructions for the operation of the user terminal (300-3). At least some of these application programs can be downloaded from an external server (e.g., server (200-3) etc.) via wireless communication. In addition, at least some of these application programs can exist on the user terminal (300-3) from the time of shipment for the basic functions of the user terminal (300-3) (e.g., call reception, call transmission functions, message reception, call transmission functions). Meanwhile, the application program can be stored in the memory (320-3), installed on the user terminal (300-3), and driven by the processor (310-3) to perform the operation (or function) of the user terminal. Additionally, the memory (320-3) according to one embodiment of the present invention can store instructions for the operation of the processor (310-3).

Output section (330-3)

According to the present invention, the output unit (330-3) is for generating output related to vision, hearing, or tactile sensations, and may include at least one of a display unit (331-3), an audio output unit (332-3), a haptic module (333-3), and an optical output unit (334-3).

According to the present invention, the display unit (331-3) can implement a touch screen by forming a mutual layer structure with the touch sensor or forming an integral structure. This touch screen can function as a user input unit (343-3) that provides an input interface between the user terminal (300-3) and the user (U), and at the same time, can provide an output interface between the user terminal (300-3) and the user (U).

Input section (340-3)

According to the present invention, the input unit (340-3) may include a camera (341-3) or an image input unit for inputting an image signal, a microphone (342-3) or an audio input unit for inputting an audio signal, and a user input unit (343-3, for example, a touch key, a mechanical key, etc.) for receiving information from a user. Voice data or image data collected by the input unit (340-3) may be analyzed and processed into a user's control command.

Interface section (350-3)

According to the present invention, the interface unit (350-3) serves as a passageway for various types of external devices connected to the user terminal (300-3). The interface unit (350-3) may include at least one of a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I/O (Input/Output) port, a video I/O (Input/Output) port, and an earphone port. In the user terminal (300-3), appropriate control related to the connected external device (for example, a computing device (100-3)) may be performed in response to the connection of the external device to the interface unit (350-3).

In addition, the interface unit (350-3) according to one embodiment of the present invention can transmit a command or data input by a user through the input unit (340-3) or the output unit (330-3) to the processor (310-3), the memory (320-3), the communication module (370-3), etc. through a bus (not shown). For example, the interface unit (350-3) can provide data regarding a user's touch input input through the touch panel to the processor (310-3). For example, the interface unit (350-3) can output a command or data received from the processor (310-3), the memory (320-3), the communication module (370-3), etc. through the bus through the output unit (330-3). For example, the interface unit (350-3) can output voice data processed by the processor (310-3) to the user through a speaker. Meanwhile, the interface unit (350-3) according to one embodiment of the present invention can be driven based on user input through the input unit (340-3).

Sensor Module (360-3)

According to the present invention, the sensor module (360-3) may include one or more sensors for sensing at least one of information within the user terminal, information about the surrounding environment surrounding the user terminal, and user information. For example, the sensor module (360-3) may include at least one of an acoustic sensor, a proximity sensor (361-3), an illumination sensor (362-3), a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor), a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor (e.g., a camera, a LIDAR sensor, etc.), a battery gauge, an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor, etc.), and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric recognition sensor, etc.). Meanwhile, the user terminal disclosed in this specification can utilize information sensed by at least two or more of these sensors in combination.

Communication Module (370-3)

According to the present invention, the communication module (370-3) may include one or more modules that enable wireless communication between the user terminal (300-3) and the wireless communication system, between the user terminal (300-3) and the computing device (100-3), or between the user terminal (300-3) and the server (200-3). In addition, the communication module (370-3) may include one or more modules that connect the user terminal (300-3) to one or more networks.

These communication modules (370-3) may include at least one of a broadcast reception module (371-3), a mobile communication module (372-3), a wireless Internet module (373-3), a short-range communication module (374-3), and a location information module (375-3).

Power supply unit (380-3)

According to the present invention, the power supply unit (380-3) receives external power and internal power under the control of the processor (310-3) and supplies power to each component included in the user terminal (300-3). Specifically, the power supply unit (380-3) includes a battery, and the battery may be a built-in battery or a replaceable battery.

Artificial intelligence model according to the present invention

According to one embodiment of the present invention, artificial intelligence (AI) may generally mean the development of a computer system or machine that can perform tasks that require human intelligence. Such tasks include reasoning, learning, problem solving, natural language understanding, pattern recognition, and decision making. Specifically, AI may be used in fields such as machine learning (ML), natural language processing (NLP), computer vision, expert systems, robotics, and deep learning (DL).

According to one embodiment of the present invention, machine learning may be a subset of AI in which a machine learns from data without being explicitly programmed. The system can improve its performance through machine learning as it is exposed to more data. Machine learning methods include supervised learning, unsupervised learning, and reinforcement learning. Natural language processing is a field of AI that allows computers to understand, interpret, and generate human language. Applications such as chatbots, language translation, and voice assistance can be provided through artificial intelligence models based on natural language processing. Computer vision includes learning models to interpret and understand visual information, such as recognizing objects in images or videos. Computer vision can be used in fields such as face recognition, self-driving cars, and medical image analysis. An expert system is an AI system that uses knowledge and inference rules to solve complex problems in a specific area, such as medical diagnosis or financial analysis. Deep learning is a subset of machine learning based on artificial neural networks, and artificial intelligence models can be designed to model or process complex patterns in data based on deep learning. Deep learning can be used to train artificial intelligence models for tasks such as image processing and natural language processing.

According to one embodiment of the present invention, the core of an artificial intelligence model is the way data is represented. In order for AI to process real-world information such as images, texts or numerical data, this information must be converted into a format that the model can understand, such as a vector or matrix, for example, a collection of vectors. Here, a vector is a simple array of numbers representing data points in a multidimensional space. For example, in NLP, when expressing a word, a 300-dimensional vector can be used, where each number in the vector encodes a specific aspect of the meaning of the word. The specific numbers are merely examples and are not limited thereto.

According to one embodiment of the present invention, an artificial intelligence model, particularly a neural network, is composed of a layer of nodes (or neurons), each of which applies some transformation to the data. The core of this transformation includes weights and biases. Here, a node is a basic unit that processes input data. Each node of the network takes a vector of input data, applies some mathematical function to it, and then passes the result to the next layer of the network. Weights are values that are multiplied by the input data (vector) passing between nodes, and are parameters of the model that can be updated during learning. Biases are additional parameters that are added to the weighted sum of the inputs before the output passes through the activation function. This allows the model to move the decision boundary, which can be important in data classification tasks.

According to one embodiment of the present invention, after applying weights and biases to input data, the output of each node is passed through an activation function. The activation function determines whether a node should be activated and introduces non-linearity into the model to enable solving complex tasks. The activation function includes, for example, a sigmoid that generally converts output to a value between 0 and 1, a rectified linear unit (ReLU) that outputs 0 for negative inputs and the input value itself for positive inputs, and a softmax that converts output to a probability distribution and is used particularly for classification tasks based on probabilities across multiple classes.

According to one embodiment of the present invention, the loss function of the artificial intelligence model is a function for measuring how far the prediction of the model is from the actual target value. The learning goal of the artificial intelligence model is to minimize this loss function. By adjusting the weights and biases of the model, the loss function can be reduced, thereby improving the accuracy of the model. The loss function may include, for example, the mean squared error (MSE) that measures the square difference between the predicted value and the actual value, and the cross-entropy loss that is used to measure the difference between the predicted probability distribution and the actual distribution in a classification task. The MSE can be commonly used in regression tasks, and the cross-entropy loss can be used in the field of image processing, but these are simple examples and are not limited thereto.

According to one embodiment of the present invention, in order to minimize the loss function, the AI model uses an optimization process, for example, gradient descent or backpropagation can be used. Gradient descent is a method of repeatedly adjusting the parameters (weights and biases) of the model in a direction that reduces the loss function, and the gradient can tell the model how much and in which direction each weight should be adjusted to reduce the loss. Backpropagation is a method of propagating the error of the output in the neural network backward through the layers and updating the weights of the nodes. That is, through backpropagation, the weights can be adjusted layer by layer, starting from the output and working toward the input.

According to one embodiment of the present invention, an artificial neural network (artificial intelligence model) may include an input layer, hidden layers, and an output layer. The input layer is a layer in which data is input to the model, and each node of this layer represents a function of the input data. The hidden layer is a layer between the input layer and the output layer, and the hidden layer applies a transformation to the data so that the model can learn complex patterns. The output layer is a layer that provides the final output of the model, and for example, in a classification task, the output may be a probability vector corresponding to various classes.

According to one embodiment of the present invention, the types of artificial intelligence models include feedforward neural networks, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer-based neural networks, in which information flows in one direction from input to output through neuron layers. CNNs are designed for image and video processing, and use convolutional layers to scan images into small patches and learn spatial hierarchies of features (edges, textures, etc.). CNNs can be used in fields such as image classification, object detection, and segmentation. RNNs are artificial neural networks used for sequential data such as time series or text. Since RNNs process one element of a sequence at a time and the output depends on the current input and previous computations, they can be modified to long-term dependencies such as LSTMs (Long Short-Term Memory) and Gated Recurrent Units (GRUs). Transformer uses an attention mechanism, so the model can focus on relevant parts of the input at different processing stages, is powerful in handling long-term dependencies, and is widely used in natural language processing (e.g., BERT, GPT).

According to one embodiment of the present invention, an artificial intelligence model is trained using a large data set, which is generally divided into a training set, a validation set, and a test set. The training set is the data used to train the model and update the weights. The validation set is a subset of data that is not used during training but is used to tune the hyperparameters and prevent overfitting. The test set is a completely separate data set used to evaluate the model performance after training. The training process includes multiple epochs, where the model checks the same data multiple times and updates the weights at each step. The model performance can be monitored using metrics such as accuracy, precision, recall, or F1 score, depending on the task.

According to one embodiment of the present invention, if a model performs well on training data but performs poorly on new data that has not yet been seen, this is called overfitting, which can occur because the model learns not only the patterns of the data but also noise. Therefore, techniques such as regularization, which applies a penalty to large weights, dropout, which randomly deletes nodes during training, and early stopping, which stops training before overfitting occurs, can help prevent overfitting by simplifying the model.

The sleep analysis model and content provision model according to embodiments of the present invention may be designed based on the artificial intelligence model described above. The sleep analysis model and content provision model will be described in detail below.

How to analyze sleep

Data for sleep analysis

Figure 44 is a schematic diagram of a data set for sleep analysis according to one embodiment of the present invention.

Data according to one embodiment of the present invention may be raw data obtained through a sensor module (160-3) of a computing device (100-3) and/or a sensor module (360-3) of a user terminal (300-3). Here, the raw data may be information in the time domain having amplitude, phase, and frequency.

In addition, data according to one embodiment of the present invention may be obtained by converting acquired raw data into information including changes in frequency components of the raw data along the time axis.

Alternatively, data according to one embodiment of the present invention may be data converted from raw data into information in the frequency domain rather than information in the time domain. Here, Fourier transform or wavelet transform may be performed to convert the data into information in the frequency domain.

Additionally, information converted into the frequency domain according to one embodiment of the present invention may be information having amplitude and frequency.

In addition, data according to one embodiment of the present invention may correspond to a spectrogram among the acoustic information converted into information in the frequency domain.

Alternatively, data according to one embodiment of the present invention may be a Mel spectrogram that applies a Mel scale to a spectrogram. Specifically, a Mel spectrogram may be obtained through a Mel filter bank for a spectrogram. In general, the human cochlea may have different vibrating parts depending on the frequency of the voice data. In addition, the human cochlea has a characteristic of detecting frequency changes well in a low-frequency band and not detecting frequency changes well in a high-frequency band. Accordingly, a Mel spectrogram may be obtained from a spectrogram by utilizing a Mel filter bank so as to have a recognition ability similar to the characteristics of a human cochlea for voice data. That is, the Mel filter bank may apply a small filter bank in a low-frequency band and apply a wider filter bank as it goes to a high-frequency band. In other words, the processor may obtain a Mel spectrogram by applying a Mel filter bank to a spectrogram to recognize voice data similar to the characteristics of a human cochlea. A Mel spectrogram may include frequency components reflecting human hearing characteristics. That is, a spectrogram generated in response to sleep sound information in the present invention and subject to analysis using a neural network may include the Mel spectrogram described above.

Here, information converted into information in the frequency domain, such as a spectrogram or a mel spectrogram according to one embodiment of the present invention, may be information converted into a domain having amplitude and frequency, including changes in frequency components of acoustic information along the time axis.

In addition, data according to embodiments of the present invention can be input to an artificial intelligence model based on image processing as a visualization of the above-described information. For example, information converted into information including changes in the frequency components of raw data along the time axis can be visualized and used as an input to an artificial intelligence model. Alternatively, information converted into the frequency domain can be visualized and used as an input to an artificial intelligence model. An artificial intelligence model into which the above information is input can be an artificial intelligence model based on image processing.

How to collect data sets

Figure 44 is a schematic diagram of a data set for sleep analysis according to one embodiment of the present invention.

According to an embodiment of the present invention, sleep sound information among sleep state information may be collected through polysomnography (PSG) in a hospital environment, or may be collected by a user in a home environment, etc., through a microphone built into a user terminal such as a wearable device or smartphone.

A data set according to an embodiment of the present invention may be collected and configured through an electroencephalography (Video-EEG, Video-electroencephalography) during a polysomnography (PSG) or a microphone during a polysomnography (PSG). Alternatively, the data set may be configured by collecting an acoustic signal generated during sleep through a microphone built into an electronic device such as a user terminal.

Hereinafter, a method for a processor to derive a final sleep analysis result by using bio-signal, movement information (ACTIGRAPHY), and sleep sound information (SOUND) will be described. In the following specification, even if it is expressed only as a 'processor' for convenience, it may refer to a processor (110-3) of a computing device (100-3), a processor (210-3) of a server (200-3), and/or a processor (310-3) of a user terminal (300-3).

First, the processor can derive the final sleep analysis result using the weights. Specifically, the processor can derive the secondary sleep analysis result by applying the same weights to the first sleep analysis result and the sleep analysis result using sleep sound information. Alternatively, the processor can derive the secondary sleep analysis result by applying different weights to the first sleep analysis result and the sleep analysis result using sleep sound information. For example, a 30% weight can be applied to the contact-type first sleep analysis based on HRV and Actigraphy, and a 70% weight can be applied to the AI analysis using sleep sound to derive the final secondary sleep analysis result.

In another embodiment, the processor may determine that the user has entered a given sleep stage only when the sleep stages in the first sleep analysis result and the second sleep analysis result completely match, thereby deriving the final sleep analysis result.

In another embodiment, the processor can utilize a method of training an AI sleep analysis model that uses bio-signal, movement information (ACTIGRAPHY), and sleep sound information (SOUND) as inputs. The training method of the AI sleep analysis model will be described in more detail below, but briefly, by inputting all three pieces of information into the deep learning input layer, an AI sleep analysis model that performs sleep analysis by three factors can be created.

In another embodiment, the processor first performs secondary sleep analysis using sleep sound information (SOUND) using the AI sleep analysis model described below, and then additionally extracts AI confidence levels for sleep stages for each time zone. If the extracted confidence level is lower than a preset value, the sleep stage of the corresponding time zone uses the sleep stage results derived from the primary sleep analysis. In other words, by additionally employing the primary sleep analysis results based on the secondary sleep analysis results, more reliable sleep analysis results can be derived.

In another embodiment, the processor first obtains statistics on the portions that are inconsistent with the actual analysis results in the AI sleep analysis model described below. The statistics may be input by the user, but may also be obtained by itself from multiple user data. The processor may additionally employ the primary sleep analysis results in the portions that are inconsistent with the actual analysis results in the obtained statistics, focusing on the secondary sleep analysis results (analysis based on SOUND).

In another embodiment, the processor may utilize a method of learning an AI sleep analysis model based on the primary sleep analysis results obtained by bio-signal, movement information (ACTIGRAPHY), and sleep sound information (SOUND). The learning method of the AI sleep analysis model will be described in more detail below, but briefly, by inputting two pieces of information (primary sleep analysis results and sleep sound information) into the deep learning input layer, an AI sleep analysis model that performs sleep analysis by two factors can be created.

According to the present invention, sleep stages can be divided into NREM (non-REM) sleep and REM (Rapid eye movement) sleep, and NREM sleep can be further divided into plural stages (e.g., 2 stages of Light and Deep, 4 stages of N1 to N4). The setting of sleep stages can be defined based on generally used sleep stages, but can also be arbitrarily set in various ways depending on the designer. Through sleep stage analysis, not only sleep quality but also sleep diseases (e.g., sleep apnea) and their fundamental causes (e.g., snoring) can be predicted.

Hereinafter, a method for learning and predicting multiple sleep state information according to an embodiment of the present invention will be described by way of example. However, the specific description related to sleep states described below is merely an example, and the present invention is not limited thereto, and it should be understood that learning may also be performed on at least one or more of other sleep state information (e.g., movement information, snoring information, sleep disorder information, etc.) not mentioned and/or differences in sleep state information due to environmental differences (e.g., race, etc.).

According to the present invention, in order to learn or predict sleep stage information, acoustic information acquired over a long time interval may be required. In addition, in order to learn or predict sleep state information (e.g., snoring or apnea information, etc.) other than sleep stage information, acoustic information acquired over a relatively short time interval (e.g., 1 minute) before and after the time when the corresponding sleep state occurs may be required.

According to an embodiment of the present invention, acquired acoustic information is converted into information including changes in frequency components along the time axis, information in the frequency domain, or a spectrogram, and processed as inputs of the same artificial intelligence model, thereby learning a plurality of pieces of sleep state information. Alternatively, information including changes in frequency components along the time axis of the acquired acoustic information may be visualized and used as inputs of an image processing-based artificial intelligence model, thereby learning a plurality of pieces of sleep state information. For example, information of a spectrogram according to an embodiment of the present invention may be used as inputs to the same feature extraction model, and the output information may be input into different feature classification models to perform learning. Through this method, an artificial intelligence model according to embodiments of the present invention may be learned to predict various pieces of sleep state information (for example, information on a user's sleep stage or information on sleep events such as apnea information and snoring information) based on a single piece of acoustic information.

In this way, various sleep state information can be complementarily learned based on a single acoustic information. For example, a deep learning model that only learns sleep stages may have a problem of incorrectly predicting a wake state if a state with loud noises such as apnea or snoring is recognized. On the other hand, in the case of a deep learning model designed to learn multiple sleep state information, the above problem can be prevented as a result of complementarily learning other sleep state information such as apnea or snoring in addition to sleep stages.

The specific descriptions regarding the time intervals described above and the specific descriptions regarding sleep state information are merely examples for explaining the present invention, and the present invention is not limited thereto.

Environmental sensing information

In embodiments of the present invention, environmental sensing information may be acquired through the sensor module (160-3) of the computing device (100-3) and/or the sensor module (360-3) of the user terminal (300-3). The environmental sensing information may refer to environmental sensing information related to sleep acquired in a space where the user is located. Here, the environmental sensing information (or environmental sensing information) may be sensing information acquired in a space where the user is located using a non-contact method. The environmental sensing information may be information related to the user's sleep acquired from a smart watch, smart home appliance, etc.

For example, the environmental sensing information may be sound information acquired in a bedroom where a user is sleeping. According to an embodiment, the environmental sensing information acquired through the computing device (100-3) and/or the user terminal (300-3) may be information that serves as a basis for acquiring the user's sleep state information in the present invention. For a specific example, the environmental sensing information may include sleep sound information due to the user's breathing, heart rate, movement, etc. In addition, the environmental sensing information may include noise information regarding the user's sleep environment. For example, it may include noise information commonly occurring in daily life (sound information related to cleaning, sound information related to cooking, sound information related to watching TV, cat sounds, dog sounds, bird sounds, car sounds, wind sounds, rain sounds, etc.) or other noise information. Meanwhile, the environmental sensing information may include at least one of biometric information (electrocardiogram, brain waves, pulse information, information regarding muscle movement, etc.).

According to one embodiment of the present invention, sleep state information regarding whether the user is before sleep, during sleep, or after sleep can be obtained through environmental sensing information obtained in relation to the user's activity. As another specific example, environmental sensing information regarding the user's activity can include the user's heart rate, the user's breathing, and the user's sleep environment's illumination and noise information.

In addition, the environmental sensing information may be at least one of noise information commonly occurring in daily life (sound information related to cleaning, sound information related to cooking, sound information related to watching TV, cat sounds, dog sounds, bird sounds, car sounds, wind sounds, rain sounds, etc.) or other biometric information (electrocardiogram, brain waves, pulse information, information related to muscle movement, etc.).

Preprocessing and/or conversion process of environmental sensing information

FIG. 45a is a diagram for explaining noise reduction according to one embodiment of the present invention. FIG. 45b is a diagram for explaining a process of preprocessing and converting data according to one embodiment of the present invention. FIG. 45c is a diagram for explaining a process of converting data according to one embodiment of the present invention.

Preprocessing method of environmental sensing information

As illustrated in FIG. 45a, sound information extracted from a user, or sleep sound information (raw data) extracted therefrom, may undergo a noise reduction preprocessing process. When the noise reduction preprocessing process is performed, noise (e.g., white noise) included in the raw data may be removed. The noise reduction process may be performed using an algorithm such as spectral gating or spectral subtraction to remove background noise. Furthermore, in the present invention, the noise removal process may be performed using a deep learning-based noise reduction algorithm. The deep learning-based noise reduction algorithm may use a noise reduction algorithm specialized for the user's breathing or respiration sounds, in other words, a noise reduction algorithm learned through the user's breathing or respiration sounds. In addition, in order to perform noise reduction preprocessing, a deep learning-based noise reduction method performed on raw sound information in the time domain, not the frequency domain, may be used. For deep learning-based noise reduction, information such as sleep sound information, which is necessary for inputting a sleep analysis model, can be maintained, and other sounds can be attenuated.

According to one embodiment of the present invention, noise reduction may be performed not only on acoustic information obtained through PSG test results, but also on acoustic information obtained through a microphone built into a user terminal such as a smartphone. At least one of the above-described preprocessing processes may be performed during the learning process of sleep state information or during the inference process.

As illustrated in FIG. 45b, after performing noise reduction preprocessing on environmental sensing information including sleep sound information, the preprocessed information is converted into information including changes in frequency components along the time axis (e.g., spectrogram, etc.), and can be processed as input to an artificial intelligence model according to embodiments of the present invention. Meanwhile, as illustrated in FIG. 4c, environmental sensing information including sleep sound information may be converted directly into information including changes along the time axis (e.g., spectrogram, etc.), and can be processed as input to an artificial intelligence model according to embodiments of the present invention, without performing noise reduction preprocessing.

According to embodiments of the present invention, environmental sensing information including sleep sound information, etc. is converted into information in the frequency domain or a spectrogram, and an inference model is generated based on the converted information in the frequency domain or a spectrogram. Here, the raw data illustrated in FIGS. 4b and 4c may be environmental sensing information that has not undergone preprocessing or conversion, and the environmental sensing information may include acoustic information (audio data). In addition, the environmental sensing information may include any other form of data that may be acquired during sleep (e.g., electroencephalogram (EEG) data, electrooculogram (EOG) data, electromyogram (EMG) data, electrocardiogram (ECG) data, image data, etc.). Hereinafter, for convenience, the raw data and/or environmental sensing information are discussed assuming that they are audio data, but it should be clearly understood by those skilled in the art that the form of the data is not limited thereto.

According to one embodiment of the present invention, in sleep analysis using environmental sensing information, the protection of the user's privacy cannot be overlooked. The present invention performs a process of preprocessing and/or converting the environmental sensing information in order to protect the user's privacy. At this time, a method of converting the sleep sound information included in the environmental sensing information into information (for example, information in the form of a spectrogram) including changes in the frequency components along the time axis based only on the amplitude excluding the phase can be used. In another embodiment, it is also possible to convert the sleep sound information included in the environmental sensing information into information (for example, information in the form of a spectrogram) including changes in the frequency components along the time axis using both the phase and the amplitude.

Through these methods, not only privacy can be protected, but also the processing speed can be improved by reducing the data capacity. One embodiment of the present invention can train and/or generate a sleep analysis model using a set of information (e.g., information in the form of a spectrogram) that includes changes in frequency components along the time axis, which are converted based on sleep sound information. If sleep sound information (e.g., raw audio data) that has not been preprocessed and/or converted is used as is for training an artificial intelligence model, the amount of information is very large, so the amount of computation and computation time increase significantly, and since unwanted signals are included, there is a concern that the computational precision may be reduced, and if all of the user's audio signals are transmitted to the server, there is a concern that privacy may be violated. Therefore, one embodiment of the present invention trains and/or generates a sleep analysis model using a dataset that has been preprocessed and/or converted on sleep sound information in the above-described ways, so the amount of computation and computation time can be reduced, and the privacy of an individual can be protected.

For example, among the sound information acquired through sensor modules, etc., the sleep sound information (e.g., the user's breathing sound, etc.) required for sleep stage analysis may be relatively smaller than other noises, but if this is converted into information including changes in frequency components along the time axis (e.g., information in the form of a spectrogram), the identification of the sleep sound information may be relatively superior compared to other surrounding noises.

Meanwhile, when converting to a spectrogram according to an embodiment of the present invention, personal information cannot be identified by converting the resolution of the frequency domain to a low level, and personal information cannot be identified from the restored signal if it is configured with a frequency resolution (frequency bins) of less than a certain number (e.g., 20). Here, the number of frequency bins is exemplified as 20, but it should be understood that the present invention is not limited thereto, and any number that does not allow personal information to be identified from the restored signal can be assumed.

In addition, according to an embodiment of the present invention, a method may be included for converting acquired sleep sound information into information (e.g., information in the form of a spectrogram, etc.) including changes in frequency components along a time axis in real time (e.g., 5 seconds, 10 seconds, 15 seconds, 20 seconds, 30 seconds, 60 seconds, 5 minutes, etc.). The time unit described above is merely an example for convenience of explanation and is not limited thereto, and sleep sound information for a time shorter than 5 seconds may be converted immediately whenever it is acquired, or sleep sound information for a time longer than 5 minutes may be acquired and then converted.

In addition, according to an embodiment of the present invention, compression of the frequency resolution of information (e.g., information in the form of a spectrogram) including changes in frequency components along the time axis can be performed on the user's smartphone rather than on a server or cloud, thereby preventing leakage of personal information.

Anonymization of environmental sensing information including sleep sound information according to embodiments of the present invention can be performed for natural language and respiratory sounds, and such anonymization can be performed by converting environmental sensing information including sleep sound information into natural language converted spectrograms, respiratory sound converted spectrograms, etc., respectively. In sleep analysis according to the present invention, only the information necessary for the analysis model can be utilized to improve the computational speed and reduce the computational load. Meanwhile, information including a change in a time axis of a frequency component according to an embodiment of the present invention (for example, information in the form of a spectrogram) can be a mel spectrogram to which a mel scale is applied, but is not limited thereto.

Method for converting environmental sensing information

Figure 46 is a drawing for explaining a method of obtaining a spectrogram corresponding to sleep sound information in a sleep analysis method according to the present invention.

According to an embodiment of the present invention, the processor (110-3), the processor (210-3), and/or the processor (310-3) can convert sleep sound information (SS) and generate a spectrogram (SP) corresponding thereto. Raw data in the time domain, which is the basis for generating the spectrogram (SP), can be acquired or received. The raw data according to the present invention can be acquired through a polysomnography (PSG) test in a hospital environment, or can be acquired by a user in a home environment, etc., through a microphone built into a user terminal such as a wearable device or a smartphone.

In addition, raw data may be acquired through a user terminal (10-3) such as a wearable device or a smartphone from a start time input by the user to an end time, or may be acquired from a time when the user operates the device (e.g., setting an alarm) to a time corresponding to the device operation (e.g., alarm setting time), or may be acquired by automatically selecting a time point based on the user's sleep pattern, or may be acquired by automatically determining a time point based on sound (such as the user's voice, breathing sound, or peripheral device (TV, washing machine) sound) or changes in illumination included in the environmental sensing information of the user's intended sleep time.

According to an embodiment of the present invention, the processor (110-3), the processor (210-3), and/or the processor (310-3) may perform a fast Fourier transform on the sleep sound information (SS) to convert it into information including changes in frequency components of the sleep sound information along the time axis. Specifically, this information may be information in the frequency domain, and may be a spectrogram or a Mel spectrogram to which a Mel scale is applied. The information including changes in frequency components along the time axis, the information in the frequency domain, or the spectrogram (SP) is intended to visualize and understand sounds or waves, and may be a combination of waveform and spectrum characteristics. In addition, this information may be visualized by representing the difference in amplitude according to the change in the time axis and the frequency axis of the frequency components of the sound information as a difference in printing density or display color. In this case, the visualization enables acquisition of sleep state information as input to an artificial intelligence model based on image processing. Through this method, time-series sleep analysis is possible by converting sound signals into image signals, utilizing data over a relatively long period of time, and the accuracy of sleep analysis can be improved compared to analysis based on raw sleep sound information.

According to one embodiment of the present invention, preprocessed sound-related raw data can be cut into 30-second units and converted into a spectrogram. Accordingly, the 30-second spectrogram has a dimension of 20 frequency bins x 1201 time steps. In the present invention, various methods such as reshaping, resizing, and split-cat can be used to change a rectangular spectrogram into a shape close to a square, thereby converting it into a shape close to a square. Alternatively, information can be preserved by using these methods.

Meanwhile, the present invention can use a method to simulate breathing sounds measured in various home environments by adding various noises generated in a home environment to clean breathing sounds. Since sounds have an additive property, they can be added to each other. However, if original audio signals such as mp3 or pcm are added and converted into spectrograms, the consumption of computing resources becomes very large. Therefore, the present invention proposes a method to convert breathing sounds and noises into spectrograms respectively and add them. Through this, it is possible to secure robustness in various home environments by simulating breathing sounds measured in various home environments and utilizing them for deep learning model learning.

Preprocessing of converted information

The purpose of converting data according to one embodiment of the present invention into information including changes in frequency components along the time axis, or information in the frequency domain, or a spectrogram, is to use the converted information as input to a sleep analysis model and infer, through a learned model, which sleep state or sleep stage a pattern in the information corresponds to. However, before using the information as input to the sleep analysis model, several preprocessing steps may be necessary.

In addition, according to one embodiment of the present invention, the converted information may be converted into image information so that the sound information can be input to an image processing-based artificial intelligence model. Meanwhile, before the converted information is input to an artificial intelligence model, an additional preprocessing process may be performed, and these preprocessing processes may be performed only during the learning process, or may be performed not only during the learning process but also during the inference process. Or, they may be performed only during the inference process.

According to an embodiment of the present invention, a preprocessing method for performing data augmentation on a spectrogram may be included.

Data augmentation is to secure a sufficient amount of training data sets or to conduct sufficient learning by assuming diverse and irregular environments.

A data augmentation preprocessing method according to an embodiment of the present invention may include a pitch shifting preprocessing method that inflates the amount of data by adding Gaussian noise to a spectrogram, or slightly raises or lowers the pitch of the overall acoustic information, and a TUT (Tile UnTile) augmentation method that converts a spectrogram or a mel spectrogram into a vector during a learning process, and randomly cuts (tiles) the converted vector at the input stage of a node (neuron) and recombines (untile) it after the output of the node (neuron).

The TUT (Tile UnTile) augmentation according to an embodiment of the present invention may include a step of randomly cutting and combining a spectrogram or vector at the input and output stages of a node (neuron) in order to increase the amount of learning data of various patterns when a spectrogram is used as an input of a learning model. A spectrogram or vector randomly cut at the input stage of a node (neuron) may have missing data or a loss in the amount of information compared to the information of a spectrogram or vector input to a layer of a corresponding neural network that is not cut. In this case, limited information may be input to the node (neuron) and learned. A node (neuron) that inputs a cut spectrogram or vector may output a vector after the operation. At this time, it may be combined (Untile) again in the same way as it was cut before being used as an input of a node (neuron) of the next neural network layer.

In addition, TUT augmentation according to an embodiment of the present invention can contribute to increasing the accuracy or reliability of a learning model by inducing learning of data with missing information by randomly cutting spectrograms or vectors at the input and output stages of nodes (neurons) and merging them in the same manner.

In addition, the data augmentation preprocessing method according to the embodiment of the present invention may include a noise addition augmentation method that adds noise generated in various environments (e.g., external sounds, natural sounds, fan sounds, door opening or closing sounds, animal sounds, human conversation sounds, movement sounds, etc.) other than Gaussian noise. Alternatively, the noise addition augmentation according to the embodiment of the present invention may include a method of adding on a domain converted into a mel spectrogram to which mel scale is applied, as well as a method of adding on a domain converted into a mel spectrogram to which mel scale is applied. According to the method of adding on a mel scale according to the embodiment of the present invention, the time required for hardware to process data can be shortened.

Meanwhile, the specific description of the types of noise described above is merely a simple example for explaining the noise addition augmentation of the present invention, and the present invention is not limited thereto.

According to an embodiment of the present invention, after performing a data augmentation preprocessing process of information or spectrogram in the frequency domain, such as the pitch shifting, noise augmentation, or TUT augmentation described above, a preprocessing method of converting information or spectrogram in the frequency domain into a shape close to a square may be performed.

According to one embodiment of the present invention, before data augmentation is performed on frequency domain information or spectrogram as input to a deep learning model, such as CNN, Transformer, Vision Transformer (ViT), or Mobile Vision Transformer (MobileViT), the information or spectrogram on the frequency domain converted to a shape close to a square can be input to an AI deep learning model.

In performing preprocessing to convert information or spectrogram in the frequency domain into a shape close to a square according to an embodiment of the present invention, the conversion into a shape close to a square can be performed in various ways, such as reshaping, resizing, and split-cat.

In the case of performing preprocessing to convert to a shape close to a square using a resizing method according to an embodiment of the present invention, a method of lowering the resolution of the x-axis and increasing the resolution of the y-axis by copying the values can be performed.

In addition, according to an embodiment of the present invention, a preprocessing method is performed to convert the entire 30-second spectrogram having a dimension of 20 frequency bins × 1201 time steps into a shape close to a square at once by resizing, and missing information can be supplemented by utilizing an interpolation method.

The split-cat preprocessing method according to an embodiment of the present invention may include a method of splitting a spectrogram into a predetermined size and then merging (cat) the data into a shape close to a square using a concatenation function. In other words, the split-cat method means a method of splitting a spectrogram into patches and then merging the patches into a shape close to a square in order to perform learning on a patch-by-patch basis in a deep learning model based on ViT (Vision Transformer) or Mobile ViT (Mobile Vision Transformer).

For example, according to one embodiment of the present invention, a 30-second spectrogram having a dimension of 20 frequency bins × 1201 time steps can be converted into a dimension of 150 frequency bins × 160 time steps. Then, a resizing technique can be used to convert the 30-second spectrogram having a dimension of 150 frequency bins × 160 time steps into a dimension close to 160 frequency bins × 160 time steps. Through this process, it is possible to convert a spectrogram corresponding to every 30-second interval into a shape close to a square.

When learning is performed using a spectrogram converted to a shape close to a square as input, the deep learning model based on the Transformer learning model can exhibit improved learning performance. The specific numerical descriptions of the bins, division time units, and number of divisions of the above-mentioned spectrogram are only examples, and the present invention is not limited thereto.

Information or spectrogram on the frequency domain according to an embodiment of the present invention may be unsuitable as an input to a deep learning model because its values are very small and, if not converted to a different scale, it is expressed very brightly in a part where the value is greater than a certain level, whereas it is expressed very dark in the remaining part. Accordingly, before using information or spectrogram on the frequency domain according to an embodiment of the present invention as an input to a deep learning model, a preprocessing step of converting it to a dB scale (log scale) may be performed.

In performing log scale conversion preprocessing according to an embodiment of the present invention, the maximum value of the log value can be set to 0 as a basic base value, and the remaining values can be converted into log values.

According to an embodiment of the present invention, a spectrogram converted into a log value may be additionally subjected to normalization preprocessing so that the average of all values becomes 0 and the standard deviation becomes 1, and then used as input for a deep learning model.

By using the data preprocessed in this way as input to an image processing deep learning model, information such as sleep state information can be learned or inferred through image analysis of the spectrogram. The specific numerical description of the maximum log value of the spectrogram mentioned above is only an example, and the present invention is not limited thereto.

Sleep analysis model

Fig. 47 is a flowchart exemplarily showing a method for analyzing a user's sleep state through sound information according to one embodiment of the present invention. According to one embodiment of the present invention, the method may include a step (S610-3) of obtaining sleep sound information related to the user's sleep. According to one embodiment of the present invention, the method may include a step (S620-3) of performing preprocessing on the sleep sound information. According to one embodiment of the present invention, the method may include a step (S630-3) of performing analysis on the preprocessed sleep sound information to obtain sleep state information. The steps illustrated in Fig. 6 described above may be changed in order as necessary, and at least one or more steps may be omitted or added. That is, the steps described above are merely an embodiment of the present invention, and the scope of the rights of the present invention is not limited thereto.

Figures 48a and 48b are diagrams for explaining the overall structure of a sleep analysis model according to one embodiment of the present invention. In the present invention, sleep state information can be obtained through a sleep analysis model that analyzes a user's sleep stage based on acoustic information (sleep acoustic information).

In the present invention, since the sleep sound information (SS) is a sound related to breathing and body movement acquired during the user's sleep time, it can be a very small sound, and accordingly, the present invention can perform an analysis of the sound by converting the sleep sound information (SS) into a spectrogram (SP) as described above. In this case, since the spectrogram (SP) includes information showing how the frequency spectrum of the sound changes over time, it is possible to easily identify a breathing or movement pattern related to a relatively small sound, and thus the efficiency of the analysis can be improved. Specifically, it may be difficult to predict whether the state is at least one of an awake state, a REM sleep state, a light sleep state, and a deep sleep state based only on a change in the energy level of the sleep sound information, but by converting the sleep sound information into a spectrogram, it is possible to easily detect a change in the spectrum of each frequency, and thus analysis corresponding to a small sound (e.g., breathing and body movement) can be made possible.

According to an embodiment of the present invention, the processor (110-3), the processor (210-3), and/or the processor (310-3) may process information in the frequency domain or the spectrogram (SP) converted according to the embodiment of the present invention as input to a sleep analysis model to obtain sleep state information. Here, the sleep analysis model is a model for obtaining sleep state information related to a change in the user's sleep stage, and may output sleep state information by inputting sleep sound information obtained during the user's sleep. In an embodiment, the sleep analysis model may include a neural network model configured through one or more neural networks.

Feature extraction model and feature classification model

FIG. 49 is a diagram for explaining a feature extraction model and a feature classification model according to one embodiment of the present invention. FIG. 50 is a diagram for explaining in detail the operation of a sleep analysis model according to one embodiment of the present invention.

The sleep analysis model used in the present invention may include a feature extraction model that extracts one or more features for each predetermined epoch, and a feature classification model that classifies each of the features extracted through the feature extraction model into one or more sleep stages to generate sleep state information. The feature extraction model may extract features related to breathing sounds, breathing patterns, and movement patterns by analyzing a time-series frequency pattern of a spectrogram (SP). In one embodiment, the feature extraction model may be configured through a part of a neural network model that is pre-learned through a learning data set.

The sleep analysis model used in the present invention may include a feature extraction model and a feature classification model. The feature extraction model may be a deep learning learning model based on a natural language processing model capable of learning the time-series correlation of given data. The feature classification model may be a learning model based on a natural language processing model capable of learning the time-series correlation of given data. Here, the deep learning learning model based on a natural language processing model capable of learning the time-series correlation may include, but is not limited to, Tarnsformer, ViT, MobileViT, and MobileViT2.

A learning data set according to one embodiment of the present invention may be composed of data in the frequency domain and a plurality of pieces of sleep state information corresponding to each piece of data. Alternatively, a learning data set according to one embodiment of the present invention may be composed of a plurality of spectrograms and a plurality of pieces of sleep state information corresponding to each spectrogram. Alternatively, a learning data set according to one embodiment of the present invention may be composed of a plurality of Mel spectrograms and a plurality of pieces of sleep state information corresponding to each Mel spectrogram.

For convenience of explanation, the configuration and execution of a sleep analysis model according to one embodiment of the present invention will be described in detail based on a data set of spectrograms. However, the learning data utilized in the sleep analysis model of the present invention is not limited to spectrograms, and information including changes in frequency components of environmental sensing information including sleep sound information along the time axis, spectrograms, or mel spectrograms can be utilized as learning data and/or inference data.

Among the sleep analysis models according to the embodiment of the present invention, the feature extraction model can be pre-trained by a one-to-one proxy task that inputs one spectrogram and learns to predict sleep state information corresponding to one spectrogram. In the case of adopting a CNN deep learning model in the feature extraction model according to the embodiment of the present invention, learning may be performed by adopting a structure of a FC (Fully Connected Layer) or a FCN (Fully Connected Neural Network). In the case of adopting a MobileViTV2 deep learning model in the feature extraction model according to the embodiment of the present invention, learning may be performed by adopting a structure of an intermediate layer.

Among the sleep analysis models according to one embodiment of the present invention, a feature classification model can be trained to predict sleep state information of each spectrogram by inputting a plurality of continuous spectrograms, and to predict or classify overall sleep state information by analyzing a sequence of the plurality of continuous spectrograms.

In addition, according to an embodiment of the present invention, after pre-learning is performed through a One-to-one proxy task for a feature extraction model, fine tuning can be performed through a Many-to-Many task for the pre-learned feature extraction model and feature classification model. For example, a sequence of 40 consecutive spectrograms can be input into multiple feature extraction models learned through a One-to-one proxy task, and 20 pieces of sleep state information can be output, thereby inferring sleep stages. The specific numerical descriptions related to the number of spectrograms, the number of feature extraction models, and the number of sleep state information described above are merely examples, and the present invention is not limited thereto.

Hereinafter, a feature extraction model and a feature classification model generated or learned based on a spectrogram converted according to an embodiment of the present invention will be described in detail. Meanwhile, the sleep analysis model of the present invention is not limited to being generated or learned based on a spectrogram, and as described above, may be generated or learned based on information including changes in frequency components of raw acoustic information along the time axis or information in the frequency domain. In addition, inference of sleep state information through the sleep analysis model may also be performed based on information including changes in frequency components of raw acoustic information along the time axis or information converted into information in the frequency domain.

FIG. 51a and FIG. 51b are diagrams for explaining the performance of sleep event judgment and noise addition using a spectrogram in a sleep analysis method according to the present invention. As shown in FIG. 51a and FIG. 51b, according to a sleep analysis model according to one embodiment of the present invention, sleep events such as apnea can be detected with high reliability even when a spectrogram is damaged by noise.

Feature extraction model

FIG. 49 is a diagram for explaining a feature extraction model and a feature classification model according to one embodiment of the present invention.

According to the present invention, the feature extraction model can be composed of an independent deep learning model learned through a learning data set. The feature extraction model can be learned through a supervised learning or unsupervised learning method. The feature extraction model can be learned to output output data similar to the input data through the learning data set. Specifically, only the core feature data (or feature) of the input spectrogram can be learned through the hidden layer. In this case, during the decoding process through the decoder, the output data of the hidden layer can be an approximation of the input data (i.e., the spectrogram) rather than a perfect copy value.

According to the present invention, each of a plurality of spectrograms included in a learning data set may be tagged with sleep state information. Each of the plurality of spectrograms may be input to a feature extraction model, and an output corresponding to each spectrogram may be stored by matching with the tagged sleep state information. Specifically, when first learning data sets (i.e., a plurality of spectrograms) tagged with first sleep state information (e.g., shallow sleep) are input, features related to the output for the corresponding input may be stored by matching with the first sleep state information. In an embodiment, one or more features related to the output may be displayed in a vector space. In this case, since the feature data output corresponding to each of the first learning data sets is an output through a spectrogram related to the first sleep stage, they may be located at a relatively close distance in the vector space. That is, learning may be performed so that the plurality of spectrograms output similar features corresponding to each sleep stage.

The feature extraction model through the aforementioned learning process can extract features corresponding to a spectrogram when a spectrogram (e.g., a spectrogram converted in response to sleep sound information) is input.

According to one embodiment of the present invention, the processor (110-3), the processor (210-3), and/or the processor (310-3) may process a spectrogram (SP) generated in response to sleep sound information (SS) as an input of a feature extraction model to extract a feature. Here, since the sleep sound information (SS) is time-series data acquired in a time-series manner during the user's sleep, the processor (110-3), the processor (210-3), and/or the processor (310-3) according to embodiments of the present invention may divide the spectrogram (SP) into predetermined epochs. For example, the processor (110-3), the processor (210-3), and/or the processor (310-3) according to embodiments of the present invention may divide the spectrogram (SP) corresponding to sleep sound information (SS) into 30-second units to obtain a plurality of spectrograms. For example, if sleep sound information is acquired during the user's 7 hours (i.e., 420 minutes) of sleep, the processor (110-3), the processor (210-3), and/or the processor (310-3) according to embodiments of the present invention can divide the spectrogram into 30-second units to acquire 840 spectrograms. The specific numerical descriptions of the sleep time, the spectrogram division time unit, and the number of divisions described above are only examples, and the present invention is not limited thereto.

According to an embodiment of the present invention, the processor (110-3), the processor (210-3), and/or the processor (310-3) may process each of the plurality of segmented spectrograms as an input of a feature extraction model to extract a plurality of features corresponding to each of the plurality of spectrograms. For example, when the number of the plurality of spectrograms is 840, the number of the plurality of features extracted by the feature extraction model may also be 840. The specific numerical descriptions related to the above-described number of spectrograms and the plurality of features are merely examples, and the present invention is not limited thereto.

Meanwhile, the feature extraction model according to the embodiment of the present invention may perform learning by a one-to-one proxy task. In addition, in the process of learning to extract sleep state information for one spectrogram, the feature extraction model may be learned to extract sleep state information by combining another NN (Neural Network).

According to an embodiment of the present invention, if learning is performed through a simple pre-learned Neural Network, the learning time of a feature extraction model can be shortened or the learning efficiency can be increased.

For example, according to one embodiment of the present invention, a spectrogram divided into 30-second units may be trained to output sleep state information by using the output vector as an input to a feature extraction model and using the output vector as an input to another NN.

Meanwhile, according to one embodiment of the present invention, the processor (110-3), the processor (210-3), and/or the processor (310-3) according to embodiments of the present invention may generate a feature extraction model by utilizing multiple source data. In addition, the feature extraction model may include a dimensionality reduction network function (e.g., an encoder).

In an embodiment, each of a plurality of frequency domain information or spectrograms (i.e., a plurality of transformed pieces of information corresponding to a plurality of source data) utilized as learning data may be labeled with sleep stage information. For example, the plurality of source data may be information about sleep sounds obtained in a specific space (e.g., a hospital), and information about a plurality of sleep states (i.e., sleep stages) may be pre-labeled.

Each of the transformed plurality of pieces of information can be input to a dimensionality reduction network function, and the output corresponding to each transformed piece of information can be matched with the labeled sleep stage information. Specifically, when the first sleep stage information (e.g., shallow sleep) is input to the dimensionality reduction network function using the labeled first learning data sets (e.g., multiple spectrograms related to the source data), features related to the output of the dimensionality reduction network function for the corresponding input can be matched with the first sleep stage information.

In an embodiment, one or more features related to the output of the dimensionality reduction network function may be represented in a vector space. In this case, since the feature data output corresponding to each of the first learning data sets is an output through a spectrogram related to the first sleep stage (e.g., an output through a spectrogram corresponding to the same class), they may be located at a relatively close distance in the vector space.

That is, learning of a dimensionality reduction network function can be performed so that multiple spectrograms output similar features corresponding to each sleep stage, but the specific learning method of the dimensionality reduction network is not limited.

Through the learning process described above, the feature extraction model can extract features corresponding to the converted information when inputting information converted in response to sleep sound information.

Feature classification model

Figure 50 is a drawing for explaining in detail the operation of a sleep analysis model according to one embodiment of the present invention.

According to one embodiment of the present invention, the processor (110-3), the processor (210-3), and/or the processor (310-3) may process a plurality of features output through the feature extraction model as inputs of a feature classification model to obtain sleep state information. In an embodiment, the feature classification model may be a neural network model modeled to predict a sleep stage corresponding to a feature. For example, the feature classification model may be a model configured to include a fully connected layer and classify a feature into at least one of the sleep stages. For example, when the feature classification model inputs a first feature corresponding to a first spectrogram, the first feature may be classified as shallow sleep. Or, for example, when the feature classification model inputs a second feature corresponding to a second spectrogram, the second feature may be classified as deep sleep. Or, when the feature classification model inputs a third feature corresponding to a third spectrogram, the third feature may be classified as REM sleep. Or, for example, if a feature classification model takes as input a fourth feature corresponding to the fourth spectrogram, it can classify the fourth feature as a noise.

In addition, in one embodiment of the present invention, the feature classification model may be a neural network model modeled to predict an event occurring in sleep in response to a feature. For example, the feature classification model may be configured to include a fully connected layer and may be a model that classifies a feature into at least one of events occurring in sleep. For example, when the feature classification model inputs a first feature corresponding to a first spectrogram, the first feature may be classified as an occurrence of a sleep apnea event. For example, when the feature classification model inputs a second feature corresponding to a second spectrogram, the second feature may be classified as an occurrence of a sleep hypoapnea event. For example, when the feature classification model inputs a third feature corresponding to a third spectrogram, the third feature may be classified as a normal sleep state. For example, when the feature classification model inputs a fourth feature corresponding to a fourth spectrogram, the fourth feature may be classified as an event of snoring during sleep. For example, if a feature classification model takes as input the fifth feature corresponding to the fifth spectrogram, it can classify the fifth feature as a sleep talking event during sleep.

According to one embodiment of the present invention, a feature classification model can perform classification on a plurality of features. The feature classification model can classify each of the plurality of features into at least one of the plurality of sleep stages. In addition, according to one embodiment of the present invention, a feature extraction model extracts features so that the feature classification model can easily classify which sleep stage it corresponds to, and the feature classification model can be trained to better classify the features transferred from the feature extraction model (i.e., to better classify into a specific sleep stage). In other words, through adversarial learning, the feature classification model can better classify the features transferred from the feature extraction model. In one embodiment, the feature classification model can be trained to easily classify classes between features in order to well perform sleep stage classification or sleep event classification corresponding to the features.

According to one embodiment of the present invention, the processor (110-3), the processor (210-3), and/or the processor (310-3) may update the feature extraction model through first learning information related to a first loss between the feature extraction model and the feature classification model according to a learning performance result of the feature extraction model and the feature classification model. Accordingly, the updated feature extraction model may extract features that enable the feature classification model to classify classes (i.e., sleep stages) well. In other words, the updated feature extraction model may extract features such that each of the features related to the same sleep stage is clustered.

According to one embodiment, the discriminator model may be a neural network model that distinguishes whether each of the plurality of features transmitted from the feature extraction model is a feature related to the source data or a feature related to the target data. For example, the discriminator model may determine whether the feature related to the input is a feature corresponding to sleep sound information acquired in a hospital or a feature corresponding to sleep sound information acquired in real life of an individual user. That is, the discriminator model may receive at least one of the plurality of features as an input and distinguish whether the feature related to the input is a feature related to the source data or a feature related to the target data.

That is, in the process of inferring sleep state information, rather than performing prediction of sleep state information corresponding to each single spectrogram, the accuracy of the output can be improved by utilizing spectrograms corresponding to multiple epochs as inputs so that both past and future information can be considered. Meanwhile, in addition to spectrograms, the accuracy of the output can be improved by performing inference using information including changes in frequency components along the time axis corresponding to multiple epochs or information in the frequency domain as inputs.

Figure 52 is a drawing for explaining sleep stage analysis using a spectrogram in a sleep analysis method according to the present invention.

According to one embodiment of the present invention, after the primary sleep analysis based on actigraphy and HRV, the secondary analysis based on sleep sound information uses the sleep analysis model as described above, and as illustrated in FIG. 52, when the user's sleep sound information is input, the corresponding sleep stage (Wake, REM, Light, Deep) can be immediately inferred. In addition, the secondary analysis based on sleep sound information can extract the time point at which a sleep event (sleep apnea, hyperventilation, snoring, etc.) occurs through the singularity of the Mel spectrum corresponding to the sleep stage.

Figure 53 is a diagram for explaining sleep event determination using a spectrogram in a sleep analysis method according to the present invention. According to one embodiment of the present invention, sleep events may include, but are not limited to, sleep apnea, hyperventilation, hypopnea, snoring, sleep talking, and bruxism.

As illustrated in Fig. 53, a breathing pattern is analyzed in one transformed frequency domain information or spectrogram or Mel spectrogram, and when a characteristic corresponding to an apnea or hyperpnea event is detected, the corresponding point in time can be determined as the point in time when the sleep event occurred. At this time, a process of classifying it as snoring rather than apnea or hyperpnea through frequency analysis may be further included.

Figure 54 is a drawing showing an experimental process for verifying the performance of a sleep analysis method according to the present invention.

As shown in Fig. 54, the user's sleep image and sleep sound are acquired in real time, and the acquired sleep sound information is immediately converted into a spectrogram. At this time, a preprocessing process of the sleep sound information can be performed. The spectrogram can be input into a sleep analysis model, and the sleep stage can be immediately analyzed. In addition, when a CNN or Transformer-based deep learning model is employed in a feature classification model according to an embodiment of the present invention, the operation can be performed as follows.

According to one embodiment of the present invention, a spectrogram containing time series information can be input to a CNN-based deep learning model to output a vector with a reduced dimension. The vector with the reduced dimension can be input to a Transformer-based deep learning model to output a vector containing time series information.

According to an embodiment of the present invention, the average pooling technique can be applied to the output vector of a Transformer-based deep learning model by inputting it into a 1D CNN (1D Convolutional Neural Network), and performing an averaging operation on the time series information to convert it into an N-dimensional vector containing time series information. In this case, the N-dimensional vector containing time series information corresponds to data that still contains time series information, although it has a different resolution from the input data.

According to an embodiment of the present invention, multi-epoch classification can be performed on a combination of N-dimensional vectors containing output time series information, thereby making predictions for multiple sleep stages. In this case, the output vectors of Transformer-based deep learning models can be used as inputs to multiple FCs (Fully Connected Layers) to make predictions for continuous sleep state information.

In addition, when a deep learning model based on ViT or Mobile ViT is employed in a feature classification model according to one embodiment of the present invention, the operation can be performed as follows.

According to one embodiment of the present invention, a spectrogram including time series information can be used as an input for a deep learning model based on Mobile ViT to output a vector with a reduced dimension. In addition, according to one embodiment of the present invention, features can be extracted from each spectrogram as an output of the deep learning model based on Mobile ViT.

According to an embodiment of the present invention, a vector with a reduced dimension can be input to an Intermediate Layer, and a vector containing time series information can be output. The Intermediate Layer model can include at least one of a linearization step containing vector information, a layer normalization step for inputting mean and variance, or a dropout step for deactivating some nodes.

According to an embodiment of the present invention, by performing a process of inputting a vector with a reduced dimension to an intermediate layer and outputting a vector containing time series information, overfitting can be prevented.

According to an embodiment of the present invention, the output vector of the Intermediate Layer can be used as an input of a ViT-based deep learning model to output sleep state information. In this case, sleep state information corresponding to information in a frequency domain containing time series information, a spectrogram, or a Mel spectrogram can be output. In addition, according to an embodiment of the present invention, sleep state information corresponding to a series of configurations of information in a frequency domain containing time series information, a spectrogram, or a Mel spectrogram can be output.

Meanwhile, in addition to the AI model mentioned above, various deep learning models may be employed in the feature extraction model or feature classification model according to the embodiment of the present invention to perform learning or inference, and the specific description related to the types of the deep learning models mentioned above is merely an example, and the present invention is not limited thereto.

Performance of sleep analysis method according to the present invention

Comparing with the results of polysomnography (PSG), we were able to confirm that the results of the sleep analysis model that uses sleep sound information as input are very accurate.

Existing sleep analysis models predict sleep stages using ECG (Electrocardiogram) or HRV (Heart Rate Variability) as input, but the present invention can analyze and infer sleep stages by converting sleep sound information into frequency domain information, spectrogram, or mell spectrogram and using them as input. Therefore, since sleep sound information is converted into frequency domain information, spectrogram, or mell spectrogram and used as input, unlike existing sleep analysis models, sleep stages can be sensed or acquired in real time by analyzing the specificity of sleep patterns.

Figure 55 is a graph verifying the performance of a sleep analysis method according to the present invention, and is a drawing comparing the polysomnography (PSG) result (PSG result) and the analysis result (AI result) using the AI algorithm according to the present invention.

As illustrated in FIG. 55, the sleep analysis results obtained according to the present invention are not only highly consistent with polysomnography, but also include more precise and meaningful information related to sleep stages (Wake, Light, Deep, REM). The hypnogram illustrated at the bottom of FIG. 54 indicates the probability of belonging to one of the four classes (Wake, Light, Deep, REM) for each 30 seconds when predicting sleep stages by inputting user sleep sound information. Here, the four classes represent the awake state, the lightly asleep state, the deep asleep state, and the REM sleep state, respectively.

Fig. 56 is a graph verifying the performance of the sleep analysis method according to the present invention, and is a diagram comparing the polysomnography (PSG) result (PSG result) with the analysis result (AI result) using the AI algorithm according to the present invention in relation to sleep apnea and hypopnea. The hypnogram illustrated at the bottom of Fig. 56 indicates the probability of which of the two diseases (sleep apnea, hypopnea) it belongs to every 30 seconds when predicting a sleep disease by receiving user sleep sound information. When using the sleep analysis according to the present invention, as illustrated in Fig. 56, the sleep state information obtained according to the present invention is not only highly consistent with the polysomnography, but also includes more precise analysis information related to apnea and hypopnea.

The present invention can analyze a user's sleep analysis in real time and identify the point where a sleep event (sleep apnea, hyperventilation, hypopnea, snoring, sleep talking, teeth grinding, etc.) occurs. If a stimulus (tactile, auditory, olfactory stimulus, etc.) is provided to the user at the moment when the sleep event occurs, the sleep event can be temporarily alleviated. That is, the present invention can stop the user's sleep event and reduce the frequency of the sleep event based on accurate event detection related to the sleep event. In addition, according to the present invention, there is also an effect that highly accurate sleep analysis is possible by performing sleep analysis in a multimodal manner.

In addition, in the inference step of inferring sleep stages through learning, sleep may be composed of not only a certain period of time (e.g., 30 seconds, 20 minutes, etc.) like learning data, but also a sleep time (e.g., 5 hours, 8 hours, etc.). In order to accurately infer sleep stages, post-processing may be performed to increase the accuracy of inference in light of the sleep duration. The specific numbers related to the time intervals described above are merely examples, and the present invention is not limited thereto.

According to one embodiment of the present invention, post-processing can be performed using medical information to infer the depth of sleep according to the sleep duration.

According to one embodiment of the present invention, post-processing can be performed through artificial intelligence learning using sleep stage information data according to sleep duration.

According to an embodiment of the present invention, analysis of sleep state information based on acoustic information may include a detection step for sleep events (e.g., sleep apnea, hyperventilation, hypopnea, snoring, sleep talking, teeth grinding, etc.). However, since the characteristics of sleep sound patterns are reflected over time, it may be difficult to identify them with only short acoustic data at a specific point in time. Therefore, in order to model acoustic information, analysis should be performed based on the time series characteristics of acoustic information.

In addition, sleep events that occur during sleep (e.g., apnea, hypopnea, snoring, sleep talking, etc.) have various characteristics related to sleep events. For example, there is no sound during an apnea event, but when the apnea event ends, a loud sound may occur as air passes through again, and the characteristics of the apnea event can be learned in time series to detect sleep events.

Meanwhile, according to one embodiment of the present invention, when analyzing acoustic information in the time domain, only a portion cut off by the length of a predetermined section on the time axis can be analyzed. A portion corresponding to the length of a predetermined section can be called a window (time window), and according to one embodiment of the present invention, the length of a predetermined section on the time axis of the window can be 20 minutes, but is not limited thereto.

According to one embodiment of the present invention, sleep for 20 minutes can be analyzed based on each piece of sound information corresponding to a 20-minute window. For the measured sound window, analysis can be performed at predetermined time intervals shorter than the length of the window section. For example, sleep analysis can be performed by generating sleep state information every 5 minutes for a window corresponding to 20 minutes. By generating sleep state information for a relatively short period of time in this way, there is an effect that sleep can be analyzed in real time. In the case of analyzing sleep by window, the size of data input to the sleep analysis model can be relatively small, so there is also an effect that the sleep analysis time can be shortened.

In order to detect a sleep event occurring during sleep according to an embodiment of the present invention, the deep neural network structure for analyzing the sleep stage described above can be modified and used. Specifically, although sleep stage analysis requires time-series learning of sleep sounds, sleep event detection can occur between 10 and 60 seconds on average, so it is sufficient to accurately detect 1 or 2 epochs of 30 seconds each. Therefore, the deep neural network structure for analyzing sleep stages according to an embodiment of the present invention can reduce the input and output amounts of the deep neural network structure for analyzing sleep stages. For example, if the deep neural network structure for analyzing sleep stages processes 40 Mel Spectrograms and outputs a sleep stage of 20 epochs, the deep neural network structure for detecting sleep events can process 14 Mel Spectrograms and output a sleep event label of 10 epochs. Here, sleep event labels may include, but are not limited to, no event, apnea, hypopnea, snoring, tossing and turning, etc.

In addition, according to an embodiment of the present invention, a deep neural network structure for detecting sleep events occurring during sleep may include a feature extraction model and a feature classification model. Specifically, the feature extraction model extracts features of sleep events found in each Mel Spectrogram, and the feature classification model detects multiple epochs to find epochs containing sleep events, analyzes neighboring features, and predicts and classifies the type of sleep event in a time series manner.

In addition, according to one embodiment of the present invention, by generating sleep state information by multimodally adding other information (e.g., user's biometric information) to each sound information corresponding to a window, more accurate sleep analysis can be achieved.

Automatic measurement system

FIG. 57 is a drawing for explaining the configuration of an automatic measurement system according to one embodiment of the present invention.

As illustrated, the automatic measurement system (1000-3) according to the present invention may include a permission management tool (1100-3), a time reservation tool (1200-3), a foreground service management tool (1300-3), and a sleep data collection tool (1400-3).

The present invention can provide an automatic measurement system (1000-3) for initiating sleep measurement through a user terminal. Specifically, the authority management tool (1100-3) can collect authority required to perform sleep measurement. In addition, the sleep data collection tool (1400-3) can activate a sensor module (360) for collecting environmental sensing information of the user terminal (300) based on the authority collected by the authority management tool (1100-3).

Permissions Management Tool (1100-3)

According to one embodiment of the present invention, the permission management tool (1100-3) may perform a role of collecting permissions required for automatic sleep measurement in an Android app. Specifically, the permission management tool (1100-3) may protect the user's privacy by collecting only essential permissions such as Overlay permission permission, activation permission of the sensor module (360), foreground service permission, and time-based alarm permission during the permission collection process. In addition, the permission management tool (1100-3) may provide a visual interface so that the user can allow or deny the permission, and may also explain the necessity of the permission, such as "microphone permission is required to start automatic sleep measurement."

According to one embodiment of the present invention, the permission management tool (1100-3) may be configured to provide a user with a permission activation notification or guide the user to move to a settings screen when the permission is disabled. In addition, the permission management tool (1100-3) may detect in real time when the permission status is changed due to a system update or a setting change and provide a notification to the user.

According to one embodiment of the present invention, if the permission is disabled, the permission management tool (1100-3) may provide a user interface such as "Automatic sleep measurement cannot be started without permission." In addition, the permission management tool (1100-3) does not perform unnecessary permission requests, such as requesting only microphone permission and foreground service permission for automatic sleep measurement. In addition, the permission management tool (1100-3) may warn of this and execute a re-request process if a major function is restricted due to insufficient permission.

Broadcast triggers and sleep analysis triggers

The permission management tool (1100-3) of the present invention can operate based on a broadcast trigger for sleep measurement and a sleep analysis trigger. Specifically, the broadcast trigger can be a trigger for detecting various system events by utilizing a Broadcast Receiver of an Android system and using them as an activation point for a sensor module (360-3).

FIG. 64 is a drawing showing an embodiment of providing an activation notification of a sensor module when a broadcast trigger occurs according to the present invention.

As illustrated in FIG. 64, when a broadcast trigger occurs, the sensor module (360-3) may be activated, and the user may recognize that the sensor module (360-3) is activated through the sleep measurement confirmation UI (23-1-3). Specifically, the sleep data collection tool (1400-3) may be configured to provide a sleep measurement confirmation user interface to the user terminal (300-3) when a broadcast trigger occurs.

According to the present invention, the sleep data collection tool (1400-3) can detect an event through the activated sensor module (360-3), and the broadcast trigger of sleep measurement of the user terminal (300-3) can occur based on the event detected by the activated sensor module (360-3). For example, when wired charging of the user terminal (300-3) starts, wireless charging of the user terminal (300-3) starts, a change occurs in the accelerometer sensor, the display unit (331-3) of the user terminal (300-3) turns off, the battery status changes, another device such as a smart watch is connected due to a connection such as Bluetooth, or the user unlocks the user terminal (300-3), the sleep data collection tool (1400-3) can detect the above-mentioned event through the activated sensor module (360-3).

According to the present invention, the activated sensor module (360-3) can detect the above-described event, and when a broadcast trigger occurs, the activated sensor module (360-3) can perform activation for sleep measurement while providing a sleep measurement confirmation UI (23-1-3) from the time the broadcast trigger occurs for sleep measurement. In other words, the activated sensor module (360-3) of the present invention and the sensor module (360-3) that performs activation for sleep measurement have a difference between the two based on the time the broadcast trigger occurs, and it should be understood that the sensor module (360-3) additionally performs activation for sleep measurement based on the time the broadcast trigger occurs. That is, when a broadcast trigger occurs, the sleep data collection tool (1400-3) causes the sensor module (360-3) to additionally activate for sleep measurement.

According to one embodiment of the present invention, the sleep data collection tool (1400-3) is configured to transmit, when a broadcast trigger occurs, environmental sensing information collected by the sensor module (360-3) from the time of occurrence of the broadcast trigger to another device for sleep analysis. Specifically, the sleep data collection tool (1400-3) can transmit environmental sensing information collected by the sensor module (360-3) from the time of occurrence of the broadcast trigger to the server (200-3). In addition, the sleep data collection tool (1400-3) is configured to forcibly generate a broadcast trigger even in a low power mode of the user terminal (300-3).

According to one embodiment of the present invention, the sleep analysis trigger may include information on when sleep analysis starts. Specifically, the sleep analysis trigger may be a trigger for transmitting information on when sleep analysis starts to another device for sleep analysis. For example, the sleep data collection tool (1400-3) may start a broadcast according to the broadcast trigger to activate the sensor module (360-3), and then use the activated sensor module (360-3) to detect events such as when the user no longer touches the mobile phone, when movement is minimized, or when external noise decreases below a predetermined decibel. At this point, since the sensor module (360-3) is additionally performing activation for sleep measurement, it is possible to sense an event that is more sensitive than the event detection point for the broadcast trigger.

According to one embodiment of the present invention, the sleep data collection tool (1400-3) is configured to detect a first event through the activated sensor module (360-3), and a broadcast trigger of sleep measurement of the user terminal (300-3) may be configured to occur based on the detected first event. Here, the first event may be at least one of initiation of wired charging of the user terminal (300-3), initiation of wireless charging of the user terminal (300-3), detection of a change in an accelerometer sensor of the sensor module (360-3), switching of the display unit (331-3) of the user terminal (300-3) to an OFF state, a change in a battery status of the user terminal (300-3), connection of the user terminal (300-3) with another device such as a smart watch, and unlocking of the user terminal (300-3). In this case, the sleep data collection tool (1400-3) can additionally activate the sensor module (360-3) for sleep measurement when a broadcast trigger occurs, and can detect a second event through the additionally activated sensor module (360-3) for sleep measurement. At this time, the sleep analysis trigger of the sleep measurement of the user terminal (300-3) is configured to occur based on the detected second event, and the detected second event can be configured to be detected by a stimulus above a threshold by the sensor module (360-3). Here, the second event configured to be detected by a stimulus above the threshold by the sensor module (360-3) includes: initiation of wired charging of the user terminal (300), initiation of wireless charging of the user terminal (300-3), detection of a change in the accelerometer sensor of the sensor module (360-3), change in the battery status of the user terminal (300-3), connection of the user terminal (300-3) with another device such as a smart watch, unlocking of the user terminal (300-3), palm swiping for the user terminal (300-3), finger swiping for the user terminal (300-3), finger tapping for the user terminal (300-3), voice input for the user terminal (300-3), vibration detection of the user terminal (300-3), screen wheel scroll for the user terminal (300-3), pressing a specific user interface of the user terminal (300-3), pressing the display unit (331-3) for a predetermined time or longer, It may be at least one of moving two or more fingers simultaneously on the display unit (331-3), flipping the user terminal (300-3), activating the proximity sensor of the user terminal (300-3), detecting a temperature change in the user terminal (300-3), and detecting a change in illumination around the user terminal (300-3).

According to one embodiment of the present invention, when a broadcast trigger is generated, the sleep data collection tool (1400-3) can additionally activate the sensor module (360-3) for sleep measurement, and detect a second event through the additionally activated sensor module (360-3) for sleep measurement. At this time, the sleep analysis trigger of the sleep measurement of the user terminal (300-3) is configured to occur based on the detected second event, and the detected second event can be configured to be detected by a stimulus below the threshold by the sensor module (360-3). Specifically, the second event configured to be detected by the sensor module (360-3) by a stimulus below the threshold is at least one of the following: when there is no touch input to the display unit (331-3) of the user terminal (300-3) for a predetermined period of time or longer; when there is no movement of the accelerometer for the user terminal (300-3) for a predetermined period of time or longer; when noise around the user terminal (300-3) decreases and a sound below the threshold is detected for a predetermined period of time or longer; when illuminance around the user terminal (300-3) decreases and illuminance below the threshold is detected for a predetermined period of time or longer; when the locked state of the user terminal (300-3) is maintained for a predetermined period of time or longer; when data transmission and reception with a network connected to the user terminal (300-3) is not performed for a predetermined period of time or longer; when the battery consumption rate of the user terminal (300-3) falls below the threshold; and when a change in activity data of another device connected to the user terminal (300-3) is not detected for a predetermined period of time or longer.

According to one embodiment of the present invention, the sleep data collection tool (1400-3) or the time reservation tool (1200-3) can set the alarm to be accurately executed even in the low power mode by utilizing the function of forcing the occurrence of an alarm action or a broadcast trigger that indicates whether a broadcast trigger is activated even in the low power mode. In addition, when a predetermined time for a broadcast trigger is set through the reservation time setting user interface (18b-5-3), the sleep data collection tool (1400-3) or the time reservation tool (1200-3) can set the corresponding time as a broadcast trigger activation requirement. In addition, the sleep data collection tool (1400-3) or the time reservation tool (1200-3) can additionally activate the activated sensor module (360-3) for sleep measurement when a broadcast trigger is generated.

Fig. 58

FIG. 58 is a diagram showing a screen where an app requests and manages specific permissions in an Android system according to one embodiment of the present invention.

According to one embodiment of the present invention, the permission management tool (1100-3), the sleep data collection tool (1400-3), or the time scheduling tool (1200-3) may request the permission required to perform automatic sleep measurement in an Android app. The permission of the time-based alarm permission user interface (17a-1-3) illustrated in (a) of FIG. 58 may be an essential permission for the app to set an alarm at a specific time and start automatic sleep measurement based on it. In this case, the permission management tool (1100-3) may guide the user to understand and allow the permission, and may provide an intuitive switch interface and explanatory text for this purpose.

As shown in (b) of Fig. 58, the permission of the Overlay permission user interface (17b-1-3) may be a permission that supports an app to display an overlay UI on top of another app. Specifically, through such permission, a user can check the sleep measurement status in real time at the top of the screen or easily recognize that automatic measurement is in progress. The permission management tool (1100-3) can explain the necessity of such permission to the user, and if not permitted, can guide the user to go to the settings page and activate the permission.

In addition, the permission management tool (1100-3) of the present invention can start automatic sleep measurement based on various trigger events by utilizing the Broadcast Receiver function of Android and the function that can force the alarm operation that indicates whether the broadcast trigger is activated even in the low power mode. For example, the Broadcast Receiver receives events such as charging start, display OFF, and user motion detection, and when such a trigger is detected while acquiring the permission of the Overlay permission user interface (17b-1-3), the sensor module (360-3) of the user terminal can be activated to start recording immediately. Since this function of the permission management tool (1100-3) does not work if the permission of the Overlay permission user interface (17b-1-3) is not present, the permission management tool (1100-3) can provide the user with a warning message that "Automatic sleep measurement cannot be started if the permission is not present" when the permission is insufficient, and can retry the permission request or guide the user to the settings page.

According to one embodiment of the present invention, the permission management tool (1100-3) can detect when a user rejects a permission request or when a specific permission is released and provide a notification to the user. For example, when the permission of the time-based alarm permission user interface (17a-1-3) is disabled, the permission management tool (1100-3) can induce permission activation by displaying a message saying, "You must activate the alarm permission for automatic sleep measurement." In addition, the permission management tool (1100-3) can retry the permission request or provide the user with a detailed explanation about the functions of the app that may be restricted due to insufficient permission.

According to one embodiment of the present invention, the permission management tool (1100-3) can continuously check and manage the permission status to support the stable operation of the app. Specifically, even if the user suspends a specific permission or the permission is released due to an Android system update, it can detect this and guide the recovery procedure.

According to one embodiment of the present invention, the alarm permission activation switch interface (17a-2-3) located in the time-based alarm permission user interface (17a-1-3) illustrated in (a) of FIG. 58 may function as a component that allows a user to directly activate or deactivate the alarm permission. Specifically, the permission management tool (1100-3) may guide the user to easily grant the permission by utilizing the alarm permission activation switch interface (17a-2-3). Specifically, the alarm permission activation switch interface (17a-2-3) intuitively shows whether the alarm permission is activated, and when the user activates the switch, the Android system may immediately reflect the permission. If the user deactivates the alarm permission activation switch interface (17a-2-3), the permission management tool (1100-3) may display a message saying, “Automatic sleep measurement cannot be started when the alarm permission is deactivated,” explain the necessity of activating the permission, and induce reactivation.

According to one embodiment of the present invention, the permission management tool (1100-3) can continuously monitor the alarm permission activation switch interface (17a-2-3) to immediately notify the user when the permission status changes. For example, when the user deactivates the alarm permission, the permission management tool (1100-3) can notify the user of the status change by displaying a message saying, "The alarm permission has been turned off. Please activate it again for automatic sleep measurement."

Time Reservation Tool (1200-3)

According to one embodiment of the present invention, the time scheduling tool (1200-3) can utilize the function of forcing an alarm action indicating whether a broadcast trigger is activated even in the low power mode of Android in order to accurately execute a scheduled task without being affected by the battery optimization settings. Specifically, the time scheduling tool (1200-3) can provide an alarm even when the device enters the power saving mode by utilizing the function of forcing an alarm action indicating whether a broadcast trigger is activated even in the low power mode. For example, if a user sets sleep measurement to start automatically at a specific time, the time scheduling tool (1200-3) can execute a task at that time to activate the foreground service and the sensor module (360-3).

According to one embodiment of the present invention, the time reservation tool (1200-3) can check in real time whether the authority of the time-based alarm authority user interface (17a-1-3) is activated, detect when a scheduled task fails or an alarm is not executed, provide a notification to the user, and induce a reset or re-request process.

Foreground Service Management Tool (1300-3)

According to one embodiment of the present invention, the foreground service management tool (1300-3) can automatically recover a service that is unexpectedly interrupted, or dynamically manage the service so that it can be resumed. Specifically, the foreground service management tool (1300-3) can detect a task interruption situation and execute a service recovery procedure. For example, if a task is interrupted due to Android Doze mode or lack of system resources, it can detect this and immediately perform an operation to restart the service. In this case, the foreground service management tool (1300-3) can perform a role of enabling scheduled tasks to be executed even in low-power mode by using a function that can force an alarm operation indicating whether a broadcast trigger is activated even in low-power mode. In addition, when a user wants to suspend a task, the foreground service management tool (1300-3) can temporarily suspend the service status and, when a resume command is issued, can reactivate it so that the task can be continued.

According to one embodiment of the present invention, the foreground service management tool (1300-3) can monitor the service status in real time to check whether it is running normally, and can immediately provide a notification to the user when the service is abnormally terminated or stopped. For example, the service status can be conveyed to the user by displaying a message such as "Sleep measurement has been stopped. It is recovering." or "Sleep measurement has been resumed normally." In addition, when the service is completed, the foreground service management tool (1300-3) can deactivate the sensor module (360-3), store the collected data, and provide a task completion notification to the user.

Fig. 59

FIG. 59a is a diagram showing a main screen of a user interface for automatic sleep measurement through an app according to one embodiment of the present invention. FIG. 59b is a diagram showing an Auto Tracking setting screen for setting an automatic measurement schedule according to one embodiment of the present invention. FIG. 59c is a diagram showing a user interface for displaying results after automatic sleep measurement is completed according to one embodiment of the present invention.

Permissions Management Tool (1100-3)

As illustrated in FIG. 59a, the permission management tool (1100-3) can check in real time the status of permissions required for sleep measurement, such as the permission to activate the sensor module (360-3) (RECORD_AUDIO), the foreground service permission, and the Overlay permission permission, when the app is executed. When the user presses the button of the sleep measurement start user interface (18a-3-3) to start manual measurement, presses the button of the sleep measurement auto start user interface (18a-4-3) to set automatic measurement, or presses the button of the sleep report request user interface (18a-1-3), the permission management tool (1100-3) can check whether these permissions are activated and, if necessary, provide a notification to the user or guide them to a settings page. In addition, the battery saving mode ignore user interface (18a-2-3) is a button that guides the user to ignore the battery optimization settings of Android, and can play a role in resolving the problem of work restrictions that may occur in the battery saving mode. For example, the permission management tool (1100-3) may prompt the user to disable battery optimization settings by displaying a message such as "Battery optimization is enabled and background tasks may be limited."

As illustrated in FIG. 59b, the time reservation tool (1200-3) can provide an automatic measurement setting user interface (18b-1-3). Specifically, the time reservation tool (1200-3) can provide a user with an automatic measurement start time user interface (18b-2-3) and an automatic measurement end time user interface (18b-3-3) for sleep measurement. Accordingly, when the user sets the start time and the end time and then activates the automatic measurement execution switch user interface (18b-4-3) and presses the reservation time setting user interface (18b-5-3), the time reservation tool (1200-3) can register an alarm at the set time by utilizing Android AlarmManage. In addition, when the automatic measurement execution switch user interface (18b-4-3) is activated, the time reservation tool (1200-3) can also provide a user with an automatic measurement start time user interface (18b-2-3) and an automatic measurement end time user interface (18b-3-3) for sleep measurement. In this case, if the automatic measurement execution switch user interface (18b-4-3) is disabled, the time scheduling tool (1200-3) may not provide the user with the automatic measurement start time user interface (18b-2-3) and the automatic measurement end time user interface (18b-3-3) for sleep measurement.

As illustrated in FIG. 59c, the permission management tool (1100-3) can check the permission status required by the app so that the sleep data result screen can be displayed correctly. Specifically, if the sensor module (360-3) activation permission (RECORD_AUDIO), the foreground service permission, etc. are not activated, the permission management tool (1100-3) can detect this and provide a notification to the user. For example, the permission management tool (1100-3) can provide a message such as "The microphone permission is disabled, so sleep data collection may be stopped. Please activate the permission in the settings." In addition, the battery power saving mode ignore user interface (18c-2-3) can guide the user on a setting method so that work is not restricted due to the battery power saving mode. For example, the permission management tool (1100-3) can display a message such as "Battery optimization is enabled, so background work may be restricted. To ignore the setting, click the Ignore Battery Opt button." Additionally, the user can check the result data after the measurement is completed through the result data check user interface button (18c-1-3).

Time Reservation Tool (1200-3)

According to one embodiment of the present invention, the time reservation tool (1200-3) can activate a broadcast trigger by utilizing AlarmManager of Android based on the start time and end time set in the automatic measurement setting user interface (18b-1-3). Specifically, the time set by the user through the automatic measurement start time user interface (18b-2-3) is reserved through AlarmManager, and the time reservation tool (1200-3) can execute an alarm even in the power saving mode (Doze mode) by utilizing a function that can force an alarm operation even in the low power mode. For example, if the user sets the start time as "11:30 PM", the time reservation tool (1200-3) can determine the corresponding time as the first start point and start the broadcast.

According to one embodiment of the present invention, the automatic measurement execution switch user interface (18b-4-3) is a switch area that allows a user to activate or deactivate an automatic tracking function, and can control whether a broadcast trigger and a sleep analysis trigger are operated. When the user activates the automatic measurement execution switch user interface (18b-4), the time reservation tool (1200-3) can determine a first start point according to a set start time, and execute a broadcast trigger through AlarmManager. On the other hand, when the automatic measurement execution switch user interface (18b-4-3) is deactivated, a reserved schedule is stopped, and the user can reactivate it through the reservation time setting user interface (18b-5-3).

According to one embodiment of the present invention, the time reservation tool (1200-3) can detect a logical conflict between the set start time and end time (e.g., when the end time is earlier than the start time). In this case, a warning message such as "The end time is earlier than the start time. Please set the time again." can be displayed to guide the user to modify the schedule. In addition, when the saved schedule conflicts with an existing reservation, a message such as "The entered time overlaps with an existing reservation time. Please set the time again." can be provided to prevent the conflict.

According to one embodiment of the present invention, the time reservation tool (1200-3) can trigger an alarm by utilizing the AlarmManager of Android when the end time set by the user is reached. Specifically, when the alarm is executed, the app automatically terminates the sleep measurement task, and during this termination process, the time reservation tool (1200-3) can record that the measurement has been completed normally. For example, the time reservation tool (1200-3) can store the time when the measurement has ended, check whether the sleep data collection has been completed, and reflect the measurement completion status in the internal database of the app. The information recorded in this way can be utilized to provide detailed data such as sleep start and end times to the user on a subsequent result screen.

According to one embodiment of the present invention, the time reservation tool (1200-3) can perform management so that the user can maintain or modify the schedule previously set. In addition, even after the sleep measurement is completed, the time reservation tool (1200-3) can retrieve previously set schedule information from the database and display it on the user interface when the user re-executes the app. Accordingly, the user can repeatedly use the same settings without having to input a new schedule. For example, when the user wants to modify or delete the schedule, the time reservation tool (1200-3) can process such changes and perform a database update.

According to one embodiment of the present invention, the time reservation tool (1200-3) can be linked with a button of the measurement start user interface (18c-3-3) to enable a user to set a new schedule. For example, when a user attempts to set a new automatic measurement schedule, the time reservation tool (1200-3) can check for this and provide a warning so that it does not conflict with an existing schedule.

Sleep Data Collection Tool (1400-3)

As described above, the sleep data collection tool (1400-3) can detect sounds (e.g., snoring, environmental noise, etc.) occurring during the user's sleep in real time and collect data by activating the sensor module (360-3) of the present invention after the user clicks the button of the sleep measurement start user interface (18a-3-3). Here, the sleep data collection tool (1400-3) is linked with the permission management tool (1100-3) to check whether the sensor module (360-3) activation permission (RECORD_AUDIO) is allowed, and if there is a missing permission, it can provide a notification to the user or perform a permission request.

According to one embodiment of the present invention, the sleep data collection tool (1400-3) can perform filtering and preprocessing processes to filter out unnecessary background noises and identify valid data, in addition to simply recording data. For example, the sleep data collection tool (1400-3) can remove low-level noises and identify and store user's sleep noises, such as snoring or tossing and turning, as main data. In addition, the sleep data collection tool (1400-3) can be linked with a foreground service to manage background tasks so that they are not interrupted, and can be set so that data collection is not interrupted by Android's battery optimization or power saving mode. For example, when the user activates a button of the battery power saving mode override user interface (18a-2-3), the sleep data collection tool (1400-3) can ignore the battery optimization settings.

Fig. 60

FIG. 60 is a diagram for explaining a process for setting up automation of sleep measurement using a Shortcuts app in an iOS environment according to one embodiment of the present invention.

Permissions Management Tool (1100-3)

According to one embodiment of the present invention, the permission management tool (1100-3) can support smooth linkage between a shortcut app and a sleep measurement app in an iOS environment, and can perform a role of checking and managing necessary permissions. Specifically, in order to automate sleep measurement through a shortcut app, the sleep measurement app must implement App Intent to be linked with the shortcut app, and for this purpose, sensor module (360-3) activation permission (RECORD_AUDIO) and background execution permission may be required. At this time, the permission management tool (1100-3) can check whether these permissions are activated when a user attempts to set or execute a sleep measurement task in the shortcut app.

According to one embodiment of the present invention, if a required permission is missing, the permission management tool (1100-3) can detect this and provide a notification to the user. For example, the permission management tool (1100-3) can guide the user through a message such as "The sensor module (360-3) activation permission is disabled, so sleep measurement cannot be started. Please activate the permission in the settings." In addition, the permission management tool (1100-3) can provide the user with an option to directly move to the settings screen or explain in detail the permission activation method so that the permission setting is not complicated.

According to one embodiment of the present invention, when setting or executing a sleep measurement task in a shortcut app, the permission management tool (1100-3) can continuously monitor the permission status according to the user's settings. Specifically, when the permission is suspended or deactivated during automation execution, the permission management tool (1100-3) can immediately detect this, provide a notification to the user, and take measures to prevent the task from being suspended. In addition, since there may be restrictions such as battery optimization or background task restrictions in the iOS environment, the permission management tool (1100-3) can support user settings to overcome these restrictions. For example, the permission management tool (1100-3) can provide a message such as "The battery optimization setting is activated, which may cause sleep measurement to be suspended. Please disable battery optimization in the settings."

Time Reservation Tool (1200-3)

According to one embodiment of the present invention, the time reservation tool (1200-3) can perform management so that sleep measurement can start and end at an accurate time according to the automation schedule set in the shortcut app. When the user sets a specific time (e.g., 11 p.m.) or a trigger condition (e.g., sleep mode activation) by pressing the button of the New Automation user interface (19b-1-3) in (b) of Fig. 60, the time reservation tool (1200-3) can receive such information and perform coordination so that the task of the sleep measurement app is automatically executed at the set time. In this case, the time reservation tool (1200-3) can transmit the time reservation information to the sleep measurement app through linkage with the shortcut app and store and manage the corresponding schedule internally in the app.

Sleep Data Collection Tool (1400-3)

According to one embodiment of the present invention, the sleep data collection tool (1400-3) can perform a role of collecting data in real time by activating the sensor module (360-3) when the automation of the shortcut app is executed. Specifically, when the sleep measurement app is executed according to the conditions set in the shortcut app (e.g., specific time, Wi-Fi connection, etc.), the sleep data collection tool (1400-3) can detect and store the user's sleeping noise (snoring, movement, etc.) in real time. In this case, the collected data can be structured and organized into information such as start time, end time, and noise event.

Fig. 61

FIG. 61 is a diagram for explaining a process of setting up automation in an iOS Shortcuts App according to one embodiment of the present invention.

Permissions Management Tool (1100-3)

Figure 61(a) shows the initial steps for setting up a new automation by selecting the New Blank Automation user interface (20a-1) in the Shortcuts app.

According to one embodiment of the present invention, when creating an automation based on the New Blank Automation user interface (20a-1-3) or another trigger that the user can select, an App Intent must be implemented so that a specific function of the sleep measurement app (e.g., start sleep measurement) is exposed to the shortcut app. In this case, the permission management tool (1100-3) can check whether the App Intent is set correctly and whether the shortcut app and the sleep measurement app can be linked. For example, if the sleep measurement app is not set as a trigger in the shortcut app, the permission management tool (1100-3) can request the user to update the app or confirm the permission. Specifically, when the user selects the New Blank Automation user interface (20a-1-3) to set up a new automation and tries to proceed to the next step, the permission management tool (1100-3) can check whether the sensor module (360) activation permission (RECORD_AUDIO) and the background execution permission are activated. For example, the permission management tool (1100-3) may provide the user with a message such as "The sensor module (360-3) activation permission is disabled, which may limit automation settings. Please activate the permission on the settings screen." if the permission is not activated, and may provide an option to navigate to the settings page.

Fig. 61(b) shows various trigger option screens that can set automation conditions of the shortcut app. Specifically, conditions such as the Time of Day user interface (20b-1-3), the Sleep user interface (20b-3-3), or the Alarm user interface (20b-2-3) allow the user to set sleep measurement to be executed in specific situations. Specifically, when the user selects a specific trigger (e.g., "Sleep" or "Time of Day") and attempts to set automation, the permission management tool (1100-3) can check whether all permissions required for executing the automation are maintained. If the permission is suspended or not activated, the permission management tool (1100-3) can detect this and provide a notification to the user. For example, the permission management tool (1100-3) can provide a message such as "Background execution permission is required. If the permission is not present, the automation may not be executed properly."

In (a) and (b) of FIG. 61, the permission management tool (1100-3) can continuously check the permission status and immediately notify the user if a problem occurs, since the sleep measurement task may fail if the activation permission of the sensor module (360-3) is removed or the background execution is stopped while the automation is running. In addition, the permission management tool (1100-3) can guide the user to a solution in a simple and intuitive manner if the permission is disabled. For example, the permission management tool (1100-3) can provide a message such as "You need to enable the background execution permission to set the Wi-Fi connection conditions. Would you like to move to the settings screen?"

Time Reservation Tool (1200-3)

As illustrated in (a) of FIG. 61, when a user selects the New Blank Automation user interface (20a-1-3), the user can move to a step where the user can set trigger conditions (e.g., specific time, sleep mode, etc.) of the new automation. In this case, the time scheduling tool (1200-3) can manage alarm scheduling so that the sleep measurement app can be executed at a specified time based on the time information entered by the user. For example, if the user has set "start sleep measurement automatically at 10 p.m.," the time scheduling tool (1200-3) can schedule an alarm at that time. In addition, when the user selects the New Blank Automation user interface (20a-1-3) and inputs a new schedule, the time scheduling tool (1200-3) can review the previously set schedule information. In this case, if there is a possibility that the schedules overlap or conflict, the time scheduling tool (1200-3) can detect this and provide a warning to the user. For example, the time scheduling tool (1200-3) might provide the message "A sleep measurement is already set for 10 PM. Would you like to add a new schedule?"

As illustrated in (b) of FIG. 61, when a user selects the Time of Day user interface (20b-1-3) to set a specific time or selects the Sleep user interface (20b-3-3) to set a sleep mode activation as a trigger, the time scheduling tool (1200-3) can link this with the sleep measurement app. For example, when 10 p.m. is set in the Time of Day user interface (20b-1-3), the time scheduling tool (1200-3) can schedule an alarm so that the sleep measurement app is automatically executed at that time. In addition, when the user adds a new condition (e.g., specific time, sleep mode), the time scheduling tool (1200-3) can store and manage this in the internal database of the app. For example, after setting 10 PM in the Time of Day user interface (20b-1-3), when trying to add a condition with the same time overlapping in the mode of the Sleep user interface (20b-3-3), a warning message could be provided saying, "Two conditions may conflict in the same time zone. Please check and change the setting."

As shown in (b) of Fig. 61, a condition such as the Sleep user interface (20b-3-3) can be set to be automatically executed when the sleep mode of iOS is activated. At this time, the time reservation tool (1200-3) can detect this trigger condition. For example, if the user sets the sleep mode activation as a trigger, the time reservation tool (1200-3) can operate to send an alarm signal to the sleep measurement app as soon as the sleep mode is turned on to automatically start the measurement.

Sleep Data Collection Tool (1400-3)

As illustrated in (a) of FIG. 61, when a user selects the New Blank Automation user interface (20a-1-3) to set up automation based on specific conditions (e.g., specific time, sleep mode activation, etc.), the sleep data collection tool (1400-3) can prepare to collect data when the conditions are met. That is, if the user sets "start sleep measurement at 10 p.m.," the sleep data collection tool (1400-3) can automatically activate the microphone.

As illustrated in (b) of FIG. 61, when a user selects the Time of Day user interface (20b-1-3) to set a specific time (e.g., 10 p.m.) or selects the Sleep user interface (20b-3-3) to set the sleep mode activation as a trigger, the sleep data collection tool (1400-3) can automatically activate the microphone to collect data when the corresponding condition is met. For example, when the condition of the Sleep user interface (20b-3-3) is set and the sleep mode is activated, the data collection tool (1400-3) can record snoring sounds, ambient noise events, etc. in real time. Accordingly, the collected data can be organized in a structured form. For example, the data can be organized with information such as start time, end time, snoring duration, and occurrence time of the noise event. In this case, the organized data is stored in an internal database of the app and can be utilized thereafter in a result screen or a report generation module.

Fig. 62

FIG. 62 is a diagram illustrating a process by which a user adds a new action (e.g., starting sleep measurement) to an automated task to be executed at a specific time in a shortcut app according to one embodiment of the present invention.

Permissions Management Tool (1100-3)

(a) of Fig. 62

As illustrated in (a) of FIG. 62, when a user presses a button on the Add Action user interface (21a-2-3) to add a new action (e.g., launching a sleep measurement app), the permission management tool (1100-3) can check whether the permissions required for executing the action are properly activated. For example, to add a “sleep data collection” task, the sensor module (360-3) activation permission (RECORD_AUDIO) and background execution permission are required, and the permission management tool (1100-3) can check whether these permissions are activated. In this case, the permission management tool (1100-3) can scan the app settings and report the permission status to the user. In addition, the permission management tool (1100-3) can detect if the required permission is deactivated and provide a notification to the user. For example, when a user tries to add an action to the Start Sleep user interface (21c-1-3), if the sensor module (360-3) activation permission is disabled, the permission management tool (1100-3) may provide the message "The automation may not run because the microphone permission is disabled. Please enable the permission in the settings screen."

(b) of Fig. 62

As illustrated in (b) of Fig. 62, the permission management tool (1100-3) can check and manage the required permission status so that the automated task set by the user can be immediately executed as an option of the Run Immediately user interface (21b-6-3). In Fig. 62 (a), if the user executes the sleep measurement app through the Open App user interface (21a-3-3) at a specific time (e.g., 10 p.m.) and sets the "Sleep Measurement Start App Intent" to be registered, the permission management tool (1100-3) can check whether all the permissions required for the corresponding task are activated. Specifically, the permission management tool (1100-3) can check the status of the sensor module (360-3) activation permission (RECORD_AUDIO) and the background execution permission required for the sleep data collection task, and can notify the user if there is any missing permission. For example, the permission management tool (1100-3) may provide a notification such as "Microphone permission is required to start sleep measurement. Please enable the permission in Settings."

According to one embodiment of the present invention, the permission management tool (1100-3) can check the linkage with the App Intent and perform management so that the work is not interrupted due to iOS policies such as background execution restrictions. In addition, the permission management tool (1100-3) can provide guidance to the user that simplifies the permission setting process or provide an option to go directly to the settings page.

(c) of Fig. 62

In (c) of Fig. 62, the permission management tool (1100-3) can check and manage the necessary permissions so that the execution of the sleep measurement app, which is a task set by the user, and the task of the Start Sleep user interface (21c-1-3) can be accurately executed. Specifically, the permission management tool (1100-3) can check whether the sensor module (360-3) activation permission (RECORD_AUDIO) and the background execution permission required for the task of the Start Sleep user interface (21c-1-3) are activated. If the necessary permission is disabled, the permission management tool (1100-3) can notify the user of the problem through a message such as "Microphone permission is required. Please activate the permission on the settings screen." and provide an option to move directly to the settings screen.

According to one embodiment of the present invention, since the task of the Start Sleep user interface (21c-1-3) is executed through the App Intent of the sleep measurement app, the permission management tool (1100-3) can check whether the App Intent is properly exposed and properly linked with the shortcut app. In this case, if the App Intent is not properly set or a problem occurs due to background execution restrictions, the permission management tool (1100-3) can guide the user to a solution. In addition, the permission management tool (1100-3) continuously monitors so that the necessary permissions are not interrupted even during task execution, and can immediately detect and provide a notification to the user when the microphone usage permission or background execution permission is restricted. In addition, if the tasks of the Open App user interface (21a-3-3) and the Start Sleep user interface (21c-1-3) are executed in succession, the permission management tool (1100-3) can individually check and manage the permissions required for each task to prevent problems from occurring during the execution process.

Time Reservation Tool (1200-3)

(a) of Fig. 21

As illustrated in (a) of FIG. 62, since the user has set a time-based condition such as At 10:00 PM, daily user interface (21a-1-3), the time scheduling tool (1200-3) can receive such trigger information and schedule an alarm so that a task can be executed exactly at the set time. For example, if the user has set a condition such as At 10:00 PM, daily user interface (21a-1-3), the time scheduling tool (1200-3) can generate a notification at the corresponding time by linking with the alarm manager of the system or the Notification Center of iOS. In addition, the time scheduling tool (1200-3) can check whether the trigger condition is set correctly and can notify the user of an incorrect setting (e.g., overlapping time settings). Specifically, the time scheduling tool (1200-3) can analyze the frequency set for the trigger (e.g., Daily, Weekly, Monthly) to systematically manage the alarm schedule. In addition, the time scheduling tool (1200-3) can detect if another task is scheduled for the already set time zone and prevent a conflict. For example, if a sleep data collection task is already scheduled for "10:00 PM" and another task is added at the same time, the time scheduling tool (1200-3) can provide a warning message saying, "The set time overlaps. Please modify the existing task or select a different time."

(b) of Fig. 62

As illustrated in (b) of FIG. 62, the time reservation tool (1200-3) can perform a role of managing the task so that it is executed accurately according to the time and repetition cycle set by the user. Specifically, when the user selects the Time of Day user interface (21b-1-3) and inputs a specific time (e.g., 10 p.m.), the time reservation tool (1200-3) can set a trigger so that the task can be executed at the corresponding time by linking with the system's alarm manager or iOS Notification Center based on the corresponding information. In addition, when the user selects the Daily user interface (21b-2-3), the schedule is managed so that the task can be repeated every day, and when a repetition cycle such as the Weekly user interface (21b-3-3) or the Monthly user interface (21b-4-3) is selected, the task can be scheduled according to the cycle.

According to one embodiment of the present invention, when a user selects an option of the Run Immediately user interface (21b-6-3), the time reservation tool (1200-3) can apply a setting so that a task can be automatically executed without user confirmation when a set time has arrived. If the Run After Confirmation user interface (21b-5-3) is selected, a notification can be sent to the user at the set time and processing can be performed so that the task can be executed after confirmation. In addition, the time reservation tool (1200-3) can verify whether the set conditions have been correctly applied before executing the task, and can provide a notification to the user if there is an incorrect input (e.g., when the set time is in the past compared to the current time).

(c) of Fig. 62

In (c) of Fig. 62, the time reservation tool (1200-3) can perform a role of managing a task to be executed at a specific time (e.g., 10 PM) set by the user. Specifically, based on the condition set by the user as At 10:00 PM, daily user interface (21a-1-3), the time reservation tool (1200-3) can set a trigger in conjunction with the alarm manager of the automatic measurement system (1000-3) or the Notification Center of iOS so that the task can be executed at the corresponding time. Through this, the sleep measurement app can be opened at 10 PM every night and the reservation can be completed so that the task of the Start Sleep user interface (21c-1-3) can be executed. In addition, the time reservation tool can manage a schedule so that the task is executed periodically by processing a repeat cycle (e.g., Daily user interface (21b-2-3)) set by the user. If another repeating cycle, such as Weekly user interface (21b-3-3) or Monthly user interface (21b-4-3), is set, the time scheduling tool (1200-3) can adjust the task to be repeatedly executed according to the cycle. In addition, if the task execution time or repeating cycle is changed, the time scheduling tool (1200-3) can delete the existing reservation information and re-set the trigger according to the newly changed conditions.

According to one embodiment of the present invention, the time reservation tool (1200-3) can play a role in preventing conflicts between schedules. Specifically, the time reservation tool (1200-3) can detect if another task is already scheduled for the same time zone (e.g., 10 p.m.) or if there is a duplicate repeat setting, and can provide a warning message to the user. For example, a notification such as "There is already a task scheduled for 10 p.m. To add a new task, please modify the existing task or select a different time" can be provided to prevent conflicts between tasks.

Fig. 63

FIG. 63 is a drawing showing a screen for setting a state in which a sleep measurement termination task according to one embodiment of the present invention can be automatically registered and executed via App Intent.

Permissions Management Tool (1100-3)

(a) of Fig. 63

As illustrated in (a) of FIG. 63, the permission management tool (1100-3) can check and manage the necessary permissions so that the task of the Stop Sleep user interface (22a-1-3) can be normally registered and executed as automation. Specifically, the sleep measurement termination task is executed through App Intent, and for this purpose, background execution permission and data access permission are required. In this case, the permission management tool (1100-3) can check whether these permissions are activated, and if necessary, provide a notification to the user to guide the settings. In addition, the permission management tool (1100-3) can manage the permissions related to the automation execution modal that appears when the user locks the mobile phone. Specifically, when the user selects “Accept” in the modal, the permission management tool (1100-3) can maintain the necessary permissions so that they are not interrupted during task execution. In addition, if background execution permission or data access permission is restricted or suspended during task execution, the permission management tool (1100-3) can detect this and provide a notification to the user, such as "The permission required for automation execution has been suspended. Please check the permission on the settings screen."

According to the present invention, the permission management tool (1100-3) can check whether the App Intent of the Stop Sleep user interface (22a-1-3) is properly linked to the shortcut app and can be executed. Specifically, if the App Intent is not exposed or there is a problem in the linkage status, the permission management tool (1100-3) can detect this and provide the user with a notification and a solution. For example, the permission management tool (1100-3) can prevent problems in advance so that the task can be stably executed through linkage with the App Intent even in a situation where background execution may be restricted.

(b) of Fig. 63

In (b) of Fig. 63, the permission management tool (1100-3) can check and manage the necessary permissions so that the task (termination of sleep measurement) of the Stop Sleep user interface (22a-1-3) is normally executed. Specifically, the permission management tool (1100-3) can execute such a task through App Intent. Specifically, the permission management tool (1100-3) can check whether the background execution permission, data access permission, and microphone usage permission required for the task of the Stop Sleep user interface (22a-1-3) are properly activated. If a permission is missing, the permission management tool (1100-3) can notify the user of this and guide them to resolve the problem by providing an option such as "Go to the settings screen." For example, the permission management tool (1100-3) can provide a message such as "Background execution permission is required to terminate sleep measurement. Please activate it in the settings."

According to one embodiment of the present invention, when the mobile phone is in a locked state, a modal asking whether to execute automation may be displayed, and the permission management tool (1100-3) may maintain the necessary permissions so that the task can be executed normally even after the user selects "Accept." Specifically, the permission management tool (1100-3) may detect and prevent permission interruption that may occur during this process, and may provide a notification such as "The background execution permission was interrupted while the task was being executed. Please check the settings." If a problem occurs during execution, the permission management tool (1100-3) may check whether the task of the Stop Sleep user interface (22a-1) can be properly called by linking it to the shortcut app as an App Intent. Specifically, if the App Intent does not operate normally or a problem occurs in the linkage status, the permission management tool (1100-3) may notify the user of this and guide them on how to resolve the problem. For example, the permission management tool (1100-3) may provide a message such as "The sleep measurement end task may not be executed. Please check the settings."

According to one embodiment of the present invention, while an automated task is being executed, the permission management tool (1100-3) can continuously monitor the permission status to ensure the continuity of the task. Specifically, the permission management tool (1100-3) can manage the permission so that it is not interrupted due to the battery saving mode of iOS or system constraints, and can provide a notification to the user so that the user can take appropriate action when necessary.

(c) of Fig. 63

As illustrated in (c) of FIG. 63, the permission management tool (1100-3) can check whether background execution permission, microphone usage permission, data access permission, etc. required for executing the tasks of the Start Sleep user interface (21c-1-3) and the Stop Sleep user interface (22a-1-3) are activated. Specifically, if the permission management tool (1100-3) detects a lack of permission or a restricted state before each task is executed, it can notify the user of this and guide the user to move to the settings screen to resolve the problem. For example, the permission management tool (1100-3) can provide a message such as "Background execution permission is required to execute the sleep measurement end task. Please activate it in the settings."

According to one embodiment of the present invention, when the mobile phone is in a locked state, iOS may display a modal asking whether to execute automation. In this case, the permission management tool (1100-3) may maintain the necessary permissions so that the task can be executed normally even after the user selects "Accept." In addition, the permission management tool (1100-3) may check the status so that the two automation tasks can be executed in conjunction with the shortcut app through the App Intent. Specifically, if the App Intent is not properly called from the shortcut app or there is a problem in the conjunction status, the permission management tool (1100-3) may detect this and guide the user on how to solve the problem. For example, a notification such as "The sleep measurement end task may not be executed. Please check the settings" may be provided.

Time Reservation Tool (1200-3)

(a) of Fig. 63

In (a) of Fig. 63, when a user attempts to add a task of the Stop Sleep user interface (22a-1-3) to automation, the time scheduling tool (1200-3) can prepare a trigger so that the task can be executed at the set time. For example, if the user sets 6:00 AM as the sleep end time, the time scheduling tool (1200-3) can schedule a trigger to be executed at that time in conjunction with an alarm manager (such as iOS Notification Center). In addition, the time scheduling tool (1200-3) can check whether the set time is not earlier than the current time or whether there is no other conflicting schedule. Specifically, if the user sets an incorrect time, the time scheduling tool (1200-3) can detect it and provide guidance to correct it.

(b) of Fig. 63

In (b) of Fig. 63, the time scheduling tool (1200-3) can schedule an alarm to execute a task of the Stop Sleep user interface (22a-1-3) according to a set condition (6:00 AM). For example, the time scheduling tool (1200-3) can set a notification and task execution trigger in conjunction with the iOS system so that the automation is executed at 6:00 AM. In addition, based on the repeat condition set by the user in the Daily user interface (22b-1-3), the time scheduling tool can set a repeat schedule so that the task is executed at the same time (6:00 AM) every day. Accordingly, the user can automatically end sleep data at the same time every day, and the task can be continuously executed without a separate setting change. In addition, the time scheduling tool (1200-3) can check whether there is another automated task set for the same time zone. For example, if there is a schedule to execute the same task in another app at 6:00 AM, the time scheduling tool (1200-3) can detect this and notify the user of the possibility of a conflict and provide an opportunity to correct it.

(c) of Fig. 63

In (c) of Fig. 63, two automated tasks are set. Specifically, automated tasks of a Start Sleep task execution user interface (22c-2-3) at 10 p.m. and a Stop Sleep task execution user interface (22c-1-3) at 6 a.m. can be set. Specifically, the time scheduling tool (1200-3) can perform management so that the two tasks can be executed independently. For example, after the task of the Start Sleep task execution user interface (22c-2-3) at 10 p.m. is executed, adjustments can be performed so that no resource usage conflict occurs between the tasks until the task of the Stop Sleep task execution user interface (22c-1-3) is executed at 6 a.m. In addition, the time scheduling tool (1200-3) can immediately update the schedule status when a user adds or deletes a task. For example, when a user additionally registers another task, the time scheduling tool (1200-3) can check for conflicts by comparing it with the existing schedule and adjust the task priorities. Additionally, the time scheduling tool (1200-3) can monitor the trigger status so that the set task is executed at the exact specified time. Specifically, the time scheduling tool (1200-3) can maintain notifications and task execution triggers so that task execution is not interrupted due to iOS's power saving mode or system restrictions.

Artificial intelligence model based on Transformer model

Fig. 65

Figure 65 is a diagram explaining the structure of the Transformer model that is the basis of the Large Language Model.

In the present invention, the content-generating artificial intelligence can generate content based on one or more data arrays regarding the user's sleep. The content-generating artificial intelligence can be, but is not limited to, a large language model. For example, the large language model can be, but is not limited to, a large language model based on a Generative Pretrained Transformer (GPT) based on a Transformer model or a Bidirectional Encoder Representations from Transformers (BERT).

In one embodiment of the present invention, the content-generating artificial intelligence may be a generative artificial intelligence based on a Transformer model. As illustrated in FIG. 65, the Transformer model may be composed of an encoder and a decoder, and may utilize attention, self-attention, and multi-head attention mechanisms.

As illustrated in FIG. 65, a Trnasformer model according to an embodiment of the present invention may include an encoder. Specifically, the encoder may be composed of six identical layers, and each layer may be composed of two sublayers. Here, the first sublayer may be a multi-head self-attention mechanism layer, and the second sublayer may be a position-wise fully connected feed-forward network. This allows an input data array to interact with another data array, thereby enabling a better understanding of the structure and context of a sentence.

As illustrated in FIG. 65, a Transformer model according to an embodiment of the present invention may include a decoder. Specifically, the decoder may be composed of six identical layers, and each layer may be composed of three sublayers. Here, the first sublayer may be a multi-head self-attention mechanism, the second sublayer may be a position-wise fully connected feed-forward network, and the third sublayer may be a multi-head attention for the output of the encoder. Here, the first sublayer undergoes a masking process so as not to pay attention to a position after the current position in order not to refer to information after the current position in order to predict the next token at each time point.

As illustrated in FIG. 65, the Trnasformer model according to an embodiment of the present invention may include an attention mechanism. Specifically, a query and a series of key, value pairs may be mapped to output. The output is calculated as a weighted sum of values, and the weight assigned to each value may be calculated by a compatibility function of the query and the corresponding key. The attention mechanism may be used to train the sequence-to-sequnce model to pay more attention to important parts.

As illustrated in FIG. 65, a Transformer model according to an embodiment of the present invention may include a self-attention mechanism and a multi-head attention mechanism. Specifically, the self-attention mechanism may assign different weights to tokens at different positions and perform weight summation in order to pay attention to all tokens at different positions in an input sequence. Specifically, the multi-head attention mechanism may use multiple self-attention mechanism layers in parallel and use different weights for each in order to learn and consider various features simultaneously.

In one embodiment of the present invention, the content-generating artificial intelligence may be a large-scale language model, and the large-scale language model may be a GPT (Generative Pretrained Transformer)-based model. Specifically, the GPT model may utilize the decoder part of the Transformer. For example, by utilizing the decoder, learning and output of a sentence may be processed in a manner of predicting the next word in a given context, and the next word may be predicted based on each word.

In one embodiment of the present invention, the content-generating artificial intelligence may be a large-scale language model, and the large-scale language model may be a BERT (Bidirectional Encoder Representations from Transformers)-based model. Specifically, the BERT model may utilize the encoder part of the Transformer. For example, it may be trained in the MLM (Masked Language Model) method, and the MLM method is a learning method that masks some words and predicts the words. In this case, since the entire sentence is processed at once, information from both sides of the sentence can be obtained, and contextual information can be considered in all parts of the sentence at the same time, and the meaning of the sentence can be understood more accurately through this.

Artificial intelligence model based on DIFFUSION model

FIG. 66 is a diagram for explaining an inverse diffusion model (Inverter Model) of a DIFFUSION model in a content-generating artificial intelligence according to an embodiment of the present invention.

Referring to FIG. 66, the content-generating artificial intelligence according to one embodiment of the present invention may be a generative artificial intelligence based on a DIFFUSION model. The DIFFUSION model may be used to generate various types of data, such as images, voices, and texts, and may be trained to generate results close to actual data while gradually reducing noise in the data. In this case, the trained DIFFUSION model may operate in a manner of gradually spreading a given data array.

According to one embodiment of the present invention, the content-generating artificial intelligence may be a generative artificial intelligence based on a DIFFUSION model. The DIFFUSION model may be composed of a noise model (Nosie Model) and an inverted diffusion model (Inverter Model). Specifically, the noise model (Noise Model) is a function that adds noise to initial data, and this function generates noise that the generator uses as a starting point and reduces the noise over time. That is, the noise model function initially has a very large amount of noise, but can be adjusted to gradually add only fine noise to the data.

According to one embodiment of the present invention, the content-generating artificial intelligence may be a generative artificial intelligence based on a DIFFUSION model. The DIFFUSION model may be composed of a noise model and an inverted diffusion model. Specifically, the inverted diffusion model (Inverter Model) is a function that restores a generated result to initial data, and this function can perform an operation to remove noise that returns the result generated by the generator to the initial data as input.

According to one embodiment of the present invention, a content-generating artificial intelligence may be a generative artificial intelligence based on a DIFFUSION model. The DIFFUSION model may be composed of a noise model (Nosie Model) and an inverted diffusion model (Inverter Model). Specifically, the DIFFUSION model alternately uses the noise model and the inverted diffusion model, and can improve the data and noise addition and removal processes in the process of generating a desired output. For example, the noise used in the DIFFUSION model may use random noise in the early stage of learning, but may generate output similar to actual data through training. The DIFFUSION model may be used in various fields such as data generation, image restoration, and image editing through learning how to add and remove noise.

GAN-based artificial intelligence model

FIG. 67 is a diagram for explaining a generator and a discriminator of a GAN (Generative Adversarial Network) in a content-generating artificial intelligence according to one embodiment of the present invention.

Referring to FIG. 67, the content-generating artificial intelligence according to one embodiment of the present invention may be a generative artificial intelligence based on a GAN (Generative Adversarial Network) model. The GAN model may include a generator and a discriminator. The GAN model may be used to generate various types of data such as images, voices, and texts, and may be trained to generate outputs similar to actual data.

According to an embodiment of the present invention, a content-generating artificial intelligence may be a generative artificial intelligence based on a GAN (Generative Adversarial Network) model. The GAN model may include a generator. Specifically, the generator may receive a given random vector noise as input and generate fake data similar to real data, and may initially generate random outputs and improve its ability to be difficult to distinguish from real data during the training process. Specifically, the generator may be trained to better deceive the discriminator.

According to an embodiment of the present invention, a content-generating artificial intelligence may be a generative artificial intelligence based on a Generative Adversarial Network (GAN) model. The GAN model may include a discriminator. Specifically, the discriminator may serve as a classifier that distinguishes between fake data generated by the generator and real data. Specifically, the discriminator may be trained to distinguish between fake data generated by the generator and real data to make accurate predictions. Specifically, the discriminator may be trained to better distinguish between fake data generated by the generator and real data.

The content-generating artificial intelligence according to one embodiment of the present invention may be a combination of one or more of a large-scale language model, a DIFFUSION model, and a GAN model, but is not limited thereto.

The content-generating artificial intelligence according to one embodiment of the present invention may be a generative artificial intelligence based on a combination of one or more of a large-scale language model, a DIFFUSION model, and a GAN model, but is not limited thereto.

The above-described content-generating artificial intelligence is a generative artificial intelligence model according to an embodiment of the present invention, and is not limited thereto.

Mamba-based artificial intelligence model

Meanwhile, according to one embodiment of the present invention, a deep learning architecture based on Mamba can be utilized. The Mamba model is a model that focuses on efficiently processing long sequence data by applying a state space model (SSM) used in control theory and signal processing to deep learning. The Mamba model according to one embodiment of the present invention can be a model developed to enable efficient processing by filtering out unnecessary data and focusing on important information even in long sequence data by introducing a mechanism that selectively processes information according to the importance of input data. In addition, the Mamba model according to one embodiment of the present invention can maintain high efficiency and fast processing speed even in long sequences since the computational cost increases linearly with the sequence length. The Mamba model according to one embodiment of the present invention shows excellent performance in various fields such as language processing, audio analysis, and genomics, and can exhibit excellent performance even in long sequence lengths (e.g., 1 million tokens).

Agent configuration in the content delivery model

According to one embodiment of the present invention, the artificial intelligence model is a reasoning or inference model and can perform content providing operations according to the present invention by utilizing the agent configuration.

In an artificial intelligence model, especially a natural language-based artificial intelligence model utilized as a reasoning engine or inference engine, an agent refers to a system or component of software designed to perform tasks autonomously using language understanding and generation functions. An agent can interpret user input, respond to questions, and perform tasks based on programmed goals or learned behaviors. In an artificial intelligence model system based on natural language processing, prompts and tools are components of an agent that guide the output of the model, improve the response, and activate specific functions to achieve the purpose of the present invention. Meanwhile, an agent configuration according to an embodiment of the present invention can be written in various languages such as English, Chinese, Japanese, Korean, and Hindi.

Prompt in the content delivery model

A prompt is a predefined instruction or message provided to an artificial intelligence model to guide a response, tone, or action in a conversation. The context for how the artificial intelligence interacts with the user can be set in advance through the system prompt. Specifically, in the agent design of an artificial intelligence model based on natural language processing (e.g., LLM, etc.), a prompt is a key element for defining and guiding the agent's actions. According to one embodiment of the present invention, prompts can be divided into a system prompt and a user prompt.

System Prompt

According to one embodiment of the present invention, a system prompt determines the basic behavioral pattern and response style of the model, and provides an overall framework for how the model should respond, including the agent's role, purpose, guidelines, constraints, available tools, the agent's internal thought steps for engaging in a conversation with a user, or the internal thought flow for engaging in a conversation, so as to maintain consistency and quality of the conversation. A system prompt according to one embodiment of the present invention may include a 'Role' prompt for specifying what role an agent performs or for suggesting guidelines such as the model's response style, tone, course of action, or persona, a 'Purpose' prompt for specifying a goal that the agent must achieve, a 'Steps' prompt for specifying what thought flow or steps the model must go through to have a conversation with a user, a 'Constraints' prompt for specifying constraints such as what the model must avoid or what policies it must adhere to, a 'Tools and Capabilities' prompt for suggesting tools or functions that the model can use, an 'Output Format' prompt for suggesting an output format of the model, and an 'Examples' prompt for suggesting examples of the model's output. Meanwhile, at least one of the above-described prompts may be configured to be integrated, in which case the integrated prompt may perform the function of at least one of the remaining prompts.

System prompt to specify a role

A system prompt according to one embodiment of the present invention may include a Role prompt for specifying what role an agent performs. A Role prompt according to one embodiment of the present invention may include a prompt for providing instructions for the model's response style, tone, course of action, persona, etc. A Role prompt according to one embodiment of the present invention may include instructions for defining a counselor's persona (e.g., gender, age, occupation, hobbies, place of residence, height, weight, body type, favorite food, disliked food, tone, formality level, personality traits, etc.). For example, a Role prompt may be expressed as a directive for specifying a role, such as "You are a sleep analyst agent that analyzes a user's sleep.", "You are a knowledgeable sleep analysis support agent.", "You are an agent that provides useful content to a user based on sleep analysis." For example, the role prompt according to one embodiment of the present invention may be designed in a manner such as "Respond in a friendly and professional manner", "Answer questions concisely", "Respond creatively and flexibly to user's questions and/or responses", "Provide one response to user's questions and/or responses", "Keep responses to N sentences (where N is a natural number)", "Maintain a positive tone", "Maintain a casual and friendly tone", "Speak in a dialect determined by the user's tone of voice", and accordingly, the output/generated data of the artificial intelligence model (e.g., a phrase providing sleep analysis information, a phrase providing content, a response phrase, etc.) may be output/generated in a pre-designed manner (e.g., a tone, a level of formality, a personality trait, etc.). According to the persona of the content providing model specified by the role prompt according to one embodiment of the present invention, the user may recognize the content providing model as an expert in a field related to sleep analysis, or may feel emotions such as trust and/or friendliness.

System prompt to specify purpose

A system prompt according to one embodiment of the present invention may include a purpose prompt for specifying a purpose that the agent must achieve. For example, the purpose prompt may be expressed as a directive for specifying a purpose, such as "Analyze the user's sleep and, if necessary, request additional information to provide a smooth sleep analysis service or provide content," or "Provide accurate sleep analysis results to the user to increase satisfaction."

Meanwhile, a purpose prompt according to one embodiment of the present invention may include instructions explaining the purpose of a method for checking and/or providing sleep data of a user by a content providing model (e.g., an artificial intelligence model based on natural language processing). For example, a purpose prompt according to one embodiment of the present invention may be expressed as an instruction that the content providing model should provide sleep analysis results or content in a user-friendly manner. For example, it may include an instruction specifying a date on which measurement of environmental sensing information including sleep sound information ended and converting it to the local time of a country where the user is located.

Meanwhile, a purpose prompt according to one embodiment of the present invention may include instructions regarding how the content provision model should provide sleep analysis results or content to a user identified by a user code (e.g., ID, identification information, etc.). According to one embodiment of the present invention, the sleep analysis information is pre-stored in an external server on which the sleep analysis model is implemented, and may be transmitted to the server on which the content provision model is implemented, and provided to the user through an interactive interface by the content provision model. A purpose prompt according to one embodiment of the present invention may include instructions regarding how the sleep analysis results should be provided to the user.

In addition, a purpose prompt according to an embodiment of the present invention may include a guideline specifying information to be included in a sleep analysis result provided through a content model. Here, the analysis information to be included may include at least one of a sleep score, a total sleep time, sleep efficiency, sleep latency, a deep sleep ratio, a REM sleep ratio, sleep cycle count, a snoring ratio, and an estimated AHI, but is not limited thereto. In addition, a guideline on a method of expressing at least one of the above-described sleep analysis information when provided may also be included in the purpose prompt. Accordingly, the unit in which the sleep analysis information is displayed when provided, whether to provide a difference and/or a change rate from a past sleep session, and/or a method of expressing the same may also be determined by the purpose prompt.

System prompt to specify steps

A system prompt according to one embodiment of the present invention may include a 'Steps' prompt for specifying which thought flow or steps the model should go through to have a conversation with a user. A step prompt according to one embodiment of the present invention may be a prompt for specifying which steps the model should go through before outputting an utterance.

For example, a step prompt according to one embodiment of the present invention may include instructions to first cause the model to comprehensively review the user's sleep status information by category, and first review information that is an indicator representing sleep quality (e.g., sleep score, etc.) and/or an indicator representing sleep amount (e.g., total sleep time, etc.). A step prompt according to one embodiment of the present invention may include instructions to then cause the model to review, among the sleep status information corresponding to the remaining categories, indicators that are determined to be important (e.g., sleep efficiency data representing the ratio of the user's actual slept time to the total measured sleep time, etc.) and/or preset indicators.

The step prompt according to one embodiment of the present invention may include instructions to infer the user's health status based on the sleep state information by category that has been reviewed thereafter. For example, there may be a correlation between the user's sleep state information and health information, and the step prompt according to one embodiment of the present invention may include instructions to infer the user's health status based on the user's sleep state information. In this case, the content provision model according to one embodiment of the present invention may infer the user's health status based on the user's sleep state information by utilizing knowledge about the correlation between the sleep state information and health status that the natural language processing-based artificial intelligence model originally stored. Alternatively, the content provision model according to one embodiment of the present invention may infer the user's health status based on the user's sleep state information by utilizing knowledge previously stored through a RAG (Relevance-Augmented Generation) tool. Alternatively, the content provision model according to one embodiment of the present invention may infer the user's health status based on the user's sleep state information by utilizing knowledge about the correlation between the sleep state information and health status searched on the web by utilizing a web search tool.

A step prompt according to one embodiment of the present invention may then include instructions causing the model to output an utterance to the user based on at least one of the above-described information (e.g., information on the user's sleep state and information on the user's health).

By the instructions included in these step prompts, the utterances output by the content provision model according to one embodiment of the present invention may include information that may be more helpful to the user, or may make the user feel that the user has expertise related to sleep. As a result, the step prompts may cause the user to use the content provision model according to one embodiment of the present invention for a longer period of time or to use it more frequently, and the retention index or the user engagement index in the service applying the content provision model according to one embodiment of the present invention may be high.

Meanwhile, the order and examples of the instructions included in the above-described step prompts are described to explain one embodiment of the present invention, and the present invention is not limited thereto, and it should be understood that any instructions or order that can be assumed by a person having ordinary knowledge in the technical field to which the present invention belongs are included within the scope of the spirit of the present invention.

System prompt to specify constraints

System prompts according to one embodiment of the present invention may include constraint prompts for specifying constraints, such as content that the model should avoid or policies that it should adhere to. Constraint prompts according to one embodiment of the present invention may be expressed as directives, such as “Do not request or share personal information,” or “Do not use inappropriate language.”

System prompts to present Tools and Capabilities

System prompts according to one embodiment of the present invention may include a Tools and Capabilities prompt for suggesting tools or capabilities that the model can use. For example, the Tools and Capabilities prompt according to one embodiment of the present invention may be expressed as instructions such as "You can access an external database to verify user information," "You can use the calculator tool to calculate sleep scores," and "You can use the natural language processing tool to understand the intent of user input."

Prompt to suggest the output format of the model.

A system prompt according to one embodiment of the present invention may include an 'Output Format' prompt for presenting an output format of the model. The output format prompt according to one embodiment of the present invention may include an instruction for the model to output information that is an indicator of sleep quality (e.g., sleep score, etc.) and/or an indicator of sleep quantity (e.g., total sleep time, etc.) among the user's sleep status information before providing content or at the beginning of the content to be provided. In addition, the output format prompt according to one embodiment of the present invention may include an instruction for the model to output information that is an indicator of sleep quality (e.g., sleep score, etc.) and/or an indicator of sleep quantity (e.g., total sleep time, etc.) in a modified format. The output format prompt according to one embodiment of the present invention may include an instruction for the model to provide an interpretation of the user's sleep status information and/or the user's health information and/or other content related thereto after the model outputs the user's sleep status information and/or the user's health information. The instructions included in the output format prompts described above are examples for illustrative purposes only and the present invention is not limited thereto.

Prompt to provide an example of the model's output.

A system prompt according to one embodiment of the present invention may include an 'Examples' prompt for presenting examples of output from the model. The examples prompt according to one embodiment of the present invention may include examples of how the model should provide utterances to the user. In addition, the example prompt may also include examples of how the model should provide utterances in response to the user's input when the user makes certain inputs.

An example prompt according to one embodiment of the present invention may include an example in which the model initiates an initial conversation with a user through a greeting, and provides key indicators of sleep quantity and quality based on sleep status information of the user. Then, an example may include an example in which the model selects sleep status information included in a key category among one or more categories that classify sleep status information, and generates a comment to be provided to the user based on the selected sleep status information. In addition, an example prompt according to one embodiment of the present invention may include an example in which content is provided based on contextual information input as a user prompt.

For example, an example prompt according to one embodiment of the present invention may include an example that causes the model to output:

"Did you sleep well? Here's a summary of your last sleep.

- Sleep Score: 75 points (-12 points)

- Actual sleep time: 4 hours 25 minutes (-1 hour 12 minutes)

Yesterday, deep sleep decreased by 10% (-5%). A decrease in deep sleep time can affect sleep quality. Did you work late and then play and then sleep?"

Meanwhile, the above-described example prompts are merely examples for explanation, and the present invention is not limited thereto, and the example prompts may include all possible output examples that can be created by utilizing one or more system prompts described herein.

Examples of other system prompts

A system prompt according to one embodiment of the present invention is an instruction used to set or reset the context of a conversation, which may include an instruction to keep a memory within the session or to refer to a specific topic again during the conversation. In this case, the memory of data entered by the user within the conversation session with the user conducted through the conversational interface may be kept or referenced to generate a new response. In addition, a prompt according to one embodiment of the present invention may include an instruction that causes the content providing model to perform a specific action, such as re-producing a question for clarifying a user's question when it is unclear, or summarizing a long answer to the user's question. In this case, it may be designed in a manner such as "If the user's inquiry is unclear, ask a follow-up question to clarify." Accordingly, the content providing model may also provide a follow-up question to the user through the conversational interface. In addition, for example, a prompt according to one embodiment of the present invention may include instructions for subsequent actions, such as "when a user inquires about an account status (e.g., whether a user code exists, etc.), first check whether the user code is already stored in the server where the content provision model is implemented, and if not, guide the user to a link that allows the user to access an external application (e.g., an application provided by a sleep analysis server)."

A system prompt according to one embodiment of the present invention may include instructions on how to check related sleep analysis information in order to provide content to a user. For example, a prompt according to one embodiment of the present invention may include instructions to refer to the most recent sleep data possible in order to provide content to a user about skin care products, behavior patterns, sleep analysis information, etc. In this case, instructions may be included to list only a few types of sleep analysis information that have salient indicators or significant changes, instead of listing too many types of sleep analysis information. In addition, when a user requests content recommendations about skin care products, behavior patterns, sleep analysis information, etc., instructions may be included to output information about a specific product, a link to purchase a specific product, information that can refer to a behavior pattern, information about the effect of sleep quality on the human body, etc.

A system prompt according to one embodiment of the present invention may include instructions for causing the content providing model to retrieve sleep data. For example, the instructions may include instructions for causing the content providing model to identify the user using a user code that has been previously entered and/or previously received by the server on which the content providing model is implemented. Alternatively, if the content providing model has not previously entered and/or previously received the user code, the instructions may include instructions for causing the content providing model to guide the user to a link that allows the user to access an external application (e.g., an application provided by the sleep analysis server). Furthermore, the system prompt according to one embodiment of the present invention may include instructions for causing the content providing model to output a sentence that guides the user to access the user code through the link.

According to one embodiment of the present invention, if the prompt provides information on the total sleep time, it may include instructions to output information that the total sleep time is related to moisture retention of the skin and/or information that it is related to whether skin cell regeneration is promoted. For example, if the second prompt according to one embodiment of the present invention provides information on the deep sleep stage, it may include instructions to output information that the deep sleep stage increases collagen production for skin elasticity and/or information that it is related to whether skin cell regeneration is promoted. For example, if the prompt according to one embodiment of the present invention provides information on the REM sleep stage, it may include instructions to output information that REM sleep helps reduce skin troubles such as acne and/or information that a lack of REM sleep can increase skin sensitivity.

Meanwhile, as discussed above, one or more of the system prompts are merely examples, and at least one of them may be modeled by being integrated/omitted, and in some cases, additional prompts may be designed to achieve the purpose and/or effect sought to be achieved in the present invention.

How one or more agents utilize the System Prompt

The content providing artificial intelligence model according to an embodiment of the present invention is a model for providing beauty, health, exercise, behavior patterns and sleep-related products based on user sleep status information, and can be implemented based on an artificial intelligence model based on natural language processing. The artificial intelligence model based on natural language processing according to an embodiment of the present invention includes one or more agents, and at least one of the system prompts discussed above can be shared between one or more agents. In addition, the artificial intelligence model based on natural language processing according to an embodiment of the present invention can have one or more agents independently configuring at least one of the system prompts discussed above.

For example, one or more agents according to one embodiment of the present invention can be implemented to perform various roles, such as agents that handle beauty and sleep together, agents that handle health and exercise together, and sleep together, and one or more agents can maintain differentiation between agents while sharing some prompt elements. For a specific example, one or more agents can provide a content provision service according to sleep state information based on a common step prompt and a common output format prompt, but the remaining prompts can be selectively replaced to segment the user experience. A persona included in a role prompt can be defined as a specific personality or conversation style, and in some cases, one or more agents can use it like a module.

Meanwhile, at least one of the step prompt and the output format prompt according to one embodiment of the present invention may be designed to be applied differently depending on the situation and time zone, etc. For example, the thought flow of a model for providing sleep-related information in the morning may be different from the thought flow of a model for providing information in the evening. Accordingly, the output format may be different, such as emphasizing the recharge state based on the user's sleep analysis results in the morning and emphasizing content for inducing sleep in the evening. Alternatively, the prompt may be designed to be applied differently depending on the situation in which the agent is conversing with the user (e.g., a situation in which the user is setting up the agent through a user prompt, a situation in which the user is having a general conversation with the agent, a situation in which the user is consulting with the agent about specific content, etc.). Such flexibility can maximize the usability of the artificial intelligence model by providing information suitable for the user's needs and the situation by time zone. The embodiment of the present invention can contribute to the implementation of a content provision system based on the user's sleep state information by providing a flexible design structure in which various agents can be combined and reconfigured according to specific situations and roles.

User Prompt

According to one embodiment of the present invention, a user prompt is an input that a user delivers to a model in a conversation, and may include a problem that the model should solve or a question that the model should answer. Accordingly, the model can generate an appropriate response according to the instructions of the system prompt based on the user prompt. A user prompt according to one embodiment of the present invention may include a 'Request' component expressing a question or request that the user delivers to the model, a 'Context' component for providing background information or a situation necessary for the model to understand the user's request, and an 'Additional Information' component including detailed information necessary for the model to process the request.

For example, a 'Request' prompt that may be included in a user prompt according to one embodiment of the present invention may be expressed as a request such as "I want to know the results of my sleep analysis.", "I want to know which sleep products are suitable for me."

For example, a 'Context' prompt that may be included in a user prompt according to one embodiment of the present invention may be expressed as a request such as, "I applied the sleeping products you recommended last time and was satisfied. Please recommend similar sleeping products.", "I have to wake up early tomorrow. Please tell me what I should do before going to bed today." Meanwhile, according to one embodiment of the present invention, past conversation records with the user may also be stored in a database and input as a context prompt, and the content provision model may provide content to the user based on data such as past conversation records input as the context prompt.

For example, an 'Additional Information' prompt that may be included in a user prompt according to one embodiment of the present invention may be expressed as information such as "I was tired during the day today.", "I drank a lot of coffee during the day today.", or "I did some intense exercise today."

Meanwhile, according to one embodiment of the present invention, the user prompt may be input in a manner in which, in addition to the user inputting text, at least one of one or more blocks is selected according to a preset rule-based algorithm, or at least one of one or more texts is selected. In this case, the selected block or the selected text is each converted into natural language and input as a user prompt, and the content provision model can utilize this information input as the user prompt as contextual information to provide a service according to embodiments of the present invention.

Through a prompt according to one embodiment of the present invention, by providing clear instructions to the model, the model can clearly convey what exactly it should do, and the response quality and consistency of the model can be improved. In addition, through a prompt according to one embodiment of the present invention, a more satisfactory conversation experience can be provided to the user, and the user experience can be improved by enabling the model to generate an appropriate response that meets the user's intention. In addition, through a prompt according to one embodiment of the present invention, repetitive work can be reduced, conversation design can be carried out efficiently, and the prompt can be easily applied to various situations, so that the development time can be shortened.

Tools for the content delivery model

Meanwhile, the system prompt can perform the designed instructions based on the result of the function of the tool, which is another component of the agent. In the context of the agent of artificial intelligence based on natural language processing, the tool can perform a specific function to improve the function of the agent beyond simple conversation, or can perform functions such as retrieving information by integrating with external data. For example, in order to perform the instruction of "Speak in a dialect determined by the user's tone" in the third prompt among the system prompts, it must first be determined which dialect to speak based on the user's tone. This function can be performed in the prompt, but depending on the modeling implementation variation, the tool of the agent can perform actions such as sentiment analysis, intention classification, and tone analysis based on the user's input, and the tool can determine which dialect to speak through the prompt based on the result of the performance.

According to one embodiment of the present invention, the agent may include a Relevance-Augmented Generation (RAG) tool for accessing a pre-built database. RAG according to one embodiment of the present invention is a technology that combines information retrieval and a generative AI model, and may be a tool that operates to search for information in a pre-built database and provide utterances based on the information. RAG according to one embodiment of the present invention may be a tool that focuses on increasing the accuracy and reliability of utterance contents provided to a user by integrating search and generation. According to one embodiment of the present invention, the pre-built database may include information on the user's sleep status (e.g., sleep time, sleep cycle, or sleep efficiency data, etc.) and/or content information related thereto (e.g., health management advice, scientific research results related to sleep status, correlation data between sleep status information and health status information, a list of recommended cosmetics according to sleep status information, information on sleep products according to sleep status information, etc.).

The operation method of the RAG tool according to one embodiment of the present invention can be largely divided into two. First, the model according to one embodiment of the present invention can search for content related to the user's sleep state from a pre-built database through the RAG tool. For example, if the user wants to know the effect of stress on sleep from sleep data, the artificial intelligence model according to one embodiment of the present invention can search for related research or stress management tips from the database through the RAG tool. Second, the model according to one embodiment of the present invention can generate content to be provided to the user based on the information searched through the RAG tool. For example, if the user's sleep state information can be evaluated as "taking a long time to fall asleep," the model according to one embodiment of the present invention can generate an answer recommending tips for improving the sleep environment or related sleep products to solve this problem.

Information included in a database searched through a RAG tool according to one embodiment of the present invention may include, for example, health status information, tips for improving sleep quality, cosmetics related to sleep status, information on sleep products, etc. In addition, according to one embodiment of the present invention, the pre-built database may include the latest research results related to sleep, user reviews, or detailed information on specific products. A model according to one embodiment of the present invention may search at least one of the pieces of information included in the pre-built database and generate content to be provided to a user based on the searched information. The database searched through the RAG tool may be a static database, but may also be built as a dynamic database that integrates dynamic data in some cases. A content provision model according to one embodiment of the present invention may deliver rich and reliable information related to a user's sleep status through the RAG tool.

In addition, the tool of the agent according to one embodiment of the present invention may include a tool for accessing an external database via an API, a tool for accessing domain-specific data, a calculator tool, a natural language processing tool, a multi-step workflow automation tool, etc. For example, a tool for connecting to an external database via an API may perform a function for searching data in real time on the web, such as accessing and/or receiving a user's sleep analysis information, checking date information, checking weather information, booking an appointment, retrieving database records, and performing real-time recommendation information for products, or a function for connecting AI to an external API for task execution.

For example, a tool that accesses domain-specific data may perform actions that access structured information sources, such as searching domain-specific data (such as a product catalog or frequently asked questions) so that the AI can accurately answer queries. For example, a calculator tool may perform functions that allow the agent to perform calculations, currency conversions, date parsing, or other calculations on the fly. For example, a natural language processing tool may perform functions such as sentiment analysis, intent classification, or named entity recognition that allow the agent to understand and respond to user input in a more sophisticated way. For example, a multi-step workflow automation tool may perform functions that manage and execute complex workflows that require multiple steps, such as processing an order or generating a report (such as filling out a form, launching an external application, updating a record, etc.). Meanwhile, the description of the function of the tool herein is merely an example, and at least one of them may be integrated/omitted and modeled, and in some cases, a tool may be additionally designed to achieve the purpose and/or effect to be achieved in the present invention. The description of the structure, function, etc. of the artificial intelligence model (e.g., sleep analysis model and/or content provision model) discussed above is applied throughout the present invention, and the artificial intelligence model according to the embodiments of the present invention may be implemented in at least one of the memory (120-3) of the computing device (100-3), the memory (200-3) of the server (200-3), and the memory (320-3) of the user terminal (300-3).

Data transmission and reception between content provision server and sleep analysis server

A specific example of a tool of an agent according to one embodiment of the present invention is a tool for accessing an external database through an API, through which the content providing server can exchange data with the sleep analysis server and access sleep analysis information to be provided to the user. Here, the content providing server can be implemented to have the same configuration as the server (200-3).

Specifically, environmental sensing information including sleep sound information acquired from a computing device (100-3) and/or a user terminal (300-3) may be processed as an input of a sleep analysis model through at least one of preprocessing and/or conversion (on the other hand, it may be processed as an input of a sleep analysis model without going through both preprocessing and conversion). The sleep analysis information generated by the sleep analysis model may include sleep state information, and the sleep state information includes at least one of sleep stage information and sleep event information. The sleep analysis model may be stored in the memory (120-3) of the computing device (100-3) and executed by the processor (110-3), stored in the memory (320-3) of the user terminal (300-3) and executed by the processor (310-3), or stored in the memory of a server device for sleep analysis and executed by the processor. For convenience, the following description is based on the assumption that sleep analysis information is generated by a sleep analysis model stored in the memory of a server device for sleep analysis (i.e., a sleep analysis server). However, it should be understood that various modified examples are possible, such as sleep analysis information being generated by a sleep analysis model stored in the memory (120-3) of a computing device (100-3) and executed by a processor (110-3) and/or a sleep analysis model stored in the memory (320-3) of a user terminal (300-3) and executed by a processor (310-3). In addition, the sleep analysis server below may be implemented to have the same configuration as the server (200-3).

Sleep analysis information generated by a sleep analysis model stored in the sleep analysis server memory may be stored in the memory of the sleep analysis server. In addition, at least one of the user terminal, the content provision server, and the sleep analysis server may be connected to each other through a network, and may mutually transmit and receive data. Meanwhile, the user may mutually transmit and receive data with the sleep analysis server through the user terminal, analyze sleep from the sleep analysis server, and the sleep analysis server may identify the user terminal with ID information in the form of a 'user code'. Information regarding one or more user codes may be stored in the sleep analysis server, and may be transmitted from the sleep analysis server to the content provision server.

According to one embodiment of the present invention, context information required to provide content (e.g., past conversation records with the user, etc.) may be stored in the database of the content provision server. Meanwhile, information obtained in the process of analyzing the user's sleep state (e.g., environmental sensing information, number of steps per day, weather information, seasonal information, temperature information, humidity information, the user's coffee intake, the user's alcohol intake, the user's exercise amount/exercise intensity, etc.) may be stored in the database of the sleep analysis server. In the content provision service according to one embodiment of the present invention, a content provision model may infer and output content to be provided to the user based on the information stored in the database of the content provision server and the information stored in the sleep analysis server. Meanwhile, according to one embodiment of the present invention, context information required to provide content and information obtained in the process of analyzing the user's sleep state may be mapped to each other and stored as metadata in at least one of the databases of the content provision server and the sleep analysis server. That is, information stored through conversations with the user and information obtained in the process of measuring sleep may be mapped to each other as tags and stored as metadata. According to one embodiment of the present invention, the stored metadata may include sleep measurement data, tag information, and conversation information, and may be stored in at least one of the sleep analysis server and the content provision server by mapping them based on the time point at which the sleep data was measured or based on an event identified from the conversation information. According to one embodiment of the present invention, the stored metadata may be utilized in learning the sleep analysis model in the future or may be utilized as context information input when inferring the sleep analysis model.

1. When the user terminal operation is input first

According to one embodiment of the present invention, a user terminal (300-3) can be connected to an application for providing content of a content providing server through a communication module (370-3), and this application can provide an interactive interface through an output unit (330-3) of the user terminal (300-3).

According to one embodiment of the present invention, when a user input (e.g., a natural language sentence, button input, other user terminal operation, etc.) entered through an input unit (340-3) of a user terminal (300-3) is received by an application through an interface unit (350-3), the content provision server can identify whether the user terminal is included in one or more pre-stored user codes.

If the user code of the user terminal is not identified, a link can be output to connect to an application provided by the sleep analysis server through the above-described system prompt, etc., and the application connected through the link can be induced to check the user code and transmit the user code to the content provision server.

If the user code is identified, the content providing server can execute a tool to retrieve sleep analysis information of the corresponding user from the sleep analysis server. Accordingly, the sleep analysis information is transmitted from the sleep analysis server to the content providing server, and the content providing server can generate content related thereto based on the received sleep analysis information. The content providing server can provide at least one of the sleep analysis information and/or the content related thereto as an interactive interface output from the output unit (330-3) of the user terminal (300-3) through an application.

2. When the content provider server directly retrieves sleep analysis information

According to one embodiment of the present invention, a user terminal (300-3) can be connected to an application for providing content of a content providing server through a communication module (370-3), and this application can provide an interactive interface through an output unit (330-3) of the user terminal (300-3).

According to one embodiment of the present invention, when acquisition of environmental sensing information including sleep sound information from the input unit (340-3) of the user terminal (300-3) is completed, the sleep analysis server can transmit data to the content providing server indicating that 'sleep measurement of the corresponding user terminal has been completed'.

Meanwhile, the user terminal (300-3) here may be a terminal with a preset user code ID that automatically transmits data notifying that 'sleep measurement is complete' from the sleep analysis server to the content provision server when sleep measurement (e.g., acquisition of environmental sensing information, etc.) is terminated.

According to the present invention, the content providing server may execute a tool to retrieve the user's sleep analysis information from the sleep analysis server after receiving a notification from the sleep analysis server that 'sleep measurement of the corresponding user terminal has been completed'. When the content providing server receives the user's sleep analysis information from the sleep analysis server, the content providing server may generate content related thereto based on the received sleep analysis information. The content providing server may provide at least one of the sleep analysis information and/or the content related thereto as an interactive interface output from the output unit (330-3) of the user terminal (300-3) through an application.

At this time, since the content provision information provides at least one of the sleep analysis information and the content related thereto through the user terminal even if the user terminal operation is not input first, the provided sleep analysis information and/or content needs to be output in a way that the user can feel interested. For example, the content provision server may learn a method that can better lead to the user's follow-up conversation, and provide the sleep analysis information and/or content based on the learned method. Alternatively, the method of providing the sleep analysis information and/or content may be determined based on the user's previously stored sleep analysis information, daily changing sleep analysis information, and the user's usual speech. Alternatively, by calling up information on the news/weather of the day, etc. using the agent's tool, the topic may be changed or content related thereto may be provided.

3. When the sleep analysis server provides sleep analysis information to the content provision server

According to one embodiment of the present invention, a user terminal (300-3) can be connected to an application for providing content of a content providing server through a communication module (370-3), and this application can provide an interactive interface through an output unit (330-3) of the user terminal (300-3).

According to one embodiment of the present invention, when acquisition of environmental sensing information including sleep sound information from the input unit (340) of the user terminal (300) is terminated, the sleep analysis server can transmit the user's sleep analysis information to the content providing server together with data notifying that 'sleep measurement of the corresponding user terminal has been completed'.

Meanwhile, the user terminal (300-3) here may be a terminal with a preset user code ID that automatically transmits sleep analysis information to the content providing server along with data notifying that 'sleep measurement is complete' from the sleep analysis server when sleep measurement (e.g., acquisition of environmental sensing information, etc.) is terminated.

The content provision server may receive from the sleep analysis server a message indicating that 'sleep measurement of the user terminal has been completed' and the sleep analysis information of the user, and may then generate content related thereto based on the received sleep analysis information. The content provision server may provide at least one of the sleep analysis information and/or the content related thereto as an interactive interface output from the output unit (330-3) of the user terminal (300-3) through an application.

The sleep analysis information and/or content provided herein may be output in the manner described above to the user's benefit.

4. When the content provision server directly retrieves sleep analysis information at a specific time

According to one embodiment of the present invention, a user terminal (300-3) can be connected to an application for providing content of a content providing server through a communication module (370-3), and this application can provide an interactive interface through an output unit (330-3) of the user terminal (300-3).

According to one embodiment of the present invention, when a predetermined time has arrived, the content providing server can execute a tool to retrieve the user's sleep analysis information from the sleep analysis server. Meanwhile, the user terminal (300-3) here may be a terminal assigned a user code ID that is preset to cause the content providing server to retrieve the user's sleep analysis information from the sleep analysis server when a predetermined time has arrived.

When the content providing server receives the user's sleep analysis information from the sleep analysis server, it can generate content related thereto based on the received sleep analysis information. The content providing server can provide at least one of the sleep analysis information and/or the content related thereto as an interactive interface output from the output unit (330-3) of the user terminal (300-3) through an application.

Meanwhile, the predetermined point in time according to embodiments of the present invention may be a point in time preset by a content providing server or an application linked to the content providing server. In addition, according to embodiments of the present invention, the predetermined point in time may be a point in time when an operation of the user terminal is input (for example, a point in time when a sleep measurement end button is pressed), or may be set as a point in time when a certain period of time has elapsed from the point in time when an operation of the user terminal is input.

Meanwhile, the point in time according to embodiments of the present invention may not be a predetermined point in time, but may be an arbitrarily set point in time, or a point in time determined through learning by a content provision model.

Meanwhile, a predetermined point in time according to embodiments of the present invention may be a point in time determined based on an input operation of the user terminal while tracking input data such as a screen of the user terminal or voice input through the user terminal, or may be a point in time determined based on a preset input operation of the user terminal (e.g., Internet connection, execution of a predetermined application, etc.).

The sleep analysis information and/or content provided herein may be output in the manner described above to the user's benefit.

Linkage between sleep analysis model, content delivery model and SNS

According to embodiments of the present invention, an application provided by a content providing server may be an application that provides an interactive interface function and may be an application linked to a Social Network Service (SNS) and/or social media. That is, the Social Network Service (SNS) and/or social media application linked to the content providing server provides an interactive interface function through an output unit (330-3) of a user terminal (300-3), and when a user inputs natural language into the interactive interface provided by the Social Network Service (SNS) and/or social media application, a content providing model implemented in the content providing server linked to the application may operate, and the content providing model may provide the user with sleep analysis information and/or content related thereto based on the artificial intelligence model based on natural language processing described above.

According to one embodiment of the present invention, a linking process for linking a sleep analysis application and a social media application may be performed. Meanwhile, after the linking process according to one embodiment of the present invention, a process for determining a persona by user input among one or more personas included in a prompt of a content provision model may be performed.

The linking process according to one embodiment of the present invention may include a step of providing at least one of one or more preset blocks through a GUI (Graphical User Interface) of a social media application according to a user input. Here, the user input may include a button input, a screen touch, a text input, a voice input, a gyro sensor input, an illuminance input, etc. For example, if one of one or more quick buttons displayed together on the GUI is selected through a button input, etc., a block corresponding to the selected quick button may be displayed through the GUI. For example, if a text similar to a predetermined pattern of utterance is input by the user into the GUI of the social media application, at least one of the one or more preset blocks may be displayed through the GUI based on the user input. Meanwhile, according to one embodiment of the present invention, the similarity between the text input by the user and the predetermined pattern of utterance may be preset, and the text similarity may be determined in various ways, such as whether a keyword matches. If a text that is judged to have a low similarity to a given pattern of utterance is input, at least one of one or more preset blocks is not displayed, but an artificial intelligence model based on natural language processing can output a response to the user's text input. Therefore, by default, an artificial intelligence model based on natural language processing responds, but if a text that has a similarity level higher than a given pattern of utterance is input, at least one of one or more preset blocks is displayed, thereby allowing the user to conveniently and efficiently perform the process of linking a social media application and a sleep analysis application, thereby improving user convenience.

The linking process according to one embodiment of the present invention may include a step of transmitting a scheme for installation and/or linking of a sleep analysis application via a GUI of a social media application. Here, the scheme is for moving to a screen of a specific application or executing a specific GUI (Graphical User Interface), and may be interpreted to mean a URL Scheme, Universal Link, App Link, Deep Link, etc., and the scheme may be provided in a form that a user can input (e.g., by entering a button, touching the screen, etc.) by being included in a block that is visually provided via the GUI.

Information provided by the content delivery model

Sleep data interpretation content by category of sleep status information

According to embodiments of the present invention, the user's sleep state information may include at least one of sleep stage information, sleep stage probability information, sleep event information, and sleep event probability information. According to one embodiment of the present invention, content may be provided that interprets (or evaluates) the user's sleep data based on the user's sleep state information obtained during one or more sleep sessions.

The user's sleep data may be sleep data belonging to at least one of the following categories: Sleep Apnea category, Sleep Onset Latency category, First Cycle Sleep Quality category, REM Latency category, REM Ratio category, Deep sleep ratio category, Wake Time after Sleep Onset (WASO) category, Number of Awakening category, and Total Sleep Time category.

Generating and providing sleep data interpretation content based on large-scale language models

FIG. 68 is a diagram for explaining a method for generating interpretation content of sleep data based on a large-scale language model according to one embodiment of the present invention.

According to one embodiment of the present invention, the step of generating interpretation content of sleep data based on the large-scale language model disclosed in FIG. 68 includes the step of classifying user sleep characteristics (900-4) and the step of extracting sleep data interpretation content (940-4) based on the large-scale language model by using the classified user sleep characteristics (920-4) as input to the large-scale language model (930-4).

For example, the user's sleep characteristic (900-4) can be classified into at least one of the first sleep characteristic (922-4), the second sleep characteristic (924-4), the third sleep characteristic (926-4), and the fourth sleep characteristic (928-4), and at least one classified sleep characteristic can be input into a large-scale language model (930-4). Data output from the large-scale language model (930-4) can be input into at least one of the first large-scale language model, the second large-scale language model, the third large-scale language model, and the fourth large-scale language model, so that sleep data interpretation content can be extracted (940-4).

Description of the flow chart of Fig. 69a

FIG. 69a is a flowchart of a method for providing non-numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

As illustrated in FIG. 69a, according to one embodiment of the present invention, a method for providing non-numerical interpretation content of sleep data may include a sleep information acquisition step (S100-4), a non-numerical interpretation content generation step of sleep data (S120-4), and a step of displaying the generated sleep data interpretation content (S140-4). The interpretation content of sleep data includes at least two words, and the generated interpretation content may not be expressed in a numerical manner.

A method for providing non-numerical interpretation content of sleep data according to one embodiment of the present invention may include a sleep log information storage step of storing sleep log information related to an account assigned to a user in a memory.

In a method for providing non-numerical interpretation content of sleep data according to one embodiment of the present invention, the sleep information acquisition step (S100-4) may include a sleep information conversion step for converting sleep sound information into information in a frequency domain or a frequency-time domain.

In a method for providing non-numerical interpretation content of sleep data according to one embodiment of the present invention, the sleep information acquisition step (S100-4) may further include a sleep information inference step for inferring information about sleep by using sleep sound information as input to a deep learning model.

In a method for providing non-numerical interpretation content of sleep data according to one embodiment of the present invention, the non-numerical interpretation content generation step (S120-4) of sleep data may include a first sleep data interpretation content generation step of generating a text having high visibility among the non-numerical interpretation content of sleep data, and a second sleep data interpretation content generation step of generating a text constituting the non-numerical interpretation content of the user's sleep data.

In a method for providing non-numerical interpretation content of sleep data according to one embodiment of the present invention, the non-numerical interpretation content generation step (S120-4) of sleep data may include a third sleep data interpretation content generation step of generating advice text based on an evaluation of sleep.

The non-numerical interpretation content generation step (S120-4) of sleep data according to one embodiment of the present invention may further include a step of generating interpretation content of sleep data based on a lookup table. The step of generating interpretation content of sleep data based on the lookup table may include a lookup table user sleep characteristic classification step of classifying characteristics of a user's sleep based on sleep information, and a sleep data interpretation content extraction step based on the lookup table corresponding to the classified sleep characteristics.

The non-numerical interpretation content generation step (S120-4) of sleep data according to one embodiment of the present invention may further include a step of generating interpretation content of sleep data based on a large-scale language model, as illustrated in FIG. 68. The step of generating interpretation content of sleep data based on the large-scale language model may include a step of classifying characteristics (900-4) of a user's sleep based on sleep information, and a step of extracting interpretation content (940-4) of sleep data based on the large-scale language model as an output using the classified sleep characteristics (920-4) as an input of the large-scale language model (930-4).

According to one embodiment of the present invention, the step (S140-4) of displaying the generated sleep data interpretation content may further include a step of generating a graphical user interface. Or, the step may further include a user sleep graph generation step of generating a graph of sleep stages within the user's sleep period based on sleep information.

Description of the flow chart of Figure 69b

FIG. 69b is a flowchart of a method for providing numerical interpretation content based on a user's sleep data according to one embodiment of the present invention.

As illustrated in FIG. 69b, a method for providing numerical interpretation content based on sleep data of a user according to an embodiment of the present invention may include a sleep information acquisition step (S200-4), a non-numerical interpretation content generation step of sleep data (S220-4), a numerical interpretation content generation step of sleep data (S240-4), and a step of displaying the generated sleep data interpretation content (S260-4). The sleep data interpretation content according to an embodiment of the present invention includes at least one of sleep data interpretation content (non-numerical method) including at least two words, and sleep numerical evaluation (numerical method).

According to one embodiment of the present invention, a method of generating one or more graphical user interfaces representing an evaluation regarding sleep may include a sleep log information storage step of storing sleep log information related to an account assigned to a user in memory.

In a method for generating one or more graphical user interfaces representing an evaluation of sleep according to one embodiment of the present invention, the sleep information acquisition step (S200-4) may include a sleep information conversion step for converting sleep sound information into information in a frequency domain or a frequency-time domain.

In a method for generating one or more graphical user interfaces representing evaluations regarding sleep according to one embodiment of the present invention, the sleep information acquisition step (S200-4) may further include a sleep information inference step for inferring information regarding sleep by using sleep sound information as input to a deep learning model.

In a method for generating one or more graphical user interfaces representing an evaluation regarding sleep according to one embodiment of the present invention, a non-numerical interpretation content generation step (S220-4) of sleep data may include a first sleep data interpretation content generation step for generating text having high visibility among the evaluation regarding sleep, and a second sleep data interpretation content generation step for generating text constituting the user's evaluation regarding sleep.

In a method for generating one or more graphical user interfaces representing an evaluation regarding sleep according to one embodiment of the present invention, the non-numerical interpretation content generation step (S220-4) of sleep data may include a third sleep data interpretation content generation step of generating advice text based on the evaluation regarding sleep.

According to one embodiment of the present invention, the non-numerical interpretation content generation step (S220-4) of sleep data and the numerical interpretation content generation step (S240-4) of sleep data may further include a step of generating interpretation content of sleep data based on a lookup table. The step of generating interpretation content of sleep data based on the lookup table may include a lookup table user sleep characteristic classification step of classifying characteristics of a user's sleep based on sleep information, and a sleep data interpretation content extraction step based on the lookup table corresponding to the classified sleep characteristics.

In addition, according to one embodiment of the present invention, the non-numerical interpretation content generation step (S220-4) of sleep data and the numerical interpretation content generation step (S240-4) of sleep data may further include a step of generating interpretation content of sleep data based on a large-scale language model, as illustrated in FIG. 68. The step of generating interpretation content of sleep data based on a large-scale language model may include a large-scale language model user sleep characteristic classification step of classifying the characteristics (900-4) of the user's sleep based on sleep information, and a step of extracting interpretation content (940-4) of sleep data based on the large-scale language model as an output using the classified sleep characteristics (920-4) as an input of the large-scale language model (930-4).

According to one embodiment of the present invention, the graphical user interface display step (S260-4) may include a user sleep graph generation step for generating a graph of sleep stages within a user's sleep period based on sleep information.

Example of providing sleep data interpretation content

Sleep data interpretation content in the Sleep Apnea category

Sleep apnea refers to a condition in which breathing stops during sleep. The degree of sleep apnea can be expressed based on the Apnea-Hypopnea Index (AHI). AHI is an index that shows the average number of apneas or hypopneas that occur per hour during sleep. If the respiratory air signal decreases to less than 10% of the reference value and lasts for more than 10 seconds, it is classified as apnea, and if the respiratory air signal decreases to less than 30% of the reference value and lasts for more than 10 seconds and the oxygen saturation decreases by more than 3%, it is classified as hypopnea. If the AHI is less than 5, sleep breathing can be considered stable. If the AHI is 5 or more but less than 15, sleep breathing can be considered normal or slightly unstable. If the AHI is 15 or more, sleep breathing can be considered unstable. If the AHI value has decreased compared to the past, it can be considered an improved (better) indicator. If the AHI value is higher than in the past, it can be judged as a more deteriorated (or worse) indicator.

Sleep Data Interpretation Content in the Sleep Onset Latency Category

Sleep onset latency (SOL) is the time it takes to gradually transition from a fully awake state to sleep, and is generally defined as the time it takes to transition into stage 1 sleep (or shallow sleep), the lightest stage of non-REM sleep.

According to embodiments of the present invention, sleep onset latency or sleep onset latency (SOL) may mean the time it takes to transition from a fully awake state to at least one of a light sleep stage, a deep sleep stage, and a REM sleep stage.

The sleep onset delay time can be an indicator for the diagnosis and evaluation of hypersomnia or sleep deprivation through the Multiple Sleep Latency Test (MSLT). The MSLT is known as a representative test method with high reliability among the techniques used to quantify the degree of sleep deprivation.

On the MSLT, a sleep onset delay of 0 to 5 minutes may be assessed as severe sleep deprivation. A sleep onset delay of 5 to 10 minutes may be assessed as troublesome sleep deprivation. A sleep onset delay of 10 to 15 minutes may be assessed as manageable sleep deprivation. A sleep onset delay of 15 to 20 minutes may be assessed as little or no sleep deprivation.

According to embodiments of the present invention, if the measured sleep onset delay time is less than 30 minutes, the level of arousal is not high, but if it is 30 minutes or more, the level of arousal can be evaluated as high. Therefore, maintaining the sleep onset delay time within an appropriate range is a method for improving the quality of sleep. According to embodiments of the present invention, when the standard sleep onset delay time is set to 30 minutes, it can be determined as a medically good indicator when the measured sleep onset delay time is less than 30 minutes, and it can be determined as a medically bad (or bad) indicator when it is 30 minutes or more.

Sleep data interpretation content in the Wake time After Sleep Onset (WASO) category

Wake time After Sleep Onset (WASO) is the total time that the user is in the wake stage after entering sleep and before waking up completely. WASO does not refer to the frequency of occurrence of wake stages during sleep, but rather the duration of the wake stage. Wake time After Sleep Onset (WASO) can be calculated by adding up the times of the wake stages during sleep that occur after falling asleep. For example, if a user wakes up 5 times in 1 minute after falling asleep and before waking up, WASO can be calculated as 5 minutes. If a user wakes up 40 times in 30 seconds after falling asleep and before waking up, WASO can be calculated as 20 minutes. The quality of a user's sleep can be evaluated through WASO, which is a quantitative indicator.

Sleep data interpretation content in the REM Latency category

REM Latency refers to the time it takes for a user to enter the first REM sleep stage after entering sleep. After entering sleep, non-REM sleep stages and REM sleep stages may appear periodically, and the time it takes from the start of sleep to the occurrence of the first REM sleep can be defined as REM Latency.

If REM Latency is not properly maintained and is excessively short or long, problems such as mental illness and sleep disorders may occur, so REM Latency can be an important sleep indicator not only for sleep quality but also medically. According to one embodiment of the present invention, if REM Latency is less than 45 minutes, it can be evaluated as excessively short, and if REM Latency is 45 minutes or more and less than 2 hours, it can be evaluated as appropriate. If REM Latency is 2 hours or more, it can be evaluated as excessively long.

Sleep data interpretation content in the REM Ratio category

REM sleep stage percentage (REM Ratio) refers to the ratio of time spent in the REM sleep stage to the total sleep time measured in the last sleep session. If the ratio of REM sleep to the total sleep time is not properly maintained and is too short or long, it can have a negative impact on sleep quality, so REM Ratio can be used as an important sleep index to evaluate sleep quality.

According to one embodiment of the present invention, if the ratio of the REM sleep stage to the total sleep time is less than 15%, the REM sleep time may be evaluated as insufficient. If the ratio of the REM sleep stage is 15% or more but less than 30%, the REM sleep time may be evaluated as appropriate. If the ratio of the REM sleep stage is 30% or more, the REM sleep time may be evaluated as excessive.

Sleep data interpretation content in the category Deep sleep ratio

Deep sleep ratio refers to the ratio of time in the Deep sleep stage from the time of falling asleep to the time of waking up during a sleep session. For example, if sleep measurement started at exactly 9:00 PM, fell asleep at 9:30 PM, woke up at 7:30 AM, and ended at 8:00 AM, the ratio of time in the Deep sleep stage among the time from 9:30 PM to 7:30 AM (a total of 10 hours) is the Deep sleep ratio. It refers to the ratio of time in the Deep sleep stage to the sleep time measured in the last sleep session. If the ratio of the Deep sleep stage to the sleep time is not appropriate and is too short or long, it can have a negative effect on sleep quality, so the Deep sleep ratio can be used as an important sleep indicator to evaluate sleep quality.

According to one embodiment of the present invention, if the ratio of the deep sleep stage to the total sleep time is less than 15%, it can be evaluated that the time in the deep sleep stage is insufficient. If the ratio of the deep sleep stage is 15% or more, it can be evaluated that the time in the deep sleep stage is appropriate.

In addition, in terms of the distribution of deep sleep stages, if the ratio of deep stages to sleep time in the first half of sleep is more than 50%, it can be evaluated that the deep sleep time in the first half of sleep was appropriate. On the other hand, if the ratio of deep sleep stages measured in the first half of sleep is less than 50%, it can be evaluated that the deep sleep time in the first half of sleep was insufficient. Here, the first half of sleep can mean the period before half of the time from the time of falling asleep to the time of waking up. For example, if sleep measurement was started at exactly 9:00 PM, fell asleep at 9:30 PM, woke up at 7:30 AM, and ended at 8:00 AM, the first 5 hours out of the 10 hours from 9:30 PM to 7:30 AM can be considered the first half of sleep. In other words, in this case, the time from 9:30 PM to 2:30 AM corresponds to the first half of sleep.

Sleep data interpretation content in the First Cycle Sleep Quality category

First Cycle Sleep Quality can be determined by the ratio of the wake stage to the deep sleep stage that occurs during the first part of sleep. The first part of sleep refers to the time from the time of falling asleep to the time just before the first REM sleep stage is measured, or a specified time, whichever is shorter.

For example, if the time of falling asleep is 9:30 PM and the time of measuring the first REM sleep stage is 10:30 PM, the time from 9:30 PM to 10:30 PM can be determined as the early part of sleep. Alternatively, if the first REM sleep stage is measured later than the time of falling asleep, a predetermined time can be determined as the early part of sleep. For example, if the time of falling asleep is 9:30 PM and the first REM sleep stage is measured later (e.g., 12:30 PM), a predetermined time (e.g., 2 hours) can be determined as the early part of sleep. Here, the predetermined time is not limited to 2 hours and can be arbitrarily set to another value.

According to one embodiment of the present invention, if the awakening stage appears more than 10% during the first cycle, it can be evaluated that frequent or long-term awakening occurs in the early part of sleep. If the awakening stage appears less than 10% and the deep sleep stage appears less than 33% during the first cycle, it can be evaluated that few awakenings occur in the early part of sleep, but deep sleep is also insufficient. If the awakening stage appears less than 10% and the deep sleep stage appears more than 33% during the first cycle, it can be evaluated that few awakenings occur in the early part of sleep, but deep sleep is also sufficient.

Sleep data interpretation content in the Number of Awakening category

According to one embodiment of the present invention, the number of awakenings refers to the number of awakening stages that occur during sleep before waking up completely after falling asleep. If the number of awakenings is high, the quality of sleep may be evaluated as low. According to one embodiment of the present invention, if the number of awakenings is 30 or more, the number of awakenings may be evaluated as high. If the number of awakenings is less than 30, the number of awakenings may be evaluated as low.

Sleep data interpretation content in the Total Sleep Time category

According to one embodiment of the present invention, Total Sleep Time refers to the total time taken from falling asleep to waking up completely during a sleep session. Since the total sleep time is also related to the number of sleep stage cycles that can occur during sleep, the quality of sleep can be improved if an appropriate sleep time is maintained. In addition, the quality of sleep can be improved if the number of sleep stage cycles appears appropriately.

According to one embodiment of the present invention, if the total sleep time is 10 hours or more, the total sleep time can be evaluated as long. If the total sleep time is 6 hours or more but less than 8 hours, the total sleep time can be evaluated as appropriate. If the total sleep time is 4 hours or more but less than 6 hours, the total sleep time can be evaluated as somewhat short. If the total sleep time is less than 4 hours, the total sleep time can be evaluated as short.

In addition, according to one embodiment of the present invention, if the number of sleep stage cycles that appear during the entire sleep time is less than 4, the sleep cycles may be evaluated as insufficient. If the number of sleep stage cycles that appear during the entire sleep time is 4 or more but less than 8, the sleep cycles may be evaluated as adequate. If the number of sleep stage cycles that appear during the entire sleep time is 8 or more, the sleep cycles may be evaluated as abundant.

How to determine the importance of sleep information across categories

In providing a user with sleep interpretation by category of sleep information, there may be a demand to preferentially provide a category of sleep information that is judged to be most important in the current sleep session (or the previous sleep session). Accordingly, according to embodiments of the present invention, a method is disclosed for calculating importance (or importance score) between categories of sleep information and providing a user with sleep interpretation by category of sleep information based on the calculated importance.

According to one embodiment of the present invention, in a method for determining importance between categories of sleep information, an importance score may be calculated based on at least one of information comparing sleep data (e.g., sleep state information) generated in a previous sleep session with sleep data of a user generated during a predetermined period in the past, information comparing the data with a medical standard, and information comparing the data with sleep data of another user generated during a predetermined period in the past.

According to one embodiment of the present invention, importance parameters can be learned based on Reinforecement Learning with Human Feedback (RLHF). Reinforcement Learning with Human Feedback (RLHF) is a learning method that combines reinforcement learning and human feedback, and improves the performance of artificial intelligence by learning information output through an artificial intelligence model based on human feedback. According to one embodiment of the present invention, importance scores of categories of sleep information can be calculated based on learned importance parameters.

Meanwhile, according to one embodiment of the present invention, the importance between categories of sleep information may be determined by the instruction 'automatically select and provide categories of sleep information that appear to be important' included in the system prompt of the content provision model. Or, according to one embodiment of the present invention, the importance between categories of sleep information may be arbitrarily set for each sleep session.

Recommended sleep aids

An artificial intelligence model based on natural language processing according to one embodiment of the present invention can recommend sleeping products for improving the user's sleep quality based on the user's sleep analysis results. The artificial intelligence model according to the present invention can analyze the user's sleep pattern, sleep time, sleep quality, etc., and recommend sleeping products that the user needs, thereby helping the user build an optimal sleeping environment and improve the quality of sleep.

An artificial intelligence model based on natural language processing according to one embodiment of the present invention can recommend sleep products based on the user's sleep analysis data or by considering various factors.

Sleep-related products according to embodiments of the present invention may include products such as a quilt, a pad, a mattress, a yoga mattress, a latex mattress, a mattress cooler, a mattress cover, a bed skirt, a single comforter, a duvet, a quilt, a blanket, an electric blanket, a weighted blanket, a duvet cover, a pillow, a memory foam pillow, a smart pillow, a vibrating pillow, a body pillow, a pillow cover, pajamas, underwear, a waist belt, leggings, a sleep robe, sleep socks, a sleep cap, an eye mask, earplugs, soundproof earplugs, earphones, slippers, bedroom decorations, a bed tray, a bed frame, a bed canopy, a bed headboard, an underbed storage box, a negative pressure bedroom system, a stress ball, a toy, a doll, a tent, a baby tent, a travel sleep kit, a curtain, a smart blind, a bedside table, a snack pad, sleep lenses, accessories, and nose hair removal strips.

For example, if the analysis of a user's sleep pattern determines that the user frequently tosses and turns or has difficulty sleeping deeply, an artificial intelligence model according to an embodiment of the present invention may suggest a customized pillow, a memory foam mattress, or a special bedding product capable of controlling body pressure distribution. An artificial intelligence model according to an embodiment of the present invention may recommend a sleeping heating mat capable of controlling the temperature and humidity of a bedroom, a humidity control device, or a white noise device for blocking out noise, thereby helping the user create an optimal sleeping environment.

In addition, products related to sleep according to embodiments of the present invention may include products related to aroma, diffusers, therapy diffusers, sleep sprays, sprays for dry mouth, pillow sprays, pillow mists, air sprays, essential oils, candles, perfumes, cosmetics, bathroom products (cleaning products), laundry products, fabric softeners, fabric softener sheets, lotions, skins, sleeping packs, mask packs, etc.

In addition, sleep-related products according to embodiments of the present invention may include things that can be consumed by users, such as nutritional supplements, general foods, health functional foods, special nutritional foods, foods for special medical purposes, medicines, sleep aids, food additives, beverages, tea, decaffeinated beverages, sleepy time tea, etc.

In addition, sleep-related products according to embodiments of the present invention may include tableware, interior building materials, bathtubs, building materials, soundproof panels, electronic devices, temperature control devices, alarm clocks, negative ion generators, humidifiers, dehumidifiers, air purifiers, air conditioners, exercise equipment, yoga balls, nap chairs, home appliances, mobile devices, wearable devices, lighting devices, audio devices, music CDs, music players, smart plugs for controlling bedside electronic devices, white noise devices, pet products, beam projectors, ceiling star projectors, positive pressure machines, sleep disorder treatment devices, nighttime drug dispensers, and the like.

In addition, sleep-related products according to embodiments of the present invention may include applications for measuring sleep, such as sleep trackers. Or, they may include sleep counseling applications or sleep counseling services, products providing information for alleviating sleep disorders, products for assisting children's sleep, etc.

In addition, a sleep-related product according to embodiments of the present invention may include image-inducing information including at least one of text information, audio information, and visual information, or a device for providing image-inducing information.

In addition, the artificial intelligence model based on natural language processing according to one embodiment of the present invention can recommend a mask with a calming effect, a heating pad, or a sleeping tool that helps psychological stability if the sleep analysis result determines that the user's sleep quality is not high. In addition, sleeping products that are most suitable for the user's sleeping position or environment are suggested, and customized recommendations that reflect the user's personal preferences or lifestyle habits can be made.

Meanwhile, the artificial intelligence model according to the present invention can analyze the user's sleep state information during one or more sleep sessions and continuously recommend sleep products by reflecting the results. For example, if the user's sleep quality has improved after using a specific product, similar sleep products can be recommended by reflecting positive feedback on the product, or new products suitable for the user can be suggested. Through this, the user can continuously maintain an improved sleep environment and receive more accurate and personalized support for selecting sleep products to optimize sleep quality.

Recommend user behavior guidelines

An artificial intelligence model based on natural language processing according to one embodiment of the present invention can suggest behavioral patterns such as diet, diet therapy, and exercise to improve sleep quality and overall health based on the results of sleep analysis of the user. The model according to one embodiment of the present invention analyzes the user's sleep pattern, sleep time, sleep quality, etc. and recommends an optimal behavioral pattern accordingly, thereby providing a solution to improve the user's overall well-being.

An artificial intelligence model according to one embodiment of the present invention may provide a behavioral pattern (e.g., a diet and/or a dietary regimen) for consuming foods containing nutrients that may help improve sleep quality. For example, an artificial intelligence model according to one embodiment of the present invention may recommend consuming almonds, bananas, cherries, cabbage, and the like, which are foods rich in sleep-promoting ingredients such as magnesium, calcium, and tryptophan. In addition, an artificial intelligence model according to one embodiment of the present invention may advise to limit caffeine and alcohol intake to minimize negative effects on sleep. An artificial intelligence model according to one embodiment of the present invention may suggest to avoid excessive eating during dinner time and to maintain regular meal times, so that a user can maintain a consistent sleep pattern. The specific diet or dietary regimen-related behavioral patterns described above are merely examples for explanation and are not limited thereto.

An artificial intelligence model according to one embodiment of the present invention can recommend an appropriate exercise intensity and time based on the user's sleep analysis results. For example, an artificial intelligence model according to one embodiment of the present invention can recommend light aerobic exercise, such as walking, yoga, or stretching, to a user who has poor sleep quality or suffers from insomnia, thereby helping relieve stress and relax the body. In addition, an artificial intelligence model according to one embodiment of the present invention can recommend that exercise be performed at least 2-3 hours before sleep time, and can also guide users to avoid strenuous exercise in the evening. The specific exercise-related behavior patterns described above are merely examples for explanation and are not limited thereto.

The AI model according to one embodiment of the present invention also adjusts the behavioral patterns in a way that is suitable for the user's daily habits and environment. For example, the AI model according to one embodiment of the present invention may suggest minimizing the use of electronic devices and adjusting the temperature and lighting of the bedroom to optimize the sleeping environment. In addition, the AI model according to one embodiment of the present invention may recommend maintaining regular sleeping habits of going to bed and waking up at set times, and may suggest methods for eliminating external factors that interfere with sleep. The AI model according to one embodiment of the present invention may advise improving the quality of sleep through psychological stability methods such as meditation, deep breathing, and avoiding naps. The behavioral patterns related to the specific daily habits of the user described above are merely examples for explanation and are not limited thereto.

The artificial intelligence model according to the present invention can monitor the user's sleep analysis data during one or more sleep sessions, track changes in sleep patterns, and update and suggest behavior patterns in real time based on the latest data. Through this, the user can manage his/her sleep health more efficiently and receive customized solutions to improve the quality of sleep.

The steps of a method or algorithm described in connection with the embodiments of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination of these. The software module may reside in a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Flash Memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present invention pertains.

The components of the present invention may be implemented as a program (or application) to be executed by being combined with a computer as hardware and stored on a medium. The components of the present invention may be executed as software programming or software elements, and similarly, the embodiments may be implemented in a programming or scripting language such as C, C++, Java, assembler, etc., including various algorithms implemented as a combination of data structures, processes, routines, or other programming elements. Functional aspects may be implemented as algorithms that are executed on one or more processors.

The various embodiments disclosed herein can be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term "article of manufacture" includes a computer program, a carrier, or a medium that is accessible from any computer-readable device. For example, computer-readable media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media disclosed herein include one or more devices and/or other machine-readable media for storing information. The term "machine-readable medium" includes, but is not limited to, wireless channels and various other media capable of storing, retaining, and/or transmitting instructions(s) and/or data.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of exemplary approaches. It is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present invention based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

[Explanation of symbols]

10: User Terminal

11: Wireless Communication Department

311: Broadcast receiving module

312: Mobile Communication Module

313: Wireless Internet Module

314: Short-range communication module

315: Location Information Module

12: Input section

321: Camera

322: Microphone

323: User Input

14: Sensing section

341: Proximity sensor

342: Light sensor

15: Output section

351: Display section

352: Sound output section

353: Haptic Module

354: Optical output section

16: Interface section

17: Memory

18: Control Unit

19: Power supply section

100: Graphical User Interface Generation Device

120: Display

140: Memory

160: Processor

200: Device providing graphical user interface

220: Display

240: Memory

260: Processor

20: External Server

21: Processor

215: AI Processor

215a: Data Learning Department

215b: Data preprocessing section

215c: Data Selection Section

215d: Model Evaluation Department

22: Memory

221: Learning Model

27: Communication Module

11a: Area where object status information can be obtained

10a: Electronic devices connected to a network within the area (11a)

10b: Electronic devices connected to a network within the area (11a)

10c: Electronic devices not connected to a network within the area (11a)

10d: Electronic devices not connected to a network within the area (11a)

20a: Electronic devices outside the scope of area (11a)

20b: Electronic devices outside the scope of area (11a)

E: Sound information

P: Peculiarities related to the user's sleep

SS: Sleep sound information

SP: Spectrogram

A: Sleep analysis model

B: Feature extraction model

C: Middle layer

D: Feature classification model

E: Sleep information inference

Claims (20)

In a method for providing interpretation content of a user's sleep data, Step of obtaining user's sleep information; A step of generating user's sleep state information based on the user's acquired sleep information; A step for generating user's sleep data interpretation content based on the generated sleep state information; and A step of outputting the generated user's sleep data interpretation content; Including, The above user's sleep data interpretation content is expressed in at least one of a numerical or non-numerical manner, A method for providing interpretation content of a user's sleep data. In paragraph 1, The above user's sleep data interpretation content is generated based on comparing the generated user's sleep state information with at least one of the user's sleep data generated over a predetermined period of time in the past, medical standards, and other user's sleep data generated over a predetermined period of time in the past. A method for providing interpretation content of a user's sleep data. In the second paragraph, The above user's sleep data interpretation content is generated based on at least one of a lookup table or a large-scale language model. A method for providing interpretation content of a user's sleep data. In any one of claims 1 to 3, The above sleep status information is generated based on combining the acquired user's sleep information and other information into multimodal data. A method for providing interpretation content of a user's sleep data. In any one of claims 1 to 3, The above sleep status information is generated based on a portion of the user's acquired sleep information cut off by a predetermined length on the time axis. A method for providing interpretation content of a user's sleep data. In a system for initiating sleep measurement through a user terminal, A permission management tool that collects the permissions required to perform sleep measurements; and A sleep data collection tool that activates the sensor module of the user terminal based on the collected authority; The above sleep data collection tool is, configured to detect a first event via the above activated sensor module, The broadcast trigger of the sleep measurement of the user terminal is configured to occur based on the detected first event, When the above broadcast trigger occurs, the sensor module is further activated for sleep measurement, configured to detect a second event via an additional activated sensor module for the above sleep measurement, The sleep analysis trigger of the sleep measurement of the user terminal is configured to occur based on the detected second event, The second event detected above is configured to be detected by the sensor module by a stimulus above a threshold, A system for initiating sleep measurement through a user terminal. In paragraph 6, The above permission management tool is Configured to request at least one of the Overlay permission permission, the Foreground service permission, the Activation permission of the sensor module and the Time-based alarm permission. A system for initiating sleep measurement through a user terminal. In paragraph 6, The first event detected above is, At least one of initiation of wired charging of the user terminal, initiation of wireless charging of the user terminal, detection of a change in the accelerometer sensor of the sensor module, switching the display unit of the user terminal to the OFF state, a change in the battery status of the user terminal, connection of the user terminal with another device, and unlocking of the user terminal. A system for initiating sleep measurement through a user terminal. In paragraph 6, The second event, which is configured to be detected by a stimulus above a threshold by the sensor module, At least one of initiation of wired charging of the user terminal, initiation of wireless charging of the user terminal, detection of change in an accelerometer sensor of the sensor module, change in a battery status of the user terminal, connection of the user terminal with another device, unlocking of the user terminal, palm swiping for the user terminal, finger swiping for the user terminal, finger tapping for the user terminal, voice input for the user terminal, vibration detection of the user terminal, screen wheel scrolling for the user terminal, pressing a specific user interface of the user terminal, pressing a display unit of the user terminal for a predetermined period of time or longer, moving two or more fingers simultaneously for the display unit, flipping the user terminal, activating a proximity sensor of the user terminal, detection of a temperature change of the user terminal, and detection of a change in illumination around the user terminal. A system for initiating sleep measurement through a user terminal. In paragraph 6, The above sleep analysis trigger is Including information on the start time of sleep analysis of the measured sleep, A system for initiating sleep measurement through a user terminal. In paragraph 6, The above sleep data collection tool is, Configured to transmit environmental sensing information collected by the sensor module from the time of occurrence of the above broadcast trigger to another device for sleep analysis. A system for initiating sleep measurement through a user terminal. In paragraph 6, The above sleep data collection tool is, When the above broadcast trigger occurs, Configured to provide a sleep measurement confirmation user interface to the above user terminal, A system for initiating sleep measurement through a user terminal. In paragraph 6, The above sleep data collection tool is, Configured to force broadcast trigger generation even in the low power mode of the above user terminal, A system for initiating sleep measurement through a user terminal. A method for providing content related to a user's sleep status information, A step of obtaining environmental sensing information through a sensor module of a user terminal; A step of converting sleep sound information included in the acquired environmental sensing information into a spectrogram; A step of processing the above spectrogram as input to a sleep analysis model to generate information on the user's sleep state; and A step of providing content related to the sleep state information through a content provision model based on the above sleep state information; Including, The above sleep analysis model is an artificial intelligence model including one or more artificial neural networks, and includes a feature extraction model and a feature classification model. The above content provision model is a conversational artificial intelligence model based on natural language processing that includes one or more artificial neural networks. A method for providing content related to a user's sleep state information. In Article 14, The above feature extraction model is a model configured to analyze the frequency pattern of the user's breathing included in the above sleep sound information, The above feature classification model is a model configured to analyze the periodic pattern of the user's breathing included in the above sleep sound information. A method for providing content related to a user's sleep state information. In Article 14, The above content delivery model includes one or more prompts, The one or more prompts include a prompt for specifying a role of the content delivery model, a prompt for specifying a purpose of the content delivery model, and a prompt for suggesting an output format through the content delivery model. A method for providing content related to a user's sleep state information. In Article 14, The above content delivery model includes one or more tools, One or more of the above tools include tools for accessing external databases, The above external database is a static database or a dynamic database. A method for providing content related to a user's sleep state information. In any one of paragraphs 14 to 17, At least two of the above environmental sensing information, the above generated sleep state information, and the conversation information with the user through the content provision model are mapped to each other and configured as metadata. The above metadata is stored in at least one of the database of the sleep analysis server where the sleep analysis model is implemented and the database of the content provision server where the content provision model is implemented. A method for providing content related to a user's sleep state information. In any one of paragraphs 14 to 17, The above content provision model is: Even if there is no dialogue input from the user terminal, it is configured to proactively provide content related to the sleep state information through the conversational interface. A method for providing content related to a user's sleep state information. For a user terminal to provide content related to the user's sleep status information, sensor module; Input section; processor; Output section - said output section being configured to output an interactive interface; and Communication module; Including, The above interactive interface, Includes content related to the sleep state information received from the content providing server through the communication module, The above content is generated by the content providing server through a content providing model based on the sleep status information received from the sleep analysis server. The above content provision model is a conversational artificial intelligence model based on natural language processing that includes one or more artificial neural networks. The above sleep state information is obtained by the sleep analysis server processing the transmitted sleep sound information as input to the sleep analysis model when the sleep sound information included in the environmental sensing information acquired by the sensor module is transmitted to the sleep analysis server through the communication module. The above sleep analysis model is an artificial intelligence model including one or more artificial neural networks, including a feature extraction model and a feature classification model. A user terminal for providing content related to the user's sleep status information.

Images