WO2024225785A1 - Method, program, and device for acquiring user-customized neural network model for identifying biological information - Google Patents

Abstract

A method for acquiring a user-customized neural network model for identifying biological information according to an embodiment of the present disclosure comprises the steps of: inputting biological information sensed from a user to a trained neural network model and predicting biological information related to the state of the user; and when new data about the user is acquired, updating the trained neural network model on the basis of the acquired new data so that the neural network model is personalized, wherein the new data includes biological data measured from the user in association with the state.

Description

Method, program and device for obtaining a user-customized neural network model for identifying biometric information

The present disclosure relates to deep learning technology in the medical field, and more particularly, to a method, program and device for obtaining a user-customized neural network model for identifying biometric information.

Recently, with the development of information and communication technology, deep learning technology is being utilized in various technical fields. In particular, deep learning technology is being utilized in various ways in the medical field, which used to rely on specialized and limited personnel such as doctors and researchers to determine the patient's disease. For example, this includes inputting biometric information acquired from the patient in real time into a pre-trained deep learning model to analyze the patient's condition or disease.

Meanwhile, in the case of deep learning models used in the medical field, since they are pre-trained based on preset learning data and provided to each user, there is a problem that different results are generated for each user or subject (e.g., patients, etc.) in actual use. This is because biometric information of the same type or condition can be acquired in different forms or values for each user. This is also due to the biological characteristics of each user. Therefore, even for the same deep learning model, a method is required to personalize it for each user by reflecting the user's environment and characteristics.

The present disclosure has been made in response to the aforementioned background technology, and aims to provide a method, program and device for obtaining a user-customized neural network model for identifying biometric information.

However, the problems to be solved in the present disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood based on the description below.

According to an embodiment of the present disclosure for achieving the above-described task, a method for obtaining a user-customized neural network model for identifying biometric information, performed by a computing device including at least one processor, comprises the steps of inputting a biometric signal detected from a user into a pre-learned neural network model to predict biometric information related to a state of the user, and when new data about the user is acquired, updating the pre-learned neural network model to be personalized based on the acquired new data, wherein the new data includes biometric information measured from the user in relation to the state.

Alternatively, the updating step includes a step of calibrating the learned neural network model to be personalized based on the measured biometric information and the biometric signal detected in response to the measured biometric information.

Alternatively, the correcting step includes performing supervised learning based on biometric information measured from the user and biometric signals detected in response to the biometric information measured from the user.

Alternatively, the updating step includes, when new data about the user is acquired, adding the acquired new data to a data set acquired by accumulating the acquired new data at a previous point in time, and updating the neural network model to be personalized based on the data set, wherein the data set includes a plurality of biometric information accumulated and measured from the user at the previous point in time.

Alternatively, the method comprises the step of identifying biological information of the user, and transmitting the new data to a computing device of another user having similar biological information to the identified biological information, such that the computing device of the other user is updated based on the new data.

Alternatively, the bio-signal includes an electrocardiogram (ECG) signal, and the bio-information includes left ventricular ejection fraction (EF).

Alternatively, the method comprises the step of outputting information warning the user of reduced left ventricular ejection fraction heart failure if the left ventricular ejection fraction is below a preset value.

Alternatively, the updating step includes a step of monitoring a trend of the predicted biometric information, and if a change in the trend of the predicted biometric information is detected, updating the pre-learned neural network model to be personalized based on the acquired new data.

Alternatively, the updating step includes a step of updating the pre-trained neural network model to be personalized based on the acquired new data, if it is identified that a preset time has elapsed since the previous update time.

Alternatively, the updating step includes a step of updating the pre-learned neural network model to be personalized based on the acquired new data if the number of times biometric information related to the user's status has been predicted from a previous update point in time is greater than or equal to a preset value.

According to one embodiment of the present disclosure for achieving the above-described task, a computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs an operation of obtaining a user-customized neural network model for identifying biometric information, the operation including an operation of inputting a biometric signal detected from the user into a pre-learned neural network model to predict biometric information related to a state of the user, and an operation of updating the pre-learned neural network model to be personalized based on new data obtained about the user, wherein the new data includes biometric information measured from the user in relation to the state.

According to one embodiment of the present disclosure for realizing the task as described above, a computing device for obtaining a user-customized neural network model for identifying biometric information comprises a memory including program codes, a network unit, a sensing unit for detecting biometric information of a user, and a processor for inputting a biometric signal detected from the user into a pre-learned neural network model to predict biometric information related to a state of the user, and when new data about the user is acquired, updating the pre-learned neural network model to be personalized based on the acquired new data, wherein the new data includes biometric information measured from the user in relation to the state.

According to a method for obtaining a user-customized neural network model for identifying biometric information according to an embodiment of the present disclosure, a neural network model for identifying more accurate biometric information about a user by reflecting the characteristics of the user can be obtained. In addition, a neural network model stored in a computing device can be personalized for the user simply by obtaining biometric information about the user.

FIG. 1 is an exemplary diagram of a device for obtaining a user-customized neural network model for identifying biometric information according to one embodiment of the present disclosure.

FIG. 2 is a block diagram of a computing device according to an embodiment of the present disclosure.

FIG. 3 is a flowchart schematically illustrating a method for obtaining a user-customized neural network model for identifying biometric information according to one embodiment of the present disclosure.

FIG. 4 is an exemplary diagram illustrating updating a neural network model based on first and second biometric information according to one embodiment of the present disclosure.

FIG. 5 is a flowchart illustrating updating a neural network model based on first and second biometric information according to one embodiment of the present disclosure.

FIG. 6 is an exemplary diagram illustrating updating a neural network model based on first biometric information and a plurality of second biometric information included in a data set according to one embodiment of the present disclosure.

FIG. 7 is a block diagram of a computing device according to another embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. The embodiments presented in the present disclosure are provided so that those skilled in the art can utilize or implement the contents of the present disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be implemented in various different forms and is not limited to the embodiments below.

Throughout the specification of the present disclosure, the same or similar drawing reference numerals refer to the same or similar components. In addition, in order to clearly describe the present disclosure, drawing reference numerals of parts that are not related to the description of the present disclosure may be omitted in the drawings.

The term "or" as used herein is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified herein or the context makes clear, "X employs either A or B" should be understood to mean either one of the natural inclusive permutations. For example, unless otherwise specified herein or the context makes clear, "X employs A or B" can be interpreted to mean either X employs A, X employs B, or X employs both A and B.

The term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated concepts listed.

The terms "comprises" and/or "comprising" as used herein should be understood to mean the presence of particular features and/or components. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, other components, and/or combinations thereof.

Unless otherwise specified in this disclosure or unless the context makes it clear that the singular form is intended to be referred to, the singular should generally be construed to include “one or more.”

The term "Nth (N is a natural number)" used in the present disclosure can be understood as an expression used to mutually distinguish components of the present disclosure according to a predetermined standard such as a functional viewpoint, a structural viewpoint, or convenience of explanation. For example, components performing different functional roles in the present disclosure can be distinguished as a first component or a second component. However, components that are substantially the same within the technical idea of the present disclosure but should be distinguished for convenience of explanation may also be distinguished as a first component or a second component.

The term "acquisition" as used in this disclosure may be understood to mean not only receiving data via a wired or wireless communication network with an external device or system, but also generating data in an on-device form.

Meanwhile, the term "module" or "unit" used in the present disclosure may be understood as a term referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, a combination of software and hardware, etc. At this time, the "module" or "unit" may be a unit composed of a single element, or may be a unit expressed as a combination or set of multiple elements. For example, as a narrow concept, a "module" or "unit" may refer to a hardware element of a computing device or a set thereof, an application program that performs a specific function of software, a processing process implemented through software execution, or a set of instructions for program execution, etc. In addition, as a broad concept, a "module" or "unit" may refer to a computing device itself that constitutes a system, or an application that is executed on a computing device, etc. However, the above-described concept is only an example, and the concept of “module” or “part” may be variously defined within a category understandable to those skilled in the art based on the contents of the present disclosure.

The term "model" used in the present disclosure may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or an abstract model regarding a processing process to solve a specific problem. For example, a neural network "model" may refer to the entire system implemented as a neural network that has a problem-solving ability through learning. In this case, the neural network may have a problem-solving ability by optimizing parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network, or may include a neural network set in which multiple neural networks are combined.

The term "data" used in the present disclosure may include "images", signals, etc. The term "image" used in the present disclosure may refer to multidimensional data composed of discrete image elements. In other words, "image" may be understood as a term referring to a digital representation of an object that can be seen with the human eye. For example, "image" may refer to multidimensional data composed of elements corresponding to pixels in a two-dimensional image. "Image" may refer to multidimensional data composed of elements corresponding to voxels in a three-dimensional image.

The explanation of the terms set forth above is intended to aid in understanding the present disclosure. Therefore, if the terms set forth above are not explicitly stated as matters limiting the contents of the present disclosure, it should be noted that they are not used to limit the technical ideas of the contents of the present disclosure.

FIG. 1 is an exemplary diagram of a device for obtaining a user-customized neural network model (10) for identifying biometric information according to one embodiment of the present disclosure.

The computing device (100) according to one embodiment of the present disclosure may be a hardware device or a part of a hardware device that performs comprehensive processing and calculation of data, or may be a software-based computing environment connected to a communication network. For example, the computing device (100) may be a server that performs an intensive data processing function and shares resources, or may be a client that shares resources through interaction with a server. In addition, the computing device (100) may be a cloud system in which a plurality of servers and clients interact to comprehensively process data. Since the above description is only one example related to the type of the computing device (100), the type of the computing device (100) may be configured in various ways within a category understandable to those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 1, a computing device (100) according to an embodiment of the present disclosure may be implemented as a smart watch. However, the present disclosure is not limited thereto, and the computing device (100) may be implemented as various electronic devices such as a desktop, a laptop, a smart phone, a server device, a smart band, a smart ring, etc. In addition, the computing device (100) may be implemented as a biosignal measuring device that measures bioinformation of a user (1) (e.g., a patient, etc.). In the following, for the purpose of understanding the present disclosure, the computing device (100) will be described assuming a smart watch.

Meanwhile, a pre-learned neural network model (10) may be stored in the computing device (100). The neural network model (10) may be a model that has been learned in advance to identify the user's (1) condition, symptoms, disease, biometric information, etc. In particular, the neural network model (10) may be learned in advance according to the type, purpose of use, etc. of the computing device (100).

According to one embodiment of the present disclosure, a neural network model (10) may be pre-trained to identify another biometric information of a user (1) based on biometric information obtained from the user (1) by the computing device (100).

Here, the bio-information may include various bio-signals (e.g., electrocardiogram (ECG), electroencephalogram (EEG), photoplethysmogram (PPG), arterial blood pressure (ABP), central venous pressure (CVP)) obtained from the user (1). In addition, the bio-information may include diseases (e.g., heart failure, etc.) and disease potentials identified for the user (1), and measurements indicating the status of the user (1) (e.g., body temperature, blood pressure, left ventricular ejection fraction (EF), etc.).

Hereinafter, for the convenience of explanation of the present disclosure, biometric information acquired by a computing device (100) and input to a neural network model (10) is referred to as first biometric information, and data (i.e., output data) output through the neural network model (10) using the acquired biometric signal as input data is referred to as second biometric information.

According to one embodiment of the present disclosure, the first biometric information and the second biometric information may be different types of biometric information. The user (1) can identify the status of the user (1) with the first biometric information and the second biometric information. For example, the first biometric information may be an electrocardiogram signal, and the second biometric information may be a left ventricular ejection fraction. Through this, the user (1) can monitor the left ventricular ejection fraction and examine the disease and health status of the user (1) in real time without visiting a specialized medical institution, simply by using the computing device (100). In particular, the disease and health status of the user (1) may be related to the heart of the user (1).

As another example, the first biometric information may include biometric information obtained from the user (1) based on a non-invasive measurement method. For example, the first biometric information may include an electrocardiogram signal, an electroencephalogram signal, a photosensitive pulse wave, etc. On the other hand, the second biometric information may include biometric information obtained from the user (1) based on an invasive measurement method or biometric information obtained by direct measurement by medical staff. For example, the second biometric information may include body temperature, blood pressure (e.g., arterial blood pressure, central venous blood pressure, etc.), left ventricular ejection fraction, etc. That is, the neural network model (10) may be trained to identify the second biometric information obtained based on a non-invasive method that requires intervention by a medical professional or is generally difficult to measure, with the first biometric information obtained based on a non-invasive method that can be easily measured from the user (1). Through this, the user (1) can monitor the second biometric information without visiting a medical institution by only using the computing device (100) and can examine the condition of the user (1) in real time.

Meanwhile, the neural network model (10) may be implemented as a multi-layer perceptron (MLP), a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network, etc. In particular, the neural network model (10) may include a plurality of residual blocks that extract potential features of input first bio-information and a classifier that classifies second bio-information or a specific disease based on the extracted potential features.

Meanwhile, the computing device (100) may be manufactured in a state in which the neural network model (10) is stored. Alternatively, the computing device (100) may obtain the neural network model (10) from an external computing device (e.g., a server device, etc.) that is connected to the computing device (100). At this time, the computing device (100) may update the neural network model (10) stored in the computing device (100) to suit the user (1). Specifically, the neural network model (10) obtained by the computing device (100) may be a model that has been learned in advance based on a preset learning data set. At this time, the computing device (100) may re-learn the neural network model (10) based on new data obtained from the user (1) and update it to suit the user (1).

Here, updating the neural network model (10) may be to correct or rectify the neural network model (10) based on the biometric information acquired by the computing device (100). In particular, it may be a process of personalizing the neural network model (10) to suit the user. For example, the computing device (100) may perform a learning process of fine-tuning the neural network model (10) based on the acquired biometric information.

For example, referring to FIG. 1, if the neural network model (10) is a model learned to identify second biometric information based on first biometric information, the computing device (100) can re-learn and update the neural network model (10) based on the first biometric information acquired from the user (1). More specifically, if the first biometric information is an electrocardiogram signal and the second biometric information is a left ventricular ejection fraction, when the computing device (100) acquires an electrocardiogram signal of the user (1), it inputs the signal into the neural network model (10) to identify the left ventricular ejection fraction of the user (1), and also learns the neural network model (10) based on the acquired electrocardiogram signal to update the neural network model (10) to fit the user (1). By repeating this process, the computing device (100) can obtain a personalized neural network model (10) (i.e., a user-tailored neural network model (10)) that more accurately identifies the second biometric information (e.g., left ventricular ejection fraction) of the user (1) based on the first biometric information (e.g., electrocardiogram signal) of the user (1).

Meanwhile, referring to FIG. 1, the computing device (100) may also obtain second biometric information measured from the user. Here, the second biometric information may be obtained from an external computing device. That is, the second biometric information may be obtained directly from the user (1) by the external computing device, or may be a value calculated based on a biometric signal obtained from the user (1) by the external computing device. The second biometric information obtained from the external computing device is different from the second biometric information obtained by inputting the first biometric information into the neural network model (10). The second biometric information obtained from the external computing device may be more accurate than the second biometric information output from the neural network model (10) in that it is information directly measured from the user (1). The computing device (100) may also update the neural network model (10) based on the first biometric information obtained by directly measuring the user (1) by the computing device (100) together with the obtained second biometric information. In particular, since the acquired second biometric information can be labeled learning data, the computing device (100) can update the neural network model (10) to more accurately identify the second biometric information for the user (1) using the first and second biometric information. For example, if the computing device (100) acquires the left ventricular ejection fraction value for the user (1) from an external device, the computing device (100) can update the neural network model (10) with the acquired left ventricular ejection fraction value and the user's electrocardiogram signal corresponding to the acquired left ventricular ejection fraction.

Hereinafter, an embodiment of the present disclosure will be described in detail with reference to FIGS. 2 to 6.

FIG. 2 is a block diagram of a computing device (100) according to one embodiment of the present disclosure.

Referring to FIG. 2, a computing device (100) according to an embodiment of the present disclosure may include one or more processors (hereinafter, “processors”) (110), a memory (120), a network unit (130), and a sensing unit (140). However, FIG. 1 is only an example, and thus, the computing device (100) may further include other configurations for implementing a computing environment. In addition, only some of the disclosed configurations may be included in the computing device (100).

The processor (110) according to one embodiment of the present disclosure may be understood as a configuration unit including hardware and/or software for performing computing operations. For example, the processor (110) may read a computer program to perform data processing for machine learning. The processor (110) may process computational processes such as processing of input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. The processor (110) for performing such data processing may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). The type of the processor (110) described above is only one example, and thus, the type of the processor (110) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The processor (110) is electrically connected to other components of the computing device (100) (i.e., memory (120), network unit (130), and sensing unit (140)) and controls the overall operation of the computing device (100).

The memory (120) according to one embodiment of the present disclosure may be understood as a configuration unit including hardware and/or software for storing and managing data processed in the computing device (100). That is, the memory (120) may store any type of data generated or determined by the processor (110) and any type of data received by the network unit (130). For example, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, a RAM (random access memory), a SRAM (static random access memory), a ROM (read-only memory), an EEPROM (electrically erasable programmable read-only memory), a PROM (programmable read-only memory), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory (120) may also include a database system that controls and manages data in a predetermined system. The type of memory (120) described above is only an example, and thus the type of memory (120) can be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The memory (120) can manage, by structuring and organizing, data required for the processor (110) to perform operations, combinations of data, and program codes executable in the processor (110). For example, the memory (120) can store a neural network model (10) learned to identify second biometric information based on first biometric information. In addition, the memory (120) can store program codes that operate to perform learning on the neural network model (10) based on the acquired first and second biometric information, program codes that operate the neural network model (10) to receive time series data and perform inference according to the intended use of the computing device (100), and processed data generated as the program codes are executed.

The network unit (130) according to one embodiment of the present disclosure may be understood as a configuration unit that transmits and receives data via any type of known wired and wireless communication system. For example, the network unit (130) may perform data transmission and reception using a wired and wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), fifth generation mobile communication (5G), ultrawide-band, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity, near field communication (NFC), or Bluetooth. Since the above-described communication systems are only examples, the wired and wireless communication system for data transmission and reception of the network unit (130) may be applied in various ways other than the above-described examples.

The network unit (130) can receive data required for the processor (110) to perform calculations through wired or wireless communication with any system or any client, etc. In addition, the network unit (130) can transmit data generated through the calculation of the processor (110) through wired or wireless communication with any system or any client, etc. For example, the network unit (130) can receive medical data through communication with a cloud server that performs tasks such as standardization of medical data, a database in a hospital environment, or a computing device, etc. The network unit (130) can transmit output data of the neural network model (10), and intermediate data, processed data, etc. derived from the calculation process of the processor (110) through communication with the aforementioned database, server, or computing device, etc. As an example, the processor (110) can obtain a neural network model (10) learned to identify second biometric information based on first biometric information from an external computing device (e.g., an external server device) through the network unit (130). Additionally, the processor (110) can obtain second biometric information of the user (1) from an external computing device (e.g., a biometric signal measuring device) through the network unit (130).

The sensing unit (140) can obtain first biometric information about the user (1). In particular, the sensing unit (140) can obtain first biometric information about the user (1) based on a non-invasive method. That is, according to one embodiment of the present disclosure, the first biometric information may be biometric information measured based on a non-invasive method. As an example, the sensing unit (140) may include a plurality of electrodes. At this time, the processor (110) may obtain an electrocardiogram signal of the user (1) as the first biometric information through at least one electrode. In addition, the sensing unit (140) may include an image sensor or an optical sensor. At this time, the processor (110) may obtain a photoplethysmographic blood flow signal of the user (1) as the first biometric information through the image sensor (or optical sensor).

FIG. 3 is a flowchart schematically illustrating a method for obtaining a user-customized neural network model (10) for identifying biometric information according to one embodiment of the present disclosure.

Referring to FIG. 3, first, the processor (110) inputs a bio-signal detected from a user into a pre-learned neural network model (10) to predict bio-information related to the state (S310).

The processor (110) can obtain a neural network model (10) stored in the memory (120). At this time, the neural network model (10) can be stored in the memory (120) at the time of manufacturing the computing device (100).

At this time, a plurality of neural network models (10) may be stored in the memory (120) of the computing device (100). Specifically, a plurality of neural network models (10) corresponding to a plurality of users (1) (e.g., patients, etc.) may be stored in advance in the memory (120), and the processor (110) may select and acquire a neural network model (10) corresponding to the user (1) among the plurality of neural network models (10) based on the identification information (e.g., name, resident registration number, etc.) (or biological information) of the user (1). For example, if the computing device (100) is a biosignal measuring device installed in a specialized medical institution, a plurality of neural network models (10) corresponding to a plurality of patients who have visited the specialized medical institution may be stored in the computing device (100), and each neural network model (10) may be repeatedly updated based on the first and second biometric information acquired for each patient according to an embodiment of the present disclosure described below, and may be stored in the memory (120).

Meanwhile, the processor (110) may also obtain a neural network model (10) from an external computing device (200) (e.g., a server device, etc.) that is connected to the computing device (100) through the network unit (130).

Here, the external computing device (200) may be a server device that monitors the status (i.e., disease and health status) of the user (1) of the computing device (100) in conjunction with the computing device (100). Alternatively, the external computing device (200) may be a biosignal measuring device that measures second bioinformation of the user (1).

And, the processor (110) can detect first bio-information about the user through the sensing unit (140). For example, the processor (110) can detect and obtain a bio-signal as the first bio-information about the user through the sensing unit (140), and at this time, the bio-signal can be an electrocardiogram signal. When a command requesting prediction of the left ventricular ejection fraction of the user (1) is input, the processor (110) can detect and obtain an electrocardiogram signal about the user through the sensing unit (140).

When the first biometric information is acquired, the processor (110) can input the first biometric information into the neural network model (10) to predict the second biometric information. In particular, the second biometric information may be related to the status of the user (1). Here, the status of the user (1) may be a disease or health status of the user (1). In particular, the disease or health status of the user (1) may be related to the heart of the user (1). The neural network model (10) may be a model learned to predict the second biometric information based on the first biometric information, as described in FIG. 1. For example, if the first biometric information is an electrocardiogram signal and the second biometric information is a left ventricular ejection fraction, the neural network model (10) may be learned based on a plurality of learning data sets in which the electrocardiogram signal and the left ventricular ejection fraction corresponding to the electrocardiogram signal are matched. And, the processor (110) inputs an electrocardiogram signal detected from the user (1) into the neural network model (10) to predict the left ventricular ejection fraction of the user (1) related to heart failure with reduced left ventricular ejection fraction.

Again, referring to FIG. 3, according to one embodiment of the present disclosure, when new data about a user is acquired, the processor (110) may update the pre-learned neural network model to be personalized based on the acquired new data (S320). Here, the new data may include biometric information measured from the user in relation to the condition of the user (1) (e.g., disease and health condition). For example, the new data may be left ventricular ejection fraction measured from the user in relation to heart failure with reduced left ventricular ejection fraction.

The processor (110) can obtain second biometric information about the user (1), and perform first learning on the neural network model (10) based on the obtained second biometric information and first biometric information corresponding to the obtained second biometric information, thereby updating the neural network model (10).

Specifically, the processor (110) may receive second biometric information from an external computing device (200) through the network unit (130). Alternatively, the processor (110) may also receive second biometric information through a user interface. At this time, while the processor (110) detects and acquires first biometric information about the user (1), the processor (110) may acquire second biometric information from the external computing device (200) through the network unit (130) or through the user interface. The second biometric information is biometric information directly measured and acquired from the user (1), and is different from and may be more accurate than second biometric information predicted by inputting first biometric information into the neural network model (10).

For example, the second biometric information may be the left ventricular ejection fraction measured for the user (1) by the external computing device (200). More specifically, the external computing device (200) may observe the internal structure of the heart of the user (1) through an ultrasound examination, evaluate the size and shape of the left ventricle of the user (1), and measure the left ventricular ejection fraction. However, the present invention is not limited thereto, and the external computing device (200) may measure the left ventricular ejection fraction of the user (1) based on various methods such as the Doppler measurement method and the red wave emission tracing method. The left ventricular ejection fraction for the user (1) may be measured by medical staff using the external computing device (200).

Meanwhile, as another example, the second biometric information may include biometric information obtained from the user (1) based on an invasive measurement method or biometric information obtained by direct measurement by medical staff.

When second biometric information is acquired, the processor (110) can perform second learning on the neural network model (10) based on the acquired second biometric information to update the neural network model (10). That is, the processor (110) can correct the previously learned neural network model (10) to fit (or be personalized) the user (1) based on the second biometric information.

Specifically, when second biometric information is acquired, the processor (110) can identify first biometric information (hereinafter, first biometric information corresponding to the second biometric information) acquired through the sensing unit (140) at the time when the second biometric information is acquired (more specifically, at the time when the second biometric information is acquired by the external computing device (200), and perform second learning on the neural network model (10) using the first and second biometric information. For example, when the first biometric information is an electrocardiogram signal and the second biometric information is a left ventricular ejection fraction, the processor (110) can identify an electrocardiogram signal detected from the user (1) when the external computing device (200) measures the left ventricular ejection fraction of the user (1), and perform first learning on the neural network model (10) using the left ventricular ejection fraction acquired from the external computing device (200) and the identified electrocardiogram signal.

According to one embodiment of the present disclosure, the first learning may be a supervised learning method based on the acquired second biometric information and the acquired first biometric information corresponding to the acquired second biometric information. Specifically, the processor (110) may set the first biometric information corresponding to the second biometric information as input data and the second biometric information as a label, thereby allowing the neural network model (10) to learn the relationship between the first and second biometric information.

Meanwhile, according to one embodiment of the present disclosure, the processor (110) can continuously obtain first biometric information about the user (1). Then, the processor (110) can perform second learning on the neural network model (10) based on the obtained first biometric information, thereby updating the neural network model (10).

Specifically, the processor (110) can continuously obtain first biometric information about the user (1) through the sensing unit (140). For example, the processor (110) can set a plurality of sample points (e.g., 125 points or 250 points) for a preset time (e.g., 1 second) and repeatedly obtain first biometric information about the user (1) for each of the plurality of sample points. When the first biometric information is an electrocardiogram signal, the processor (110) can obtain the electrocardiogram signal by measuring it at a sampling rate of 500 Hz for 10 seconds through the sensing unit (140). At this time, the processor (110) can process the electrocardiogram signal by removing a 1-second region where the electrocardiogram signal starts and a last 1-second region where the electrocardiogram signal ends in order to remove noise or artifacts included in the electrocardiogram signal.

In addition, the processor (110) continuously obtains the first biometric information of the user (1) through the sensing unit (140), and the processor (110) may obtain a plurality of pieces of first biometric information by dividing the continuous first biometric information based on the set sample points.

Meanwhile, the processor (110) may periodically obtain first biometric information from an external computing device (200). For example, if the computing device (100) is implemented as a server device, the processor (110) may continuously receive first biometric information obtained from an external biometric signal measuring device that measures first biometric information by coming into contact with the body of the user (1). At this time, the processor (110) may receive first biometric information from the external biometric signal measuring device through the network unit (130).

When the first biometric information is acquired, the processor (110) can perform second learning on the neural network model (10) based on the acquired first biometric information of the user (1), thereby updating the neural network model (10). That is, the processor (110) can correct the previously learned neural network model (10) to fit the user (1) based on the first biometric information.

For example, when the processor (110) periodically acquires the first biometric information, the processor (110) may perform second learning on the neural network model (10) with the acquired first biometric information every time the processor (110) acquires the first biometric information. That is, the second learning may be performed on the neural network model (10) in accordance with the cycle in which the first biometric information is acquired.

According to one embodiment of the present disclosure, the second learning may be a self-supervised learning method based on the acquired first biometric information. Specifically, the processor (110) may extract latent features from the acquired first biometric information, assign a virtual label to the first biometric information based on the extracted latent features, and then train the neural network model (10) based on the first biometric information to which the virtual label has been assigned.

FIG. 4 is an exemplary diagram showing updating a neural network model (10) based on first and second biometric information according to an embodiment of the present disclosure. FIG. 5 is a flowchart showing updating a neural network model (10) based on first and second biometric information according to an embodiment of the present disclosure.

Meanwhile, referring to FIG. 4, for example, the first learning may be performed whenever the second biometric information is acquired, and the second learning may be performed whenever the first biometric information is acquired. At this time, when the second biometric information is acquired, the processor (110) may perform only the first learning without performing the second learning for the neural network model (10). This is to increase the accuracy of the neural network model (10) by not using the same first biometric information for each different learning (first and second learning), but only using it for the first learning that enables more accurate correction of the neural network model (10). However, this is not limited thereto.

Meanwhile, referring to FIG. 5, S510 illustrated in FIG. 5 may correspond to S310 illustrated in FIG. 3. Meanwhile, referring to FIG. 5, the processor (110) may perform the first learning and second learning processes in parallel (S522 and S524) by obtaining the first and second biometric information, respectively (S521 and S523).

At this time, the processor (110) may perform second learning on the neural network model (10) with the first biometric information of the user (1) obtained periodically based on Equations 1 to 3 below, and may perform first learning on the neural network model (10) with the second biometric information obtained. Specifically, the processor (110) may update the neural network model (10) based on the gradient descent method according to Equations 1 to 3 below. At this time, the processor (110) may perform learning with batch size 1 using data of the time step t of the neural network model (10).

<Formula 1>

<Formula 2>

<Formula 3>

는 시간 t에서의 신경망 모델의 파라미터일 수 있다.

는 t에서 획득된 제1 생체 정보이고,

는 t에서 획득된 제2 생체 정보일 수 있다. 그리고, L은 신경망 모델의 전체 손실 함수이고,

은 지도 학습 손실 함수이고,

은 자기 지도 학습 손실 함수일 수 있다.

은 t에서의 획득된 제1 생체 정보로 예측된 제2 생체 정보와 실제 t에서 획득된 제2 생체 정보의 엔트로피 손실(Cross-Entropy Loss)(

)로 정의되고,

은

를 임베딩(Embedding)한 결과인

와 입력 데이터

간의 유클리드 거리를 최소화하는 손실 함수로 정의될 수 있다. Here,

can be a parameter of the neural network model at time t.

is the first biometric information acquired from t,

can be the second biometric information obtained from t. And, L is the overall loss function of the neural network model,

is the supervised learning loss function,

can be a self-supervised learning loss function.

is the cross-entropy loss of the second biometric information predicted by the first biometric information acquired at t and the second biometric information actually acquired at t.

) is defined as,

silver

The result of embedding

and input data

It can be defined as a loss function that minimizes the Euclidean distance between the two.

FIG. 6 is an exemplary diagram showing updating a neural network model (10) based on first biometric information and a plurality of second biometric information included in a data set according to one embodiment of the present disclosure.

Meanwhile, when new data about a user is acquired according to an embodiment of the present disclosure, the processor (110) may add the acquired new data to a data set accumulated at a previous time and perform first learning on the neural network model (10) based on the data set, thereby updating the neural network model to be personalized. At this time, the data set may include at least one second biometric information accumulated at a previous time and first biometric information corresponding to at least one second biometric information that is matched and acquired.

Specifically, referring to FIG. 6, when second biometric information is newly acquired, the processor (110) may identify first biometric information acquired at the time when the second biometric information is acquired, and add the second biometric information and the first biometric information to a data set (20) accumulated and acquired at a previous time. Here, the data set (20) may include a plurality of second biometric information accumulated and acquired before the time when the second biometric information is newly acquired, and a plurality of first biometric information corresponding to the plurality of second biometric information, respectively. That is, when second biometric information is acquired, the processor (110) may update the data set (20) with the acquired second biometric information and the first biometric information corresponding to the acquired second biometric information. In addition, the processor (110) may perform first learning on the neural network model (10) based on the data set (20) to update the neural network model (10).

At this time, the processor (110) may perform second learning on the neural network model (10) with the first biometric information of the user (1) obtained periodically based on Equations 4 to 7 below, and may perform first learning on the neural network model (10) with the data set (20) including the plurality of second biometric information obtained. Specifically, the processor (110) may update the neural network model (10) based on the gradient descent method according to Equations 4 to 7 below.

<Formula 4>

<Formula 5>

<Formula 6>

<Formula 7>

는 시간 t에서의 신경망 모델의 파라미터일 수 있다.

는 t에서 획득된 제1 생체 정보이고,

는 t에서 획득된 제2 생체 정보일 수 있다. 그리고, L은 신경망 모델의 전체 손실 함수이고,

은 지도 학습 손실 함수이고,

은 자기 지도 학습 손실 함수일 수 있다.

은 t에서의 획득된 제1 생체 정보로 예측된 제2 생체 정보와 실제 t에서 획득된 제2 생체 정보의 엔트로피 손실(Cross-Entropy Loss)(

)로 정의되고,

은

를 임베딩(Embedding)한 결과인

와 입력 데이터

간의 유클리드 거리를 최소화하는 손실 함수로 정의될 수 있다.

는 데이터 셋(20)일 수 있다. Here,

can be a parameter of the neural network model at time t.

is the first biometric information acquired from t,

can be the second biometric information obtained from t. And, L is the overall loss function of the neural network model,

is the supervised learning loss function,

can be a self-supervised learning loss function.

is the cross-entropy loss of the second biometric information predicted by the first biometric information acquired at t and the second biometric information actually acquired at t.

) is defined as,

silver

The result of embedding

and input data

It can be defined as a loss function that minimizes the Euclidean distance between the two.

may be a data set (20).

Meanwhile, the processor (110) may identify biological information of the user (1) and share a data set (20) related to second biometric information with another computing device (100) of another user (1) having similar biological information to the identified biological information. Here, the biological information may include the height, weight, age, gender, etc. of the user (1). For example, when the computing device (100) is a smart watch, the processor (110) may obtain biological information of the other user (1) from information of another smart watch of the other user (1). The processor (110) may identify the similarity between the biological information of a plurality of other users (1) and the biological information of the user (1), and may identify another computing device (100) of another user (1) having similar biological information to the biological information of the user (1) based on the identified similarity. And, the processor (110) can share the data set (20) related to the second biometric information by transferring the acquired data set (20) to the identified other computing device (100). Alternatively, the processor (110) can also receive the data set (20) related to the second biometric information of the other user (1) from the other computing device (100) of the other user (1) having similar biological information to the biological information of the user (1). In particular, in the case of the second biometric information, it is acquired intermittently as it is measured by a non-invasive method or by medical staff, and the number of data may be small. Therefore, in the case of a plurality of users (1) having similar biological information, the neural network model (10) can be updated to more quickly reflect the characteristics of the user (1) by sharing the acquired data set (20) related to the second biometric information.

For example, a server device that is linked with a computing device (100) may obtain biological information of a user (1) from the computing device (100) and transmit a neural network model (10) corresponding to the obtained biological information to the computing device (100). Specifically, a plurality of neural network models (10) corresponding to a plurality of users (1) may be stored in the server device. In particular, each neural network model (10) may correspond to a user (1) group including a plurality of users (1) having similar biological information. Accordingly, when the server device receives biological information of a user (1) from the computing device (100), the server device may select biological information similar to the received biological information and transmit a neural network model (10) corresponding to the selected biological information to the computing device (100). At this time, the server device may also include the user (1) of the computing device (100) in the user (1) group corresponding to the selected biological information. The server device can share second biometric information (or a data set (20) established based on the second biometric information) obtained from each computing device (100) with multiple users (1) included in the same user (1) group.

According to one embodiment of the present disclosure, the processor (110) can identify biological information of a user (1), cluster the user (1) and a plurality of other users (1) based on the identified biological information, and allocate a data set (20) composed of new data to each clustering group. Specifically, the processor (110) can obtain biological information of a plurality of users (1) from a plurality of user (1) terminal devices that are linked to the computing device (100). At this time, the processor (110) can cluster the plurality of users (1) based on the biological information of the plurality of users (1). Then, the processor (110) can set a data set (20) for each clustering group with a second biosignal obtained from a user (1) terminal device included in each clustering group. That is, when the second biosignal is obtained from each of the plurality of user (1) terminal devices included in a specific clustering group, the processor can collect the obtained biosignals, set a data set (20) for a specific clustering group, and allocate the collected biosignals to a specific clustering group. At this time, the computing device (100) may be implemented as a server device or the like that manages the status (e.g., disease and health status) of multiple users (1).

According to one embodiment of the present disclosure, the processor (110) can input the acquired first biometric information of the user (1) into the neural network model (10) to predict second biometric information about the user (1). This may be performed through the second learning (i.e., self-supervised learning) process described above. Then, the processor (110) monitors the trend of the predicted second biometric information, and if it is detected that the trend of the predicted second biometric information has changed, the processor (110) can acquire the second biometric information from the external computing device (200). Alternatively, the processor (110) can acquire the second biometric information acquired from the external computing device (200) that is accumulated and stored in a memory. Then, the processor (110) can perform first learning on the neural network model (10) based on the acquired second biometric information to update the neural network model (10). That is, if the processor (110) detects that the trend or tendency of the second biometric information has changed, it may determine to perform an update on the neural network model (10) based on the second biometric information. For example, if the processor (110) determines that the similarity between the trend of the second biometric information of the previous period and the trend of the second biometric information of the current period is less than a preset value, it may determine to perform first learning on the neural network model (10) using the second biometric information, identifying that the trend of the second biometric information has changed.

In addition, according to one embodiment of the present disclosure, if the processor (110) identifies that a preset time has passed since the previous update time, the processor (110) may update the pre-learned neural network model to be personalized based on the acquired new data. Here, the previous update time may be the time at which the update was most recently performed. That is, if the processor (110) identifies that a preset time has passed since the most recently performed update time, the processor (110) may determine to perform an update process for the pre-learned neural network model (10). For example, assuming that the preset time is 7 days, the processor (110) may perform an update process for the neural network model (10) if it is identified that 10 days have passed since the most recently updated neural network model (10).

In addition, according to one embodiment of the present disclosure, if the number of times the biometric information related to the user's status has been predicted from a previous update time is greater than or equal to a preset value, the processor (110) may update the pre-learned neural network model (10) to be personalized based on the acquired new data. Here, the previous update time may be the time at which the most recent update was performed. For example, assuming that the preset value is 20 times, if the processor (110) identifies that the number of times the second biometric information (e.g., left ventricular ejection fraction) has been predicted through the neural network model (10) from the time at which the neural network model (10) was most recently updated is 25 times, the processor (110) may perform an update process for the neural network model (10).

Meanwhile, if the processor (110) identifies that a preset time has passed since the previous update time, it can identify whether the number of times biometric information related to the user's status has been predicted from the previous update time is greater than or equal to a preset value and determine whether to perform an update process for the neural network model (10).

According to one embodiment of the present disclosure, when second biometric information is acquired, the processor (110) inputs first biometric information corresponding to the second biometric information into the neural network model (10), acquires the second biometric information as output data, and identifies an error with the acquired second biometric information. At this time, if the error between the second biometric information acquired as output data and the second biometric information acquired from another computing device (100) is less than a preset value, the processor (110) may determine to stop an update process (e.g., a first learning process) regarding the neural network model (10). This is to determine that the personalization process of the neural network model (10) for the user (1) is completed, and to save resources of the computing device (100) required for learning.

Additionally, according to one embodiment of the present disclosure, if the processor (110) identifies that the user (1) has changed based on the acquired first biometric information, the processor (110) may stop the update process and change the neural network model (10) to a new neural network model to resume the update process.

According to one embodiment of the present disclosure, when the first biometric information includes electrocardiogram (ECG) information and the second biometric information includes left ventricular ejection fraction (EF), the processor (110) inputs the ECG information into the neural network model (10) to identify the left ventricular ejection fraction of the user (1), and predicts heart failure with reduced left ventricular ejection fraction of the user (1) based on the identified left ventricular ejection fraction. For example, the processor (110) calculates a score corresponding to the left ventricular ejection fraction of the user (1) identified through the neural network model (10), and identifies the possibility of heart failure with reduced left ventricular ejection fraction of the user (1) based on the calculated score. At this time, if the left ventricular ejection fraction is lower than a preset value or the score is higher than a preset value, the processor (110) identifies that the user's (1) has a risk of heart failure with reduced left ventricular ejection fraction, and can output a warning message, etc. through an output interface (e.g., speaker, display, etc.).

FIG. 7 is a block diagram of a computing device (400) according to another embodiment of the present disclosure.

Referring to FIG. 7, the computing device (400) includes a processor (410), a memory (420), a network unit (430), a sensing unit (440), a display (450), a user interface (460), a camera (470), and a speaker (480). Among the configurations illustrated in FIG. 7, the processor (410), the memory (420), the network unit (430), and the sensing unit (440) correspond to the configurations of the processor (110), the memory (120), the network unit (130), and the sensing unit (140) of the computing device (100) illustrated in FIG. 2, and therefore, a detailed description thereof will be omitted.

The display (450) can display various images. Here, the images include both still images and moving images. The display (450) can output guide information about activities generated based on the user (1) status. The display (450) can be implemented as various types of displays, such as an LCD (Liquid Crystal Display Panel), an OLED (Organic Light Emitting Diodes), an LCoS (Liquid Crystal on Silicon), a DLP (Digital Light Processing), etc. In addition, the display (450) can also include a driving circuit, a backlight unit, etc., which can be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc.

Meanwhile, the display (450) may be implemented as a touch screen by being combined with a touch panel, and in this case, the display (450) may perform the function of an input interface that receives a touch input from a user (1) as well as an output interface that outputs an image through the touch screen.

The user interface (460) is a configuration used by the computing device (100) to perform interaction with the user (1), and may include at least one of a touch sensor, a motion sensor, a button, a jog dial, and a switch, but is not limited thereto. The processor (410) may receive second biometric information and biological information (occupation, age, gender, etc.) of the user (1) through the user interface (460).

The camera (470) photographs an object around the user (1) to obtain an image of the object. Specifically, the camera (470) can obtain an image of food consumed by the user (1). At this time, the processor (410) can determine the nutritional status of the user (1) based on the status of the user (1) and the image of the food consumed by the user (1) and provide recommended diet information as guide information. To this end, the camera (470) can be implemented as an imaging device such as an imaging device having a CMOS structure (CIS, CMOS Image Sensor) or an imaging device having a CCD structure (Charge Coupled Device). However, the present invention is not limited thereto, and the camera (470) can be implemented as a camera module having various resolutions capable of photographing a subject. Meanwhile, the camera (470) can be implemented as a depth camera (for example, an IR depth camera, etc.), a stereo camera, or an RGB camera.

The speaker (480) is a component that outputs various audio data on which various processing operations such as decoding, amplification, and noise filtering have been performed by an audio processing unit (not shown). The speaker (480) can output various notification sounds or voice messages. According to one embodiment of the present disclosure, the processor (410) can convert an electrical signal received from an external device into a user's (1) voice and output it through the speaker (480). As an example, the speaker (480) can output a voice message warning of or suggesting a diagnosis of a disease related to second biometric information identified through a neural network model (10).

The various embodiments of the present disclosure described above can be combined with additional embodiments and can be modified within a scope that can be understood by those skilled in the art in light of the detailed description set forth above. It should be understood that the embodiments of the present disclosure are illustrative in all respects and not restrictive. For example, each component described as a single component may be implemented in a distributed manner, and likewise, components described as distributed may be implemented in a combined form. Accordingly, all changes or modifications derived from the meaning, scope, and equivalent concept of the claims of the present disclosure should be interpreted as being included in the scope of the present disclosure.

Claims (12)

A method for obtaining a user-customized neural network model for identifying biometric information, the method being performed by a computing device including at least one processor, A step of inputting a bio-signal detected from a user into a pre-trained neural network model to predict bio-information related to the user's condition; and When new data about the user is acquired, a step of updating the pre-learned neural network model to be personalized based on the acquired new data is included; The above new data is, In relation to the above condition, including biometric information measured from the user, method. In the first paragraph, The above updating steps are: A step of correcting the learned neural network model to be personalized based on the measured biometric information and the biometric signal detected in response to the measured biometric information; Including, method. In the second paragraph, The above correction steps are: A step of performing supervised learning based on biometric information measured from the user and biometric signals detected in response to the biometric information measured from the user; Including, method. In the first paragraph, The above updating steps are: When new data about the user is acquired, the acquired new data is accumulated at a previous point in time and added to the acquired data set, and the neural network model is updated to be personalized based on the data set; The above data set includes multiple biometric information accumulated and measured from the user at the above previous point in time. method. In the first paragraph, A step of identifying biological information of the user and transmitting the new data to a computing device of another user having similar biological information to the identified biological information so that the computing device of the other user is updated based on the new data; Including, method. In the first paragraph, The above biosignals are, Contains an electrocardiogram (ECG) signal, The above biometric information is, Including left ventricular ejection fraction (EF), method. In Article 6, A step of outputting information warning the user of heart failure due to reduced left ventricular ejection fraction when the left ventricular ejection fraction is below a preset value; method. In the first paragraph, The above updating steps are: A step of monitoring the trend of the predicted biometric information, and when it is detected that the trend of the predicted biometric information has changed, updating the pre-learned neural network model to be personalized based on the acquired new data; Including, method. In the first paragraph, The above updating steps are: If it is identified that a preset time has elapsed since the previous update time, a step of updating the pre-learned neural network model to be personalized based on the acquired new data; Including, method. In the first paragraph, The above updating steps are: If the number of times biometric information related to the user's status has been predicted from the previous update point in time is greater than a preset value, a step of updating the pre-learned neural network model to be personalized based on the acquired new data; Including, method. A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs an operation of obtaining a user-customized neural network model for identifying biometric information. The above actions are, An operation of predicting bio-information related to the user's condition by inputting a bio-signal detected from the user into a pre-trained neural network model; and When new data about the user is acquired, an operation of updating the pre-learned neural network model to be personalized based on the acquired new data is included; The above new data is, In relation to the above condition, including biometric information measured from the user, Computer program. A computing device that obtains a user-customized neural network model that identifies biometric information, Memory containing program codes; network unit; A sensing unit that detects the user's biometric information; and A processor that inputs a biometric signal detected from a user into a pre-learned neural network model to predict biometric information related to the user's condition, and when new data about the user is acquired, updates the pre-learned neural network model to be personalized based on the acquired new data; The above new data is, In relation to the above condition, including biometric information measured from the user, Computing device.

Images