WO2025105848A1 - Method, program, and apparatus for providing electrocardiogram reading information - Google Patents

Abstract

Disclosed are a method, a program, and an apparatus for providing electrocardiogram reading information, according to an embodiment of the present disclosure. The method may comprise the steps of: obtaining a first user command via a user interface for an electrocardiogram reading service; and displaying, via the user interface, at least one of reading information about a disease or health condition of a subject whose electrocardiogram is to be measured, and first reference information associated with artificial intelligence used for electrocardiogram reading.

Description

Method, program and device for providing electrocardiogram reading information

The present disclosure relates to data processing technology in the medical field, and more particularly, to a method for providing not only an electrocardiogram interpretation result based on artificial intelligence but also additional information related to artificial intelligence used in electrocardiogram interpretation.

An electrocardiogram is a test that records the electrical activity of the heart. An electrocardiogram is a relatively simple and cost-effective test that can check the health of the heart, and it plays an important role in the early diagnosis and management of heart disease. For example, the electrocardiogram signals measured through an electrocardiogram can be used to check whether each part of the heart is functioning normally, whether the size and location of the heart are normal, and whether there is damage to the heart muscle. In addition, the electrocardiogram signals can be used to diagnose various heart-related problems and predict the health status of a person based on these checks.

Meanwhile, as artificial intelligence technology advances, attempts to analyze electrocardiograms using artificial intelligence technology are increasing. A representative example is a service that predicts various heart diseases by inputting an electrocardiogram into a neural network-based model. These services generally simply provide the possibility of disease onset as an analysis result based on the electrocardiogram. Therefore, it is difficult to externally confirm the reliability of the interpretation results, and there is a possibility that doctors using the electrocardiogram interpretation service may not trust the interpretation results and may not use them. Therefore, in order for the electrocardiogram interpretation results of artificial intelligence to be useful in the actual medical field, it is important to provide additional information that can confirm the reliability of the interpretation results in addition to providing the possibility of disease onset as an interpretation result to doctors checking the electrocardiogram interpretation results (i.e., doctors who requested the electrocardiogram interpretation service).

The present disclosure aims to provide a method for providing information on the specifications and performance of artificial intelligence used in electrocardiogram interpretation as well as the results of interpreting an electrocardiogram using artificial intelligence.

However, the problems to be solved in the present disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood based on the description below.

According to one embodiment of the present disclosure for realizing the task as described above, a method for providing electrocardiogram reading information is disclosed. The method may include a step of obtaining a first user command through a user interface for an electrocardiogram reading service; and a step of displaying at least one of reading information regarding a disease or health condition of an electrocardiogram measurement subject, or first reference information related to artificial intelligence used for electrocardiogram reading, through the user interface.

Alternatively, the first reference information may include at least one of user information requesting the electrocardiogram reading, cutoff information on the risk derived from the reading result using the artificial intelligence, data information used for learning of the artificial intelligence, or performance evaluation information of the artificial intelligence.

Alternatively, the user information may include information about the type of user who measured the electrocardiogram of the electrocardiogram measurement subject, which is recorded in the electrocardiogram data of the electrocardiogram measurement subject.

Alternatively, the data information used for learning of the artificial intelligence may include information about the number of learning data or verification data, information about the number of patients recruited to generate the learning data or verification data, or information about the number of medical institutions that generated the learning data or verification data.

Alternatively, the performance evaluation information of the artificial intelligence may include at least one of information on a verification study by medical institution where the electrocardiogram was measured, information on a performance index according to the verification study by medical institution, information on a verification study by underlying disease of the subject whose electrocardiogram was measured, or information on a performance index according to the verification study by underlying disease.

Alternatively, the first reference information may vary depending on the type of user requesting the electrocardiogram reading.

Alternatively, the first reference information may be generated by processing previously stored source information according to the type of the identified user when the type of user recorded in the electrocardiogram data of the electrocardiogram measurement subject is identified.

Alternatively, the method may further include the steps of obtaining a second user command through the user interface; and displaying second reference information generated by adjusting the first reference information based on the second user command through the user interface.

Alternatively, if the second user command includes a first modification request for the performance evaluation information of the artificial intelligence included in the first reference information, the cutoff information for the risk derived from the reading result using the artificial intelligence included in the first reference information may be dynamically changed based on the performance evaluation information adjusted according to the first modification request.

Alternatively, the first modification request may include a first adjustment value for at least one of the performance indicators, sensitivity or specificity, included in the performance evaluation information.

Alternatively, the cutoff information can be determined by exploring the first adjustment value in a receiver operating characteristic (ROC) curve with the sensitivity and specificity as variables.

Alternatively, if the second user command includes a second modification request for cutoff information on the risk derived from the reading result using the artificial intelligence included in the first reference information, the performance evaluation information of the artificial intelligence included in the first reference information may be dynamically changed based on the electrocardiogram data input as a reading target according to the first user command, the reading information, and the cutoff information adjusted according to the second modification request.

Alternatively, if the second user command includes text data, a request for modification to the first reference information may be generated by inputting the second user command into a pre-trained large-scale language model.

According to one embodiment of the present disclosure for achieving the above-described task, a computer program stored in a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, it performs operations for providing electrocardiogram reading information, and the operations may include an operation of obtaining a first user command through a user interface for an electrocardiogram reading service; and an operation of displaying at least one of reading information regarding a disease or health condition of an electrocardiogram measurement subject, or first reference information related to artificial intelligence used for electrocardiogram reading, through the user interface.

According to one embodiment of the present disclosure for achieving the above-described task, a computing device providing electrocardiogram reading information is disclosed. The device may include a processor including at least one core; a memory including program codes executable by the processor; and an input/output unit providing a user interface for displaying at least one of reading information regarding a disease or health condition of an electrocardiogram measurement subject, or first reference information related to artificial intelligence used for electrocardiogram reading.

The present disclosure not only provides electrocardiogram interpretation results, but also provides additional information that can explain the specifications and performance of the artificial intelligence used for interpretation, thereby increasing the reliability of the interpretation results and helping customers accurately understand the interpretation results.

FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure.

FIG. 2 is a block diagram of a system according to one embodiment of the present disclosure.

FIG. 3 is a sequence diagram showing operations by a system according to one embodiment of the present disclosure.

FIG. 4 is a conceptual diagram showing the relationship between performance evaluation information and cutoff information according to one embodiment of the present disclosure.

FIG. 5 is a flowchart illustrating a method for providing electrocardiogram reading information according to one embodiment of the present disclosure.

FIG. 6 is a flowchart illustrating a method for providing electrocardiogram reading information according to an alternative embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. The embodiments presented in the present disclosure are provided so that those skilled in the art can utilize or implement the contents of the present disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be implemented in various different forms and is not limited to the embodiments below.

Throughout the specification of the present disclosure, the same or similar drawing reference numerals refer to the same or similar components. In addition, in order to clearly describe the present disclosure, drawing reference numerals of parts that are not related to the description of the present disclosure may be omitted in the drawings.

The term "or" as used herein is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified herein or the context makes clear, "X employs either A or B" should be understood to mean either one of the natural inclusive permutations. For example, unless otherwise specified herein or the context makes clear, "X employs A or B" can be interpreted to mean either X employs A, X employs B, or X employs both A and B.

The term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated concepts listed.

The terms "comprises" and/or "comprising" as used herein should be understood to mean the presence of particular features and/or components. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, other components, and/or combinations thereof.

Unless otherwise specified in this disclosure or unless the context makes it clear that the singular form is intended to be referred to, the singular should generally be construed to include “one or more.”

The term "Nth (N is a natural number)" used in the present disclosure can be understood as an expression used to mutually distinguish components of the present disclosure according to a predetermined standard such as a functional viewpoint, a structural viewpoint, or convenience of explanation. For example, components performing different functional roles in the present disclosure can be distinguished as a first component or a second component. However, components that are substantially the same within the technical spirit of the present disclosure but should be distinguished for convenience of explanation may also be distinguished as a first component or a second component.

The term "acquisition" as used in this disclosure may be understood to mean not only receiving data via a wired or wireless communication network with an external device or system, but also generating data in an on-device form.

Meanwhile, the term "module" or "unit" used in the present disclosure may be understood as a term referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, a combination of software and hardware, etc. At this time, the "module" or "unit" may be a unit composed of a single element, or may be a unit expressed as a combination or set of multiple elements. For example, as a narrow concept, a "module" or "unit" may refer to a hardware element of a computing device or a set thereof, an application program that performs a specific function of software, a processing process implemented through software execution, or a set of instructions for program execution, etc. In addition, as a broad concept, a "module" or "unit" may refer to a computing device itself that constitutes a system, or an application that is executed on a computing device, etc. However, the above-described concept is only an example, and the concept of “module” or “part” may be variously defined within a category understandable to those skilled in the art based on the contents of the present disclosure.

The term "model" used in the present disclosure may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or an abstract model regarding a processing process to solve a specific problem. For example, a neural network "model" may refer to the entire system implemented as a neural network that has a problem-solving ability through learning. In this case, the neural network may have a problem-solving ability by optimizing parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network, or may include a neural network set in which multiple neural networks are combined.

The term "control graphic" used in the present disclosure may be understood as a graphic object that executes a command and produces a visible result when manipulating a user interface. In other words, a "control graphic" may be understood as a basic unit that constitutes a user interface, and as a user interface element that displays content to be provided to a user or enables manipulation by a user.

The explanation of the terms set forth above is intended to aid in understanding the present disclosure. Therefore, if the terms set forth above are not explicitly stated as matters limiting the contents of the present disclosure, it should be noted that they are not used to limit the technical ideas of the contents of the present disclosure.

FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure.

The computing device (100) according to one embodiment of the present disclosure may be a hardware device or a part of a hardware device that performs comprehensive processing and calculation of data, or may be a software-based computing environment connected to a communication network. For example, the computing device (100) may be a server that performs an intensive data processing function and shares resources, or may be a client that shares resources through interaction with a server. In addition, the computing device (100) may be a cloud system in which a plurality of servers and clients interact to comprehensively process data. Since the above description is only one example related to the type of the computing device (100), the type of the computing device (100) may be configured in various ways within a category that can be understood by those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 1, a computing device (100) according to one embodiment of the present disclosure may include a processor (110), a memory (120), a network unit (130), and an input/output unit (140). However, FIG. 1 is only an example, and the computing device (100) may include other configurations for implementing a computing environment. In addition, only some of the disclosed configurations may be included in the computing device (100).

The processor (110) according to one embodiment of the present disclosure may be understood as a configuration unit including hardware and/or software for performing computing operations. For example, the processor (110) may process commands generated as a result of user interaction through a user interface. In addition, the processor (110) may read a computer program to perform data processing for machine learning. The processor (110) may process computational processes such as processing of input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. The processor (110) for performing such data processing and operations may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). The type of processor (110) described above is only an example, and thus the type of processor (110) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The processor (110) can generate a user interface for an electrocardiogram reading service. The electrocardiogram reading service can be understood as a service that performs electrocardiogram reading and provides electrocardiogram reading results at the request of an electrocardiogram reading requester. The user interface for the electrocardiogram reading service can include control graphics for obtaining user commands, such as control graphics for receiving an electrocardiogram reading request, control graphics for receiving electrocardiogram data, etc. In addition, the user interface for the electrocardiogram reading service can include control graphics for providing information related to electrocardiogram reading, such as reading information about a disease or health condition of an electrocardiogram measurement subject, reference information about an artificial intelligence that performed the electrocardiogram reading, etc. When the user interface is implemented through the input/output unit (140) described below, the control graphics described above can be arranged and visually expressed by each area of the user interface.

Meanwhile, the processor (110) may perform electrocardiogram reading using artificial intelligence. The processor (110) may input electrocardiogram data into the artificial intelligence to generate electrocardiogram reading data. At this time, the electrocardiogram reading data may include reading information on a disease or health condition of an electrocardiogram measurement subject. In addition, the reading information may include information that may be used to diagnose a disease or health condition of a user, such as a type of disease, a possibility of developing a disease, a possibility of worsening a disease or health condition, etc. For example, when a user command requesting an electrocardiogram reading is obtained through a user interface, the processor (110) may input the electrocardiogram data input through the user command into a neural network model. At this time, the neural network model may be a pre-learned model that extracts features from the electrocardiogram data and outputs a probability of developing a disease or worsening a health condition based on the extracted features. At this time, the learning may be supervised learning performed based on labeled data, or may be unsupervised learning or self-supervised learning depending on the structure of the neural network model. The processor (110) can perform analysis on electrocardiogram data through a neural network model to generate reading data. The processor (110) can express the reading data generated through the neural network model as a control graphic of a user interface.

The memory (120) according to one embodiment of the present disclosure may be understood as a configuration unit including hardware and/or software for storing and managing data processed in the computing device (100). That is, the memory (120) may store any type of data generated or determined by the processor (110) and any type of data received by the network unit (130). For example, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, a RAM (random access memory), a SRAM (static random access memory), a ROM (read-only memory), an EEPROM (electrically erasable programmable read-only memory), a PROM (programmable read-only memory), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory (120) may also include a database system that controls and manages data in a predetermined system. The type of memory (120) described above is only an example, and thus the type of memory (120) can be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The memory (120) can manage, by structuring and organizing, data required for the processor (110) to perform operations, combinations of data, and program codes executable by the processor (110). For example, the memory (120) can store medical data received through the network unit (130) described below. The memory (120) can store program codes for storing rules for processing medical data, program codes for causing a neural network model to perform learning by receiving medical data, program codes for causing a neural network model to perform inference by receiving medical data in accordance with the intended use of the computing device (100), and processed data generated as the program codes are executed.

The network unit (130) according to one embodiment of the present disclosure may be understood as a configuration unit that transmits and receives data via any type of known wired and wireless communication system. For example, the network unit (130) may perform data transmission and reception using a wired and wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), fifth generation mobile communication (5G), ultra wide-band, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity, near field communication (NFC), or Bluetooth. Since the above-described communication systems are only examples, the wired and wireless communication system for data transmission and reception of the network unit (130) may be applied in various ways other than the above-described examples.

The network unit (130) can receive data required for the processor (110) to perform calculations through wired or wireless communication with any system or any client, etc. In addition, the network unit (130) can transmit data generated through the calculation of the processor (110) through wired or wireless communication with any system or any client, etc. For example, the network unit (130) can receive biometric data through communication with a cloud server that performs tasks such as standardization of databases and medical data in a hospital environment, a client such as a smart watch, or a medical computing device, etc. The network unit (130) can transmit output data of a neural network model, intermediate data, processed data, etc. derived from the calculation process of the processor (110), etc. through communication with the aforementioned database, server, client, or computing device, etc.

The input/output unit (140) according to one embodiment of the present disclosure may be understood as a configuration unit including hardware and/or software for implementing a user interface. That is, the input/output unit (140) may visualize and output any type of data generated or determined by the processor (110) and any type of data received by the network unit (130). In addition, the input/output unit (140) may receive a user input that generates a command to be transmitted to any system or any client connected to the processor (110), the computing device (100), or the like via wired or wireless communication. For example, the input/output unit (140) may include a display module that may output visualized information or implement a touch screen, such as a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, etc. In addition, the input/output unit (140) may include an input module capable of recognizing actions such as a user's motion, voice, etc., such as a camera, a microphone, a keyboard, a mouse, etc. The modules described above are only examples, and thus, the modules included in the input/output unit (140) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure, in addition to the examples described above.

The input/output unit (140) may implement a user interface to output graphics generated by the processor (110) or receive user commands generated by user manipulation and transmit them to the processor (110). For example, the input/output unit (140) may output control graphics for receiving user commands regarding electrocardiogram reading, control graphics for providing information, etc., in a specific area of the user interface. In addition, the input/output unit (140) may receive a user command for a control graphic output in a specific area of the user interface. At this time, the user command for a specific graphic may be understood as an input signal generated by a user manipulation for a specific graphic. In addition, the user manipulation for a specific graphic may mean an action that a user can perform through the input/output unit (140), such as a click, a double-click, a hover, a flick, a pinch, a spread, etc. for a specific graphic. The input/output unit (140) can receive user commands and transmit them to the processor (110), thereby enabling operations to be performed to provide a user interface based on user control.

FIG. 2 is a block diagram of a system according to one embodiment of the present disclosure. FIG. 3 is a sequence diagram showing operations by a system according to one embodiment of the present disclosure.

Referring to FIG. 2, a system according to one embodiment of the present disclosure may include a client (200) that provides a user interface to a user, and a server (300) that processes information obtained through the user interface or information provided as a user interface. For example, the client (200) may correspond to the computing device (100) of FIG. 1. The client (200) may control graphics implemented in the user interface using at least one of a command or data obtained through interaction with a user. In addition, the client (200) may transmit at least one of a user command or data obtained through the user interface to the server (300). The server (300) may correspond to a form of the computing device (100) of FIG. 1 excluding the input/output unit (140). The server (300) may process information to be provided through the user interface using at least one of a command or data obtained through the user interface. Specifically, the server (300) may generate visualized information to be provided to the user through the user interface using an artificial intelligence model. The server (300) transmits processed information to the client (200), and the client (200) can provide the processed information to the user through a user interface.

Referring to FIG. 3, the client (200) can obtain the 1-1 user command through the user interface for the electrocardiogram reading service (S110). At this time, the 1-1 user command can include the user's request for electrocardiogram reading and electrocardiogram data to be read. The client (200) can transmit the 1-1 user command to the server (300). The server (300) can generate electrocardiogram reading information based on the electrocardiogram data included in the 1-1 user command (S115). Specifically, the server (300) can input the electrocardiogram data included in the 1-1 user command into a neural network model. The server (300) can extract electrocardiogram features from the electrocardiogram data through the neural network model. At this time, the electrocardiogram features are related to the electrocardiogram waveform and can include a P wave, a QRS complex, an R peak, etc. The server (300) can derive a numerical value indicating the possibility of disease onset or deterioration or the possibility of health deterioration based on electrocardiogram features through a neural network model. The server (300) can compare the numerical value derived as an output of the neural network model with a cutoff to generate reading information on the disease or health condition of the subject of electrocardiogram measurement. For example, if the numerical value derived as an output of the neural network model is less than the cutoff, the server (300) can determine that the risk of a specific disease onset or the risk of health deterioration is low, and classify the subject of electrocardiogram measurement into a low-risk group. If the numerical value derived as an output of the neural network model is greater than or equal to the cutoff, the server (300) can determine that the risk of a specific disease onset or the risk of health deterioration is high, and classify the subject of electrocardiogram measurement into a high-risk group. When the electrocardiogram reading information is generated, the server (300) can transmit the electrocardiogram reading information to the client (200). And, the client (200) can display electrocardiogram reading information through the user interface (S119).

The server (300) can identify the type of user who requested the electrocardiogram reading based on the electrocardiogram data included in the 1-1 user command (S120). For example, the server (300) can check the information about the type of user recorded in the electrocardiogram data included in the 1-1 user command. Since the electrocardiogram data included in the 1-1 user command is in XML format, the server (300) can identify the text about the type of user recorded in the electrocardiogram data in XML format to determine the type of user who requested the electrocardiogram reading. At this time, the type of user can be classified into a general hospital, a primary hospital, or a health examination center depending on the type of medical institution. In addition, the general hospital can be further classified into an outpatient clinic, an emergency room, or an intensive care unit. The server (300) can identify the type of user who requested the electrocardiogram reading in this way and provide the user with additional information necessary in addition to the electrocardiogram reading result. Meanwhile, although step S120 in FIG. 3 is expressed as being performed separately from step S115, step S120 may be performed simultaneously with step S115.

The client (200) can obtain the first-second user command through the user interface (S130). At this time, the first-second user command may include a request for reference information regarding the artificial intelligence used for electrocardiogram reading. Although step S130 in FIG. 3 is expressed as being performed separately from step S110, step S130 may be performed simultaneously with step S110. The reference information may include information regarding the specification or performance of the artificial intelligence used for electrocardiogram reading. The client (200) can transmit the first-second user command to the server (300). The server (300) can generate first reference information related to the artificial intelligence used for electrocardiogram reading based on the first-second user command (S135). If the type of user recorded in the electrocardiogram data of the electrocardiogram measurement target is identified through step S120, the server (300) can process the previously stored source information according to the type of user. The source information can be understood as a set of all information used for learning and inference of artificial intelligence. Specifically, the source information can include electrocardiogram information, information on the subject of electrocardiogram measurement, information on the medical institution that measured the electrocardiogram, information on the underlying disease of the subject of electrocardiogram measurement, clinical research information related to electrocardiogram interpretation, or learning information of artificial intelligence used for interpretation service, evaluation information of artificial intelligence, etc. The server (300) can collect and process information related to the type of user from such source information to determine the first reference information. For example, a user working in an emergency room may have a need to "see only very serious and clear myocardial infarctions, even if there are cases where myocardial infarctions are missed." On the other hand, a health screening center may have a need to "lower the cutoff criteria so that no patients are missed, even if it is possible to judge a patient who does not have myocardial infarction as myocardial infarction." Accordingly, if the type of user is an emergency room, it may be necessary to raise the cutoff for interpretation, whereas if the type of user is a health screening center, it may be necessary to lower the cutoff for interpretation. Since the content of the required information may vary depending on the type of user, the server (300) may extract and process information from the source information based on the type of user, and generate first reference information tailored to the type of user. Specifically, the server (300) may extract electrocardiogram data and reading information from the source information based on the type of user. Then, the server (300) may apply a cutoff determined depending on the type of user based on the electrocardiogram data and reading information extracted from the source information, and may calculate an index for performance evaluation. The server (300) may collate the information extracted and calculated in this way to generate first reference information.

Meanwhile, the first reference information may include at least one of user information requesting electrocardiogram interpretation, cutoff information on the level of risk derived from the interpretation results using artificial intelligence, data information used for learning of the artificial intelligence, or performance evaluation information of the artificial intelligence. The cutoff information on the level of risk derived from the interpretation results using artificial intelligence may be understood as information corresponding to the criteria for judgment of a disease or health condition. The cutoff value included in the cutoff information may be a preset value based on a clinically acceptable performance evaluation index of artificial intelligence, or may be a value that dynamically changes according to the type of user or the user's requirements. The data information used for learning of the artificial intelligence may include at least one of information on the number of learning data or verification data, information on the number of patients recruited to generate learning data or verification data, or information on the number of medical institutions that generated learning data or verification data. The performance evaluation information of the artificial intelligence may include at least one of information on a validation study by medical institution where the electrocardiogram was measured, information on a performance index according to the validation study by medical institution, information on a validation study by underlying disease of the subject whose electrocardiogram was measured, or information on a performance index according to the validation study by underlying disease. Here, the performance index may include at least one of accuracy, sensitivity, or specificity for the artificial intelligence output. In addition, the subject whose underlying disease has been verified may include at least one of a pregnant mother, a patient with LBBB (left bundle branch block) disease, a patient with atrial fibrillation RVR (rapid ventricular response) disease, or a patient complaining of chest pain.

When the first reference information is generated as described above, the server (300) can transmit the first reference information to the client (200). Then, the client (200) can display the first reference information through a user interface (S139). The client (200) and the server (300) can increase the reliability of the artificial intelligence reading result through processing and providing the first reference information.

The client (200) can obtain a second user command through the user interface (S140). At this time, the second user command may include a modification request to adjust some or all of the details included in the first reference information. The client (200) can transmit the second user command to the server (300). The server (300) can adjust the first reference information based on the second user command to generate the second reference information (S145). For example, even if the server (300) identifies the type of user and provides reference information according to the type of user, the user may need to adjust the reference information in detail or need new reference information that meets his or her needs. Accordingly, the server (300) can modify the first reference information based on the second user command indicating the needs of the user so as to generate reference information that reflects the needs of the user. When the user inputs a second user command requesting adjustment of the sensitivity and specificity among the performance indicators of the artificial intelligence to a desired value, the server (300) can search for the adjustment value included in the second user command in a receiver operating characteristic (ROC) curve using the sensitivity and specificity as variables and re-determine the cutoff value. Then, the server (300) can modify the performance evaluation information included in the first reference information based on the adjustment value included in the second user command and the re-determined cutoff value to generate the second reference information. When the user inputs a second user command requesting adjustment of the cutoff, which is a reading criterion, to a desired value, the server (300) can re-analyze the electrocardiogram data and the reading information therefor based on the cutoff to re-calculate the performance indicator. At this time, the re-analysis may be a task of comparing the output value of the artificial intelligence included in the reading information with the cutoff input through the second user command. The re-analysis may be a task of filtering the performance indicator based on the cutoff input through the second user command from the source information in which the performance indicators by cutoff are organized. Additionally, the server (300) can generate second reference information by modifying cutoff information and performance evaluation information included in the first reference information.

Meanwhile, the second user command may be numerical data for information included in the first reference, or may be text data expressing a user request. For example, the user may input the second user command in the form of text, “I want to see the AI reading performance index only for patients with LBBB as an underlying disease,” through the user interface. If the second user command is such text data, the server (300) may input the second user command into a pre-learned large language model in order to adjust the reference information by understanding the meaning inherent in the text. Then, the server (300) may generate a modification request for the first reference information through the large language model into which the second user command is input. That is, if the second user command is in the form of text, the server (300) may utilize the large language model in order to clearly recognize the user command.

When such second reference information is generated, the server (300) can transmit the second reference information to the client (200). Then, the client (200) can display the second reference information through a user interface (S149). That is, the client (200) and the server (300) can process and provide reference information that reflects the needs of the user as well as reference information suitable for the type of user through the generation of the second reference information.

FIG. 4 is a conceptual diagram showing the relationship between performance evaluation information and cutoff information according to one embodiment of the present disclosure.

The performance evaluation information included in the reference information according to one embodiment of the present disclosure may include performance indicators such as sensitivity and specificity. Sensitivity and specificity are indicators representing the performance of the test itself and are unique values for a specific test. For example, if a test for a specific disease has a sensitivity of 90% and a specificity of 95%, this will never change. Sensitivity and specificity will never change regardless of the test results. Sensitivity is the ratio of people who actually have the disease who are determined to be positive in the test. In other words, it indicates how well people with the disease are found. Specificity is the ratio of people who actually do not have the disease who are determined to be negative in the test. In other words, it indicates how well healthy people are distinguished. Therefore, sensitivity and specificity are values determined according to the test design and performance.

In the emergency room, patients with chest pain are classified into three risk levels for acute myocardial infarction. The highest risk group is defined as the "rule-in" group, the lowest risk group is defined as the "rule-out" group, and the medium risk group is defined as the "observation" group and treated accordingly. At this time, the user determines that the performance when deciding on "rule-out" for the low-risk group is a sensitivity of 99% or higher and a negative predictive value (NPV) of 99.5%, and the performance when deciding on "rule-in" for the high-risk group is a specificity of 90% and a positive predictive value (PPV) of 60-80%, which are clinically acceptable levels.

The user can set the desired sensitivity and specificity through the user interface for the electrocardiogram interpretation service. When the user-defined sensitivity and specificity are input through the user interface, the cutoff can be determined through the ROC curve. The ROC curve expresses the sensitivity and 1-specificity (false positive rate) at various cutoffs. Each point in the ROC curve corresponds to a specific cutoff. Therefore, the cutoff can be searched in the ROC curve based on the sensitivity and specificity set by the user. The cutoff determined in this way can be reflected in the reference information provided to the user.

FIG. 5 is a flowchart illustrating a method for providing electrocardiogram reading information according to one embodiment of the present disclosure.

Referring to FIG. 5, a client (200) according to an embodiment of the present disclosure may obtain a first user command through a user interface for an electrocardiogram reading service (S210). The first user command may include a 1-1 user command including a user's request for electrocardiogram reading and electrocardiogram data to be read, and a 1-2 user command including a request for reference information regarding artificial intelligence used for electrocardiogram reading. The client (200) may implement a user interface through an input/output unit to provide the user with a control graphic for generating the first user command. The client (200) may generate the first user command through a user input action for the control graphic of the user interface.

The client (200) may display at least one of reading information on a disease or health condition of an electrocardiogram measurement subject, or first reference information related to artificial intelligence used for electrocardiogram reading, through the user interface (S220). The first reference information may include at least one of user information requesting electrocardiogram reading, cutoff information on risk derived from a reading result using artificial intelligence, data information used for learning of artificial intelligence, or performance evaluation information of artificial intelligence. In addition, the first reference information may be information that varies depending on the type of user requesting electrocardiogram reading. When the first user command is obtained (S120), the client (200) may recognize user information recorded in electrocardiogram data included in the first user command, and generate first reference information according to the recognized information. Specifically, when the type of user recorded in electrocardiogram data of an electrocardiogram measurement subject is identified, the first reference information may be generated by processing previously stored source information according to the type of the identified user. At this time, processing may be a task of extracting necessary information from source information, or a task of recombining or generating information from source information using a pre-learned neural network model.

The client (200) can obtain a second user command through a user interface for an electrocardiogram reading service (S230). The second user command may include a modification request to adjust some or all of the details included in the first reference information. The client (200) may implement a user interface through an input/output unit to provide a user with a control graphic for generating the second user command. At this time, the control graphic for generating the second user command may be the same as the control graphic for generating the first user command. The client (200) may generate the second user command through a user input operation for the control graphic of the user interface. Meanwhile, the second user command may be included in the first user command. Accordingly, even if step S230 is not performed, step S240, which will be described later, may be sequentially performed after step S210 is performed. That is, step S230 may be omitted.

The client (200) can display second reference information generated by adjusting first reference information based on a second user command through a user interface (S240). If the second user command includes a first modification request for performance evaluation information of artificial intelligence included in the first reference information, cutoff information for risk derived from a reading result using artificial intelligence included in the first reference information can be dynamically changed based on the performance evaluation information adjusted according to the first modification request. The first modification request can include a first adjustment value for at least one of sensitivity and specificity, which are performance indicators included in the performance evaluation information. The cutoff information can be determined by searching the first adjustment value in an ROC curve using sensitivity and specificity as variables. If the second user command includes a second modification request for cutoff information on risk derived from the interpretation results using artificial intelligence, which is included in the first reference information, the performance evaluation information of the artificial intelligence, which is included in the first reference information, may be dynamically changed based on the electrocardiogram data input as the interpretation target according to the first user command, the interpretation information, and the cutoff information adjusted according to the second modification request.

FIG. 6 is a flowchart illustrating a method for providing electrocardiogram reading information according to an alternative embodiment of the present disclosure.

Referring to FIG. 6, the server (300) according to one embodiment of the present disclosure can identify the type of user who measured the electrocardiogram of the electrocardiogram measurement target based on the electrocardiogram data (S310). When a user command requesting electrocardiogram reading is obtained, the server (300) can confirm the type of user recorded in the electrocardiogram data. At this time, the type of user can be classified according to the type of medical institution. For example, the type of user can be classified into general hospitals, primary hospitals, health checkup centers, etc. In addition, the general hospital can be classified in detail into outpatient treatment, emergency room, intensive care unit, etc.

The server (300) can generate first reference information according to the type of user identified through step S310 (S320). The required reference information, such as the cutoff, which is an electrocardiogram reading criterion, and the performance indicators that can be derived accordingly, may differ depending on the type of user. Accordingly, the server (300) can generate first reference information by processing the source information stored in advance according to the type of user. The source information may include learning data information, verification data information, clinical research information, etc. used in building artificial intelligence. Such source information may be updated periodically.

The server (300) can obtain a second user command through a user interface for an electrocardiogram reading service (S330). When the second user command is received through a user interface implemented through the client (200), the server (300) can obtain the second user command through communication with the client (200). Meanwhile, as in step S230, the second user command may be included in the first user command requesting electrocardiogram reading. Accordingly, even if step S320 is not performed, step S340, which will be described later, may be performed sequentially. That is, step S330 may be omitted.

The server (300) can adjust the first reference information to generate the second reference information (S340). Since the detailed information included in the first reference information mutually influences the determination of each piece of information, the server (300) can adjust the detailed information included in the first reference information by considering this influence. For example, if the user wants to change the cutoff, which is a reading criterion, the server (300) can not only change the cutoff information included in the first reference information, but also modify the reading information and performance evaluation information affected by the change in the cutoff information together. Through the operation of the server (300), the user can easily check the remaining reference information that changes accordingly while changing the reference information that he or she wants.

The various embodiments of the present disclosure described above can be combined with additional embodiments and can be modified within a range that can be understood by those skilled in the art in light of the detailed description set forth above. It should be understood that the embodiments of the present disclosure are illustrative in all respects and not restrictive. For example, each component described as a single component may be implemented in a distributed manner, and likewise, components described as distributed may be implemented in a combined manner. Accordingly, all changes or modifications derived from the meaning, scope, and equivalent concept of the claims of the present disclosure should be interpreted as being included in the scope of the present disclosure.

Claims (15)

A method for providing electrocardiogram reading information, A step of obtaining a first user command through a user interface for an electrocardiogram reading service; and A step of displaying at least one of reading information regarding a disease or health condition of a subject of electrocardiogram measurement, or first reference information related to artificial intelligence used for electrocardiogram reading, through the user interface; Including, method. In paragraph 1, The above first reference information is, Including at least one of the user information requesting the electrocardiogram reading, cutoff information on the risk derived from the reading result using the artificial intelligence, data information used for learning of the artificial intelligence, or performance evaluation information of the artificial intelligence. method. In the second paragraph, The above user information is: Information about the type of user who measured the electrocardiogram of the electrocardiogram measurement subject, recorded in the electrocardiogram data of the electrocardiogram measurement subject, method. In the second paragraph, The data information used for learning the above artificial intelligence is: Including at least one of information about the number of learning data or verification data, information about the number of patients recruited to generate the learning data or verification data, or information about the number of medical institutions that generated the learning data or verification data. method. In the second paragraph, The performance evaluation information of the above artificial intelligence is: Contains at least one of information on validation studies by medical institution where electrocardiograms were measured, information on performance indicators according to the validation studies by said medical institution, information on validation studies by underlying disease of the subject whose electrocardiogram was measured, or information on performance indicators according to the validation studies by underlying disease. method. In paragraph 1, The above first reference information is, It depends on the type of user who requested the above ECG reading. method. In paragraph 6, The above first reference information is, When the type of user recorded in the electrocardiogram data of the electrocardiogram measurement target is identified, the previously stored source information is processed according to the type of the identified user, thereby generating the information. method. In paragraph 1, A step of obtaining a second user command through the user interface; and A step of displaying second reference information generated by adjusting the first reference information based on the second user command through the user interface; Including more, method. In Article 8, If the second user command includes a first request for modification of the performance evaluation information of the artificial intelligence included in the first reference information, The cutoff information for the risk derived from the interpretation results using artificial intelligence, included in the above first reference information, is Dynamically changed based on the performance evaluation information adjusted in accordance with the above first modification request, method. In Article 9, The above first amendment request is, Including a first adjustment value for at least one of the performance indicators, sensitivity or specificity, included in the above performance evaluation information. method. In Article 10, The above cutoff information is, The first adjustment value is determined by searching for it in the ROC (receiver operating characteristic) curve with the above sensitivity and specificity as variables. method. In Article 8, If the second user command includes a second request for modification of the cutoff information on the risk derived from the reading result using the artificial intelligence contained in the first reference information, The performance evaluation information of the artificial intelligence included in the above first reference information is, It is dynamically changed based on the electrocardiogram data input as a reading target according to the first user command, the reading information, and the cutoff information adjusted according to the second modification request. method. In Article 8, If the above second user command contains text data, The modification request for the above first reference information is generated by inputting the second user command into a pre-trained large language model. method. A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations that provide electrocardiogram reading information. The above actions are, An operation for obtaining a first user command through a user interface for an electrocardiogram reading service; and An action of displaying, through the user interface, at least one of reading information regarding a disease or health condition of a subject of electrocardiogram measurement, or first reference information related to artificial intelligence used for electrocardiogram reading; Including, Computer program. A computing device that provides electrocardiogram reading information, A processor comprising at least one core; a memory containing program codes executable by the processor; and An input/output unit providing a user interface that displays at least one of reading information on a disease or health condition of a subject of electrocardiogram measurement, or first reference information related to artificial intelligence used for reading the electrocardiogram; Including, device.

Images