RU2806982C1 - Device and method for analysis of medical images - Google Patents

Abstract

FIELD: image processing.

SUBSTANCE: device and method for determining pathology based on the analysis of medical images. A device is proposed that contains a unit for receiving data to be processed containing a medical image; a unit for preparing an image contained in the data to be processed for use by the neural network through performing one or more preliminary transformations on it; a block for predicting the presence or absence of pathology in an image using this neural network; and a block for transmitting the processing result to the device that requested data processing, wherein this neural network is pre-trained to predict the presence or absence of pathology, as well as the gender and age of the patient, and is configured to analyze only the prepared image during operation without using data on the patient’s gender and age shown in this image.

EFFECT: increased accuracy of automatic determination of the probability of pathology, the ability to use incompletely labeled data sets or incorrectly labeled data sets.

13 cl, 13 dwg

Description

Field of technology to which the invention relates

The present invention relates to the field of image processing, and more particularly to an apparatus and method for determining pathology based on the analysis of medical images.

State of the art

In traditional methods of diagnosing pathologies of the chest organs, which have been used in medicine for decades, the operator of an X-ray machine or doctor visually examined the X-ray image obtained by the machine and made a conclusion about the presence or absence of a particular pathology in the patient being examined.

However, the high prevalence of cancer, heart and lung diseases, as well as the covid-19 outbreak, which has led to a significant increase in mortality from these diseases, entail enormous social costs. In medical institutions carrying out diagnostic and therapeutic activities, there is overload and a shortage of qualified medical specialists in the field of radiological studies; the assessment and interpretation of the results of radiological studies is influenced by the human factor; there are no necessary resources for conducting repeated readings of studies. The practice of carrying out diagnostic measures is associated with the risk of missing pathologies during the initial analysis of radiological studies.

These factors require fundamental research in this area in order to develop new and more effective therapeutic and diagnostic tools, including in order to improve the quality of medical decisions and, if possible, speed up the medical decision-making process.

All this contributes to the introduction of modern digital data processing techniques into medicine. The digitalization of healthcare is accompanied by rapidly growing volumes of digitized patient health information, clinical databases, and medical data sets that could be used to support clinical decision making. In recent years, artificial intelligence (AI) technologies have been increasingly used to work on such problems.

For example, in known methods of modern AI diagnostics, such as WO 2021/031279 A1 or WO 2021/091661 A1, a neural network, which was previously trained on a labeled set of images, receives a digitized X-ray image of the patient’s chest, analyzes it and makes a prediction whether a given patient has a pathology, thereby helping the doctor make the best informed decision.

Attempts are being made to use a variety of neural network architectures, different training methods and different methods of image pre-processing, but the accuracy of the known methods is such that a significant part of the medical community expresses the opinion that at the current level of development such systems do not help the doctor, but, on the contrary, interfere, since they may reduce the concentration of a specialist relying partly on AI, and errors in decision-making are possible, which can affect the quality of medical services. And due to the fact that the responsibility for the decision lies with the person, the human factor is still large and labor costs and qualification requirements are high.

In this regard, scientific and technical publications have recently begun to form the opinion that it is necessary to use some additional data about the patient in addition to the images. However, at the moment there were no specific methods and devices that would allow such processing with high accuracy with minimal requirements both for the additional data itself and for the devices processing such data.

The essence of the invention

In order to overcome at least some of the above-mentioned disadvantages of the prior art, the present invention is directed to improving the accuracy of devices and methods for determining pathology based on the analysis of medical images.

According to a first aspect of the present invention, there is provided a device for determining pathology based on medical image analysis, comprising:

a data receiving unit configured to receive data containing a medical image to be processed from a device requesting data processing by communicating with it;

an image preparation unit configured to prepare, using at least one processor, a medical image contained in the data to be processed by performing one or more preliminary transformations on it for use by a neural network operating on the at least one processor;

a pathology prediction block configured to analyze the prepared image using said neural network and determine the presence or absence of pathology in the image;

a report generation unit configured to generate, using at least one processor, a report on a study conducted in this device, wherein the study report contains at least one file containing an indication of the result of the processing performed in the pathology prediction block; And

a data transmission unit configured to transmit a research report to a device that requested data processing by communicating with it,

wherein said neural network is pre-trained using at least one data set containing labeled medical images to predict the presence or absence of pathology, as well as the gender and age of the patient,

wherein the labeled medical images contained in the at least one dataset used to train the neural network contain:

images having a first label associated with an explicit indication of the male gender of the patient,

images having a second label associated with an explicit indication of the patient's female gender, and

images having a third label associated with an implicit indication of the patient's gender, wherein the third label is assigned to images that initially did not have a label indicating the patient's gender, and images that initially had a label in which the patient's gender was specified as data other than the explicit one indications of male or female,

wherein said neural network is pre-trained to predict the expected gender of the patient in the form of an indication of the first mark or the second mark, if the neural network was able to explicitly determine, respectively, the male or female gender of the patient, or in the form of an indication of the third mark, if the neural network is not it was possible to clearly determine the gender of the patient,

wherein said neural network is configured to analyze during operation only the prepared image without additional input data in the form of data on the gender and age of the patient depicted in this image.

In one embodiment, said processing result is intended to support medical decision-making about the state of at least one organ and/or cavity of the examined patient recorded in said image and/or about the treatment of the examined patient.

In one embodiment, said neural network comprises:

- a preliminary section containing a first neural network, in which each layer is sequentially connected to each subsequent layer, and the preliminary section is configured to receive said prepared image and output intermediate data;

- a first subnetwork containing a second neural network and configured to receive said intermediate data from the preliminary section and predict the probability of the presence of pathology in the analyzed image;

- a second subnetwork containing a third neural network and configured to receive said intermediate data from the preliminary site and predict the expected gender of the patient;

- a third subnetwork containing a fourth neural network and configured to receive said intermediate data from the preliminary section and predict the expected age of the patient.

In one embodiment, the preliminary section is based on the DenseNet architecture.

In one embodiment, said neural network is pre-trained to predict the expected age of the patient as an indication of one of the predefined age ranges.

In one embodiment, the size of each age range is from 7 to 15 years.

In one embodiment, the number of predefined age ranges is 10, the size of each range being 10 years.

In one embodiment, said neural network is pre-trained to predict the presence or absence of at least one sign from a pre-defined set of possible pathologies characteristic of at least one organ and/or cavity recorded in said image or for a given type of diagnostic study.

In one embodiment, the predetermined set of possible pathologies for a chest x-ray comprises at least one of the following pathologies: pleural effusion, pneumothorax, atelectasis, focal opacities, infiltration/consolidation, dissemination, cavity with decay, cavity with fluid level, calcification / calcified shadow in the lungs, violation of the integrity of the cortical layer.

In one embodiment, if it is determined that there is a pathology in the image, the report generating unit is further configured to:

identifying possible areas in which signs of pathologies were found, using computer vision methods that analyze the original image and activation within the neural network of the pathology prediction block, which were obtained by running this image through it;

generating an image with visualization of pathology in the form of a copy of the original image, on which an indication of the found areas with signs of pathologies is superimposed; And

generating a research report containing an image with pathology visualization.

In one embodiment, a combination of Grad-CAM (weighted combination of gradient-based class activation maps) and Saliency maps (saliency maps) methods is used to identify possible areas with signs of pathology, and

a heat map or outline of the boundaries of the areas is used to indicate the found areas with signs of pathologies.

In one embodiment, the study report contains a text study protocol containing in text form at least one of the following:

- source image identifier;

- indication of the probability of the presence of pathology;

- image acquisition time;

- time of receiving the processing result from the pathology prediction block;

- time of report generation;

- information about device operation errors that occurred during the analysis.

According to a second aspect of the present invention, there is provided a method for determining pathology based on the analysis of medical images, comprising the steps of:

using a communication device, receiving data to be processed containing a medical image of the patient being examined;

preparing, using at least one processor, a medical image contained in the data to be processed by performing one or more preliminary transformations on it for use by a neural network operating on the at least one processor;

using said neural network, the prepared image is analyzed and the presence or absence of pathology in the image is determined;

wherein said neural network is pre-trained using at least one data set containing labeled medical images to predict the presence or absence of pathology, as well as the gender and age of the patient,

wherein the labeled medical images contained in the at least one dataset used to train the neural network contain:

images having a first label associated with an explicit indication of the male gender of the patient,

images having a second label associated with an explicit indication of the patient's female gender, and

images having a third label associated with an implicit indication of the patient's gender, wherein the third label is assigned to images that initially did not have a label indicating the patient's gender, and images that initially had a label in which the patient's gender was specified as data other than the explicit one indications of male or female,

wherein said neural network is pre-trained to predict the expected gender of the patient in the form of an indication of the first mark or the second mark, if the neural network was able to explicitly determine, respectively, the male or female gender of the patient, or in the form of an indication of the third mark, if the neural network is not it was possible to clearly determine the gender of the patient,

Moreover, during the operation of the said neural network, only the prepared image is analyzed without additional input data in the form of data on the gender and age of the patient depicted in this image,

generating, using at least one processor, a report that contains at least one file containing an indication of the presence or absence of pathology;

transmitting a report to the doctor’s device using a communication device,

receive a report on the doctor’s device using a communication device,

using the doctor's device and using the report, a doctor's decision is made on the condition of at least one organ and/or cavity of the examined patient recorded in said image and/or on the treatment of the examined patient.

Technical result

The present invention improves the efficiency of devices and methods for determining pathology based on the analysis of medical images. This ensures:

- increasing the accuracy of determining pathologies;

- no need to process and transmit additional data about the patient during the immediate operation of the system;

- acceleration of data processing compared to systems that, in addition to transmitting the medical image itself, require processing and transmission of additional data about the patient in the process of direct work;

- reducing the load on data networks and on the medical decision support system as a whole and its components;

- ensuring increased confidentiality of the study, since only a depersonalized image is transmitted over the network;

- simplification of pathology detection compared to devices that require processing and transmission of many types of additional data about the patient;

- reduction of requirements for the qualifications of medical personnel;

- reducing the influence of the human factor (attention, fatigue, responsibility);

- reduction of requirements for device performance and a corresponding reduction in costs and energy consumption, the ability to reduce the size of devices;

- reducing requirements for devices processing additional data.

It should be understood that not each of the embodiments may simultaneously provide all of these advantages over all known solutions from the prior art. Accordingly, some embodiments may have only some of these advantages or other advantages relative to some known solutions.

These and other advantages of the present invention will become apparent upon reading the following detailed description taken in conjunction with the accompanying drawings.

Brief description of drawings

In FIG. 1 is a schematic diagram of a medical decision support system according to the present invention.

In FIG. 2 shows an example of an image to be processed in the pathology detection device.

In FIG. Figures 3 and 4 show examples of images that should not be validated.

In FIG. 5 is a block diagram of a pathology detection apparatus according to the present invention.

In FIG. 6A, 6B shows an example of cropping an image.

In FIG. Figures 7 and 8 show example images for patients at different ages.

In FIG. Figure 9 shows a block diagram of a neural network that determines pathology taking into account the gender and age of the patient.

In FIG. Figure 10 shows an example of an image on which areas of interest are plotted as a heat map.

In FIG. Figure 11 shows an example of an image on which the contours of areas of interest are plotted.

In FIG. 12 is a flowchart of the pathology detection method according to the present invention.

It should be understood that the figures may be presented schematically and not to scale and are intended primarily to enhance understanding of the present invention.

Detailed description

General overview of the system

Next, with reference to Fig. 1 will generally describe the clinical decision support system 100.

The medical organization (MO) 110 contains a device 120 for obtaining medical images. Medical images may include, for example, X-ray images, MRI (magnetic resonance imaging) images, CT (computed tomography) images, and other images obtained using radiological (i.e., radiation) or non-radiological (i.e., camera) techniques. diagnostics A medical image may also be called a diagnostic image. Accordingly, the device 120 may be an X-ray imaging device (may also be interchangeably referred to as an X-ray diagnostic machine, X-ray machine, X-ray unit, X-ray, etc.), a fluorographic imaging device, an MRI imaging device, device for obtaining CT images, etc. Hereinafter, for convenience of description, medical imaging device 120 may be referred to as an X-ray machine, but other possible embodiments of the present invention should be taken into account as noted above.

A medical professional uses a medical imaging device 120 to perform a diagnostic examination of the patient's organs. Hereinafter, in this document, for convenience of description, chest x-ray examination may be mentioned as an example of a diagnostic examination, however, it should be understood that in other embodiments of the present invention, it is possible to conduct an MRI examination, CT examination, etc., and organs The subjects to be examined can be any organs or cavities that can be examined using a medical imaging device.

The medical organization 110 according to the present invention can be a clinic, a clinic, a doctor's office, a hospital, a hospital, a hospital, a sanatorium, a medical aid station, a pharmacy, a mobile unit, a mobile fluorography room, or any other organization, room or installation equipped with a device 120 for obtaining medical images.

The result of the study is generated in the form of a medical image, for example, in DICOM (Digital Imaging and Communications in Medicine) format, NIfTI (Neural Imaging Technology Initiative in Informatics), Analyze, Minc1, Minc2 or other suitable open or proprietary format. If necessary, before saving, a medical specialist can apply filters to the image that, in his opinion, can improve the perception of the image. In addition, in one embodiment, any preset filters specific to a particular device or manufacturer may be superimposed on the image.

The generated image is stored in a local data store 131, such as a store based on or as part of a PACS (DICOM Image Archiving and Communication System), RIS (Radiology Information System), MIS (Medical Information System) 130 or other system or device suitable for storage of medical data. If necessary, additional targeted information can be added to the saved image, such as information about the patient, about the study, about the medical organization, about the medical specialist, etc.

MIS 130 (including PACS or RIS) can be deployed on a general purpose computer or server, on specialized equipment or on other hardware for storing, processing and transmitting data suitable for implementing the present invention and known to a person skilled in the art. In an illustrative non-limiting example, MIS 130 may be a computer having a 4-core processor, 8 GB of RAM, 60 GB of disk space for the CentOS 7 operating system, 2 TB of disk space for data, 1 TB of disk space for a database, and 2 network adapters.

Storage 131 is shown in FIG. 1 within MIS 130 is exemplary only, and it should be understood that other implementations are possible in which storage 131 is separate from MIS 130, is located within medical imaging device 120, or is located within another healthcare organization's internal storage system. Implementations are also possible in which a medical organization does not use special systems for storing and transmitting medical data at all, but uses other available storage and transmission means for these purposes, such as a personal computer or laptop.

The data to be processed is then generated using a data communication device 132, such as a PACS, RIS, MIS 130, or other suitable open or proprietary system. As with storage 131, although data transmission device 132 is shown in FIG. 1 as part of MIS 130, it can also be separate from MIS 130 or be part of another internal storage system of a medical organization.

In one embodiment, when generating data to be processed in the data transmission device 132, depersonalization (anonymization, de-identification, depersonalization) of the medical image of the organs of the patient being examined can be performed. To do this, information about the patient, which one way or another can be considered personal, is replaced with anonymized data, from which a third party will not be able to restore the original data without proper access to it. For example, when using a PACS system, the fields “PatientName” (patient name), “OtherPatientNames” (other patient names), “PatientID” (patient identifier) and, if necessary, other fields containing or related to personal data are anonymized.

In one of the optional embodiments, it may be necessary for the image to have some unique image identifier (ID) that does not contain personal data, so that subsequently, upon receiving back data with such an identifier, MIS 130 or other system or device having proper access to personal data, if necessary, could again compare the received data with the corresponding patient. In the illustrative non-limiting example of FIG. 2 identifier has the value “56HJ”. Specific methods for generating unique identifiers are known to those skilled in the art and are not the subject of the present invention. In this case, either one separate field or a set of several fields can act as a unique identifier.

The possible depersonalized data values and/or the format of the unique identifiers may be predefined and known to all devices in the system, only trusted devices, or only the pathology device 160 so that they can determine whether the transmitted data is depersonalized. For example, in the above example in FIG. 2 identifier has the format XXYY, where X is an Arabic numeral and Y is a letter of the Latin alphabet.

The depersonalization process can occur either completely automatically or, if necessary, part of the depersonalization process or the entire depersonalization process can be performed with the participation of a person who can delete or edit data through the appropriate interface of the data communication device 132 or the medical information system 130.

Depersonalized data to be processed, an example of which is shown in FIG. 2 are transmitted from the medical organization 110 directly or through the central medical information system 150 to the pathology determination device 160. Moreover, in one embodiment, depersonalization can be performed in the central medical information system 150, and in this case, non-anonymized data can first be transferred from the medical organization 110 to the central medical information system 150, and then depersonalized data can be transferred from it to the device 160 for determining pathology data to be processed. The depersonalization process can occur either completely automatically or, if necessary, part of the depersonalization process or the entire depersonalization process can be performed with the participation of a person who can delete or edit data through the appropriate interface of the central medical information system 150.

The pathology detection device 160, having analyzed the medical image contained in the incoming data based on artificial intelligence (AI) methods, generates a prediction whether there is a pathology in this image. If present, pathology detection device 160 indicates areas in the image containing pathologies. If there is no pathology, then the image does not change.

Optionally, the pathology detection device 160 can generate a result report (or protocol) containing a description of the result in the form of text information.

The generated data (one or more images and/or a report) is sent back (directly or indirectly) from the pathology device 160 to the healthcare organization 110 that requested the data processing.

The results obtained from the device 160 for determining pathology are provided to the responsible person, for example, a radiologist, attending physician or other medical specialist, or a person who has access to such information and is responsible for receiving and processing it in the medical organization 110, using the viewing tool 140 , and using them, he makes a decision about the state of the examined organs recorded in the image, namely the presence or absence of a particular pathology. If necessary, the medical specialist, taking into account the obtained results of the device 160 for determining the pathology, can make a decision about the treatment of the patient. The viewer 140 in FIG. 1 is simplistically indicated as a display 140, but in the preferred embodiment it represents a doctor's workstation (automated workstation). In an illustrative non-limiting example, a doctor's workstation may be a computer based on an Intel Core i3 processor or its equivalent, having 8 GB of RAM, 40 GB of free disk space, a DVD-R/RW CD reader, a network connection speed of 5 Mbit/s, monitor with a screen resolution of 1920x1080, and viewing can be performed using DICOM image viewers or through a web browser.

Thus, the system 100 helps to improve the accuracy of medical decision making.

An embodiment is also possible in which the data generated by the pathology determination device 160 is sent not directly to the medical organization 110 that requested the data processing, but first (directly or through the central medical information system 150) to a specialized expert organization 170 that produces medical opinions using the results operation of the pathology determination device 160, or to an external radiologist 180 serving as an expert or consultant. In this case, the medical organization 110, in response to the sent medical image, can receive from the expert organization 170 or from the external radiologist 180 (again, directly or through the central medical information system 150) an immediately ready conclusion or a preliminary conclusion that can be used for making a medical decision. This makes it possible to reduce the requirements for the qualifications and workload of the personnel of a medical organization 110 up to the possibility of not having a radiologist on staff at all, which provides a significant expansion of the geography of possible use of the system and allows you to receive highly qualified medical services in those locations where they were not previously available due to the lack of personnel with the appropriate qualifications, and on the other hand, allows radiologists to connect to the system from various remote locations without being tied to a specific medical organization.

It should be understood that in this document, blocks 170 and 180 are simplistically referred to as "expert body 170" and "external radiologist 180", however, technically from a system point of view, these blocks are equipment/devices for receiving, viewing, editing and transmitting data , which are managed/owned/used by an expert organization 170 and an external radiologist 180.

According to the present invention, the term “external” in relation to a radiologist means that this doctor is not on the staff of the medical organization 110 that performed the x-ray examination and requested processing of the resulting image, and/or is not physically located in this organization and/or does not have access to MIS 130 of this organization. In addition, it should be understood that the term “radiologist” in the context of the present invention is given a broad interpretation, that is, it is understood that this is a medical specialist who has proven qualifications (knowledge, abilities, skills and experience) in the analysis (interpretation) medical (diagnostic) images, that is, one or more of the following: analysis of X-ray results, analysis of CT results, analysis of MRI results, etc.

The expert organization 170 may be a medical or other organization charged with the functions of analyzing medical images and drawing up conclusions and having one or more specialists of the appropriate profile and proper qualifications in the analysis (interpretation) of medical (diagnostic) images, that is, one or more of the following : analysis of X-ray results, analysis of CT results, analysis of MRI results, etc.

Specialists of the expert organization 170 and external radiologists 180 can access data from a specialized workstation (doctor’s workstation) or using another suitable device, such as a computer, laptop, smartphone, tablet, VR helmet (virtual reality helmet), VR -glasses, etc.

In FIG. 1, for simplicity, one medical organization 110, one pathology device 160, one expert organization 170, and one external radiologist 180 are depicted, but it should be understood that other embodiments are possible. For example, in one embodiment, the clinical decision support system 100 may comprise multiple healthcare organizations 110 served by a single pathology device 160. In general, one or more healthcare organizations 110 may be served by one or more different pathology detection devices 160.

In this case, the medical organization 110 can send the same image to different devices 160 to determine pathology if they have different operating principles and/or are trained on different data sets, and then the medical organization 110 can have several processing results (reports, protocols, preliminary conclusions), which is somewhat analogous to a consultation and can increase the accuracy of medical decision-making.

In addition, the healthcare organization 110 may send different images to different devices 160 to determine pathology. In particular, this may be useful if they specialize in different types of research. For example, one pathology detection device 160 is designed to detect thoracic pathologies and another is designed to detect spinal pathologies. This ensures expansion of the functionality of the medical decision support system and the ability of a medical organization to receive a full range of services for processing and analysis of diagnostic studies within a single medical decision support system without the need for integration with many different systems in order to obtain services in different areas of research. In addition, developers and suppliers of pathology devices 160 may not strive to cover all possible types of examinations in an attempt to satisfy all the requirements of the medical decision support system 100, but rather concentrate on those types of examinations in which they have the greatest competence, which in terms of The system ensures increased accuracy and quality of service provision for each of the individual types of research.

Likewise, different expert organizations 170 and different external radiologists 180 can specialize in the types of examinations in which they have the greatest competence, which, from a system point of view, improves the accuracy and quality of service for each of the individual types of examinations.

The central medical information system 150 shown in FIG. 1, the expert organization 170 or the external radiologist 180 are not required elements of the proposed medical decision support system 100. Accordingly, in one embodiment, the healthcare organization 110 may communicate directly with the pathology detection device 160. This simplifies the implementation of the medical decision support system 100. Such an implementation may be convenient, for example, for cases where the number of medical organizations 110 served by the pathology determination device 160 is relatively small, and the expert organizations 170 or external radiologists 180 are not involved at all or their number is also relatively small.

In another embodiment, when the medical decision support system 100 includes multiple medical organizations 110 and/or multiple pathology devices 160, as well as expert organizations 170 or external radiologists 180, it is advisable to use a central medical information system 150. It should be noted , that the term “central” in this case indicates, first of all, not that this is a single central server that closes all possible connections, but that the central medical information system 150 occupies a place in the middle, in the center between other participants of the medical decision support system 100, acting as an intermediate system for collecting, storing and redistributing data. Moreover, depending on the requirements of a specific application, the central medical information system 150 can be either concentrated (centralized) or distributed, including implemented in the cloud.

The central medical information system 150 may include a plurality of data storage and processing devices. In one embodiment, the owner of the entire central health information system 150 may be a government entity, a municipal entity, an authorized agency or operator, or another government or non-government commercial or non-profit entity or organization. In another embodiment, individual devices within the central health information system 150 may be owned by different owners - for example, in different cities, different organizations (local operators) may provide their server equipment for use as data storage and processing devices, and a common operator may perform the functions supervision and regulation. If necessary, participants in the central medical information system 150 may be required to obtain a license to operate within this system and/or to have the right to access, store and process personal data.

It was described above that the result of the pathology determination device 160 can be sent to the expert organization 170 or to an external radiologist 180, but another embodiment is also possible in which such a result is only sent to the central medical information system 150, stored there and is marked as a preliminary result requiring the opinion of a specialized medical specialist. Subsequently, on their own initiative (for example, to the extent possible, in the presence of free time, in the need to gain experience and practice, etc.) or upon request, expert organizations 170 or external radiologists 180 with appropriate access rights access central medical information system 150, download the preliminary results requiring a conclusion to their devices, perform the analysis, write a conclusion and send the conclusion to the central medical information system 150, or, without downloading, view such preliminary results through their devices directly in the user interface of the central medical information system 150 and draw up conclusions directly in this interface. By downloading the results here we mean saving the corresponding files in the permanent memory of the device on the side of the expert organization 170 or an external radiologist 180 for further opening of these files from the permanent memory.

The term “request” according to the present invention covers both a task that can be performed on its own initiative by any physician or expert organization with the appropriate level of access, and a task that must be performed by a specific expert organization 170 or a specific external radiologist 180.

Further in this document, in some cases, for ease of description, the expert organization or external radiologist may be replaced by the single term “expert”. Accordingly, equipment/devices for receiving, viewing, editing, processing and transmitting data that is controlled/owned/used by the expert organization 170 or an external radiologist 180 may be referred to as expert equipment/device.

The request for an opinion may come, for example, from the medical organization 110 that originally requested the image processing, from the central medical information system 150 itself, or from the pathology device 160. In one embodiment, the request for data processing and/or drawing up of a conclusion may be not the medical organization in which the diagnostic study was performed, but some other organization or person. This may be applicable, for example, in the case where the user or owner of the medical imaging device 120 is a single medical organization, and the opinions on the images obtained by this device are compiled by an expert organization 170 or an external radiologist 180 and sent directly to the customer. This makes it possible to reduce the requirements for the qualifications of medical specialists using devices 120 for obtaining medical images, and to increase the geography of use of such devices, which ensures increased availability of quality medical services.

Reporting requests may be stored in the central health information system 150, for example, on a first-come, first-served basis such as FIFO (first in first out), priority queue, or other processing queue.

When the expert organization 170 or external radiologist 180 accepts one or more requests for work, the corresponding requests may be dequeued to avoid duplication and different experts performing the same work. In another embodiment, one or more preliminary results may initially be intended for analysis by a predetermined number of two or more experts, and then the corresponding request in the queue may contain a counter reflecting how many experts have accepted the request. When the counter reaches the required preset number, the corresponding request is removed from the queue.

Accordingly, as requests are added to the queue, as requests are removed from the queue, as requests are accepted for work by a specific expert, with a certain frequency, in a combination of the above, or in some other suitable manner, the above-mentioned loading parameters can be recalculated for an individual expert or for all experts.

Just as requests for an opinion regarding preliminary results from pathology devices 160 are generated, queued, and processed by the central medical information system 150, expert organizations 170, and external radiologists 180, requests to devices can be generated and queued. 160 to determine pathology for processing images received from medical organizations 110. Pathology devices 160 can perform image processing on a specific request or poll the central health information system 150 for pending image processing requests that the device is capable of processing. In addition, depending on the load of the pathology devices 160 and their requirements, the central medical information system 150 can adjust the number and timing of image processing requests. Image processing requests can also be assigned different priorities.

Images awaiting processing may be grouped into batches (series) for transmission to pathology determination device 160. Accordingly, the pathology determination device 160 may batch process the received images. Grouping into packages can be performed both in the medical organization 110 and in the central medical information system 150. The central medical information system 150 can change the size and content of packages received from medical organizations 110 - for example, sort images by their resolution in pixels, by priority or other parameters and create new packages, and if necessary, images from different medical organizations 110 can be added to the same package.

In one embodiment, the central medical information system 150 may perform initial validation or preprocessing of images. Also, primary validation or preprocessing is possible on the side of the medical organization 110.

In particular, an attempt may be made to extract an image from the data to be processed. If the attempt fails, it is concluded that the image file is damaged or cannot be read. The reason for this may be factors such as the absence of an image in the data, the inability to read metadata, the presence of any anomalous and unaccounted for tag values, etc. In this case, the data is not transferred for processing, and an appropriate instruction is created based on it.

A check may be made to see if the data attached to the image processing request matches the request itself. For example, if the request specifies that an AP chest X-ray is to be processed, a check may be made to see if the attached data contains an AP chest X-ray. Various options for implementing such a check are known to specialists in the art and may include, for example, a pre-trained neural network (neural network) that produces the appropriate classification, or other computer vision methods. If the check fails, then such data is not transferred for processing, and an appropriate instruction is created for it. Examples of such errors, when the study is declared as a direct chest x-ray, but in fact is not one and cannot be processed, are shown in Fig. 3 and 4.

In addition, a check may be performed whether the image is positive or negative. Various options for implementing such a check are known to specialists in the art and may include, for example, a pre-trained neural network (neural network) that produces the appropriate classification, or other computer vision methods. If the test determines that the image does not meet the input requirements of the pathology detection device 160, proper conversion of the image to a negative or a positive may be performed.

If a region of interest (VOI) is preset in the image parameters by the examining physician, a corresponding VOI LUT can be performed to produce an image containing only the region of interest. For example, if a medical professional performing a chest x-ray specified an area of interest, then according to the parameters embedded in the image, cropping is applied, resulting in only the lung area in the image.

A check may also be made to see if the image has a size (pixel resolution) equal to or greater than a predetermined minimum size supported by at least one of the pathology devices 160 capable of processing this type of examination. For example, the resulting raw chest X-ray image to be processed has a size of 800x800 pixels, and the clinical decision support system 100 contains three pathology devices 160 capable of processing this type of examination, of which two devices support a minimum input image size of 1024x1024 pixels, and the third device supports a minimum size of 512x512 pixels. Accordingly, a request to process the resulting image will only be sent to the third device 160 to determine the pathology. If the original image has a size smaller than the minimum size supported by at least one of the devices 160 for determining the pathology, then such an image is not transferred for processing, and a corresponding indication is created from it. This improves the accuracy of image processing and reduces the likelihood of errors and the need to re-analyze some images by different devices 160 to determine pathology, and thereby improve the quality of medical services.

If the pathology detection device 160 that accepts the image processing request requires certain input image parameters, then the central medical information system 150 or the healthcare organization 110 itself (for example, when the physician decision support system 100 does not contain the system 150 or if the system 100 contains only one pathology device 160 capable of processing this type of examination) can perform image preprocessing. In particular, in one embodiment, luminance and chrominance conversion of the original image may be performed. For example, the color depth of the original image may be 12-16 bits, and the conversion may result in an image with a color depth of 8 bits. In another embodiment, the image may be compressed to a desired size. Various compression methods known to those skilled in the art are applicable, so a detailed description of the compression process is not provided here.

The above functions of primary validation and preprocessing improve the flexibility and adaptability of the medical decision support system to the various technical capabilities of system participants, which ensures a high degree of scalability and the ability for as many participants as possible to participate in the system, which increases the availability of high-quality medical services.

In addition, a check can be made in the central medical information system 150 whether the data to be processed is depersonalized. To do this, it is checked whether the fields that relate to personal data contain any values, and if they do, then whether these values are depersonalized. For example, if the “PatientName” field is empty or contains a preset value of “0” or “Anonymous”, as indicated in FIG. 2 and Fig. 4, then this field is considered to be depersonalized, and the field with the value “Venus de Milo” indicated in FIG. 3, is not depersonalized. Data that has not been verified may be depersonalized by the central medical information system 150 before generating a request for data processing or a request for a conclusion.

If unique identifiers are used, a check may also be made to see if the unique image identifier is in a predefined format. For example, if the image ID must strictly be in the format XXYY, then the image ID of FIG. 2 having the value "56HJ" will pass the test, and the image IDs from FIG. 3 and 4 with values of "0" and "C3PO" will fail the test. Identifiers whose format does not meet the requirements can be converted by the central medical information system 150 before generating a request for data processing or a request for drawing up a conclusion, or the corresponding data is not transferred for processing and an indication of the impossibility of processing is generated.

It should be noted that the pathology device 160 or adjudicator devices may support several different valid depersonalized data values and unique identifier formats, in which case the central health information system 150 may check to see if the field values in the received data match. and the format of the received image identifier to at least one of the corresponding supported values and formats of this device 160 to determine the pathology or device of the expert ready and able to accept the request for work. If the value does not meet a predetermined acceptable value or the image identifier is not in the correct format, then the central medical information system 150 may perform a mapping of the data to those depersonalization requirements supported by the pathology device 160 or the examiner's device.

The above depersonalization functions make it possible to increase the flexibility and adaptability of the medical decision support system to the various technical capabilities of system participants and significantly reduce the requirements for system participants associated with the processing of personal data, which ensures a high degree of scalability and the ability for as many participants as possible to participate in the system, which increases the availability of high-quality medical services.

In one embodiment of the present invention, the central medical information system 150 may store not only those medical images that require analysis and reporting, but also training data sets containing properly labeled medical images that are intended, among other things, for training one or more devices 160 for determining pathology, namely for training the neural networks and/or computer vision algorithms built into them. Thus, there is a single archive of training data sets that can be accessed by each of the multiple pathology devices 160 included in the medical decision support system 100, while the pathology devices 160 themselves do not need to be separately stored, maintained, updated and replenished. your own training data sets, as well as look for other third-party sources of training data. Thus, the requirements for system participants are reduced, which ensures a high degree of scalability and the ability for as many participants as possible to participate in the system, which increases the availability of high-quality medical services.

In yet another embodiment of the present invention, the central medical information system 150 may store training data sets in which medical image labeling contains patient personal information such as name, gender, age, etc. Each person, device, or system accessing the system has an access level assigned to them, and depending on a particular user's access level, the central health information system 150 regulates the amount of data that user can view. For example, a user with a low access level can view the images themselves, and from their description (markup) has access only to the description of the pathology, if any. A research scientist, for example, may have access to patients' sex, age, and comorbidity information, if any, to conduct a study, but does not need to know the patient's name, so the central health information system 150 does not show the patient's names to him. At the same time, the attending physician or expert preparing the report, having received a high level of access, can search the archive for previous images of a particular patient, entering, for example, his name or medical policy number, and the central medical information system 150 provides in response to such search for all images of a given patient available in the archive. It is also possible for an implementation in which the central medical information system 150 provides all of the archived images of a given patient without the use of personal data, but simply in response to a request for previous images of the patient depicted in a particular image being analyzed or viewed by the user. This improves the quality and accessibility of education, the quality of medical services and the availability of high-quality medical services.

Moreover, one or more pathology devices 160 may also access all or part of the personal data contained in the markup. If such data is needed by the pathology detection device 160 only for the purpose of training its neural networks and/or computer vision algorithms, then it is used only during the training process and is not stored. Due to this, the accuracy of training and prediction of pathology is significantly increased, which leads to an increase in the quality of medical services, while the requirements for system participants are reduced (for example, in terms of processing and storing personal data), which ensures a high degree of scalability and the ability to participate in the system as as many participants as possible, which increases the availability of high-quality medical services.

Structure of the device for determining pathology

Next, with reference to Fig. 5, a device 200 for determining pathology based on medical image analysis according to the present invention will be described in detail. It should be noted that the pathology determination device 200 is entirely consistent with the pathology determination device 160 shown in FIG. 1, and has the same functions and capabilities, if applicable and consistent with the description of this section. In the following, for convenience of description, the organs being examined may be referred to as the chest organs and the medical image may be referred to as an x-ray image, but it will be appreciated that the principles of the present invention are applicable to other types of examinations as discussed above herein.

The pathology detection device 200 includes a data receiving unit 210, a data storage unit 220, a data validation unit 230, an image preparation unit 240, a pathology prediction unit 250, a report generating unit 260, a data transmission unit 270, and a training unit 280. Depending on the specific application, some of these blocks may be missing, as will be explained in more detail later in this document.

The data receiving unit 210 receives data to be processed containing a medical image, such as a chest X-ray image of a patient being examined. The receiving unit 210 may be a separate chip, network card, or other suitable means capable of communicating with external devices in a wired and/or wireless manner, such as through a local area network (LAN), Internet, etc. protocol. using Ethernet, optical fiber, WiFi, 4G, etc. technologies.

The data storage unit 220 stores the data received by the data receiving unit 210 so that other units of the device 200 can use it at an appropriate time. Received data can only be stored for the duration of its processing and erased when its processing is completed. For these purposes, in one embodiment, a short-term storage device such as RAM or the like is used. In another embodiment, the received data may optionally be stored for a longer period of time than the immediate processing time, and then a non-volatile storage device such as a hard disk or the like may be used.

Additionally, the data storage unit 220 may store short-term or long-term data and/or files resulting from or during the operation of other units of the device.

The data validation unit 230 receives the data to be processed directly from the data receiving unit 210 or retrieves it from the data storage unit 220. Data validation unit 230 then checks whether the received data is suitable for processing.

As mentioned above in the description section of the medical decision support system 100, some of the validation operations can be performed on the side of the medical organization itself or on the side of the central medical information system 150. In this case, if the pathology detection device 200 knows which validation operations have already been performed for the data to be processed, the data validation unit 230 may not perform these operations, thereby simplifying and speeding up the processing and thereby increasing productivity. However, in another embodiment, pathology detection device 200 may not be aware of which validation steps have already been performed, or may re-run them for additional double-checking purposes. This can provide improved processing quality.

Moving specifically to the validation operations, an attempt is made to extract an image from the data to be processed. If the attempt fails, it is concluded that the image file is damaged or cannot be read. The reason for this may be factors such as the absence of an image in the data, the inability to read metadata, the presence of any anomalous and unaccounted for tag values, etc. In this case, the data is not transferred for processing, and an appropriate instruction is created based on it. Performing this operation in the data validation block 230 is preferable because even if it has already been performed previously by other devices, the data may be corrupted in the process of being sent to the pathology device 200 or may be in a format that is not currently available for one reason or another. device 200.

A check may also be made to see if the data to be processed is consistent with the type of processing being performed in the pathology determination device 200. For example, if the pathology detection device 200 is configured to process AP chest X-rays, a check may be made to determine whether the enclosed data contains an AP chest X-ray. Various options for implementing such a check are known to specialists in the art and may include, for example, a pre-trained neural network (neural network) that produces the appropriate classification, or other computer vision methods. If the check fails, then such data is not transferred for processing, and an appropriate instruction is created for it. Examples of such errors, when the study is declared as a direct chest x-ray, but in fact is not one and cannot be processed, are shown in Fig. 3 and 4.

In addition, a check can be made to determine whether the data is depersonalized. To do this, it is checked whether the fields that relate to personal data contain any values, and if they do, then whether these values are depersonalized. For example, if the “PatientName” field is empty or contains a preset value of “0” or “Anonymous”, as indicated in FIG. 2 and Fig. 4, then this field is considered to be depersonalized, and the field with the value “Venus de Milo” indicated in FIG. 3, is not depersonalized. If the data is not depersonalized, then the corresponding image is not sent for processing and an indication of the impossibility of processing is generated based on it.

If unique identifiers are used, a check may also be made to see if the image's unique identifier is in a predefined format. For example, if the image ID must strictly be in the format XXYY, then the image ID of FIG. 2 having the value "56HJ" will pass the test, and the image IDs from FIG. 3 and 4 with values of "0" and "C3PO" will fail the test. Images whose identifier format does not meet the requirements are not sent for processing, and an indication of the impossibility of processing is generated for them.

It should be noted that the pathology device 200 may support several different valid depersonalized data values and unique identifier formats, in which case the data validation unit 230 may check whether the field values in the received data and the received image identifier format match at least one of the corresponding supported values and formats. If the value does not match the predefined valid value or the image identifier is not in the correct format, then the assumption is made that the received data is not depersonalized. In this case, they are not submitted for processing, and a corresponding instruction is created for them.

It may also be checked whether the read image has a size (pixel resolution) equal to or greater than a predefined minimum size supported by the device 200—for example, 1024 x 1024 pixels or 800 x 800 pixels, depending on the requirements of the particular application. If the original image is smaller than the minimum size, the device 200 may not provide sufficient accuracy, so the image is not sent for processing and an appropriate indication is generated from it.

In addition, a check may be performed whether the image is positive or negative. Various options for implementing such a check are known to specialists in the art and may include, for example, a pre-trained neural network (neural network) that produces the appropriate classification, or other computer vision methods. If the test determines that the image does not meet the input requirements of the pathology detection device 200, then the image preparation unit 240 can subsequently perform proper conversion of the image to a negative or a positive.

Validation allows you to filter out data for which analysis cannot be carried out or the accuracy of processing will have a deliberately low accuracy. Accordingly, the load on the most resource-intensive part of the analysis is reduced and the accuracy of prediction increases. Additionally, filtering out images that are not depersonalized ensures that device 200 does not process personal data, reducing implementation and certification requirements.

The image preparation unit 240 receives from the data storage unit 220 and/or from the data validation unit 230 a validated medical image and performs preliminary transformations on it in order to prepare it for direct use in the pathology prediction unit 250.

In particular, the image preparation performed in block 240 may be as follows.

The medical image to be processed is read and, if necessary, converted to grayscale with a preset color depth. The brightness and color parameters of the original image depend on the parameters set by the radiologist when working with the device for obtaining medical images, and are contained in the image metadata. For example, the color depth of the original image may be 12-16 bits. In this case, for example, a matrix of integer pixel values from 0 to 255 can be obtained as a result of reading and converting to an image with a color depth of 8 bits. Also, as indicated above, the appropriate conversion of the image to negative or positive can be performed.

The original size of medical images can be relatively large - for example, for x-rays it averages 2500x2500 pixels, that is, 2500 pixels in height (vertical) and 2500 pixels in length (horizontally). Images of this size can generally be processed using machine learning methods, but such models consume a lot of resources.

In addition, the range of sizes of input medical images can be quite large, which can cause inconvenience if trying to process each individual image directly at its original size. In particular, it becomes difficult to select a set of processing methods that are equally effective for different sizes.

In this regard, before processing large medical images, it is preferable to reduce their size. This allows, without significant loss of quality, to significantly reduce the resources spent: time, computing power, energy consumption. Various methods of reducing size are applicable - for example, compression, cropping, etc. Compression methods are known to those skilled in the art, so a detailed description of the compression process is not provided here. As for cropping the image, it will be described below.

For example, in one embodiment, if a region of interest (VOI) is preset in the image parameters by the medical examiner, a corresponding image cropping (VOI LUT) may be performed to produce an image containing only the region of interest. For example, if a medical specialist performing a chest x-ray specified an area of interest, then according to the parameters “built into” the image, cropping is applied, as a result of which only the lung area remains in the image.

In another embodiment, in addition to or instead of the above-mentioned VOI LUT cropping, it searches for a region of interest in the image and crops the image by clipping away portions of the image not included in the found region of interest. For framing purposes, computer vision algorithms can be used that are trained to search the image for a region of interest that includes only the area bounded by the organs or cavities to be analyzed.

To do this, first the size of the input image is reduced to a single first size. The first image size is predefined - for example, 800x800 pixels. The value chosen depends on the requirements of the specific application and the performance of the equipment used.

In particular, at the design stage, an estimate may be made of what is the smallest size of medical image that can be supplied to the device 200 from the medical organization 110 and/or from the central medical information system 150, and accordingly set the above-mentioned first size that is less than or equal to the smallest possible size of the input medical image.

However, in practice, there may also be situations where the smallest possible size of the input medical image is too small or the “less than less” principle is, for one reason or another, not suitable for a particular application. Then the first size to which all input medical images are reduced can be set according to a different principle, and in such a scenario, some of the input images will be compressed, some will be stretched, and some will remain unchanged.

The image is then cropped so that it only covers the lungs. In particular, the image is analyzed separately by different pattern recognition methods - for example, the Viola-Jones method using a cascade classifier based on Haar features (Haar Cascades), the Histogram of Oriented Gradients (HOG) method and the method based on local binary patterns (Local Binary Patterns, LBP), resulting in the formation of several rectangles, each of which indicates an area that covers only the right and left lung. The aspect ratio of the rectangles depends on the size and shape of the lungs. The analysis may be performed in parallel, sequentially, or in any other suitable manner. The pattern recognition algorithms used must be pre-trained to recognize lungs in an image having the first dimension.

The Viola-Jones method, the directional gradient histogram method, and the local binary pattern method mentioned above are slightly faster than traditional convolutional neural networks and require a slightly smaller training set.

The Viola-Jones method has very high accuracy when the recognized object in the image has a rotation angle of no more than 30-35 degrees. Accordingly, it is well applicable to the analysis of X-ray images, since the patient's position is typified to obtain the image, for example, in anterior projection or lateral projection.

The directional gradient histogram method is weakly sensitive to displacement, changes in image scale and brightness, and somewhat sensitive to changes in object orientation. However, for the reasons stated above, it is also well suited for X-ray analysis.

Methods based on local binary patterns generally have high recognition speed and accuracy, and are also weakly sensitive to image brightness and changes in object orientation.

One common rectangle is then constructed (see example in Fig. 6A), which includes all previously obtained rectangles (1, 2 and 3 for the right and left lung). The input (downscaled) image is cropped to this overall rectangle, the remaining image regions are discarded, so the cropped image contains only the lung region (see example in Fig. 6B). In one embodiment, each side of the common rectangle includes at least one of the sides of the rectangles obtained as a result of image analysis, and none of the sides of these rectangles extend beyond the boundaries of the common rectangle.

The results of different pattern recognition methods are in general not identical (in the example in Fig. 6A, rectangles 1, 2 and 3 are different), therefore such a combination of results allows you to cover the lungs completely without erroneously cutting off their parts, but at the same time, focus specifically on areas of the lungs, discarding areas not of interest for this study.

Additionally, there is a possibility that one or two of the pattern recognition methods used will fail to find the lung in the image. At the same time, the likelihood that all three methods will not find a lung is extremely small. Consequently, the use of several pattern recognition methods at once is justified, since it provides protection against data loss and increases the accuracy of prediction.

Other embodiments are also possible, when around the rectangles obtained as a result of the analysis, as shown in Fig. 6A, a small border zone (margin) of one or more pixels is additionally captured if the particular analysis methods chosen are not sufficiently accurate and may result in lost regions of interest. The sizes of indents up, down, right and left may vary. Optimal indentation sizes can be selected depending on the results demonstrated by the trained algorithms and the accuracy requirements. In a specific non-limiting example, if the image size being processed is 512x512 pixels and all three of the above pattern recognition methods are applied, the padding may be 15 pixels (or less if the padding extends beyond the border of the image). This avoids data loss and improves prediction accuracy.

The aspect ratio of the image resulting from the crop (i.e., overall rectangle) (Fig. 6B) depends on the size and shape of the organs or cavities to be analyzed. For further processing it is necessary to bring it to a unified format. To do this, the image size is changed to the second size. The second image size is predefined - for example, a square of 224x224, 320x320 or 512x512 pixels. The selected value depends on the requirements of the specific neural network used further. As for the first image size indicated above (800x800 pixels), it allows you to significantly reduce the size of the original image to speed up processing, but at the same time, as a result of cropping, you still get a relatively large image that would not have to be stretched to bring it to the second size in most cases , which could potentially reduce processing accuracy.

If necessary, the image can be normalized. In particular, pixel values from the original range (for example, [0...255]) are reduced to the range required by the neural network used further (for example, [0...1]). Various normalization methods are known to those skilled in the art and are not disclosed in detail here. In an illustrative non-limiting example, a simple normalization may be applied (X-Xmin)/(Xmax-Xmin), where X is the current pixel value and Xmin and Xmax are the minimum and maximum pixel values in the image being normalized.

It should be understood that there may be embodiments in which image normalization can be performed at an earlier stage - for example, even before the first compression of the image. However, it is preferable to perform it after reducing the cropped image to the second size, since this allows you to increase the accuracy at each of the previous stages of image preparation (pre-processing), as well as somewhat reduce the amount of calculations performed directly during normalization.

This creates an image that is ready for further processing.

The pathology prediction unit 250 receives from the data storage unit 220 and/or from the image preparation unit 240 a prepared medical image limited to the area of interest and analyzes it. In particular, the pathology prediction unit 250, using a pre-trained neural network, determines the presence or absence of pathology and, if present, determines the most likely boundaries within which it is located.

There are a number of pathological conditions of organs and cavities that can be identified by medical imaging by a physician. Creating a device that, based on AI, would analyze images and give an accurate indication for each of the many possible pathologies is difficult and seems impractical, since many pathologies have similar signs, and just a depersonalized image is often not enough for a more detailed prediction, even if neural networks are trained on large, well-labeled data sets.

Accordingly, the present invention does not propose to analyze the differential probability of a particular pathology, but instead to determine the probability of having at least one radiological sign from a predefined set of possible pathologies. For each organ and/or cavity or for each type of diagnostic test, a set of possible pathologies is preset, which the pathology detection device can learn to identify and which the pathology detection device can detect during operation.

For example, for a type of examination such as an AP chest x-ray, a predetermined set of possible pathologies that can be detected by a pathology detection device may be as follows:

- Pleural effusion;

- Pneumothorax;

- Atelectasis;

- Center of darkening;

- Infiltration/consolidation;

- Dissemination;

- Cavity with decay;

- Cavity with liquid level;

- Calcification/calcified shadow in the lungs;

- Violation of the integrity of the cortical layer.

It should be noted that the term “pneumonia” unites a large group of diseases, each of which has its own etiology, pathogenesis, clinical picture, radiological signs, characteristic laboratory findings and treatment features. Considering that many of the above radiological signs can be observed in pneumonia, the present invention makes it possible to detect, among other things, signs of pneumonia.

In relation to other organs and types of studies, the predefined set of possible pathologies may differ.

A doctor who has received the result of the above determination, has certain experience, and also, if necessary, has additional information about the patient, such as anamnesis, medical history, complaints, physical condition, test results, etc., can have a more complete picture and make a more accurate conclusion. Accordingly, it should be noted that the result of the proposed device is not a clinically significant medical opinion, but is used to support medical decision-making, and the clinical opinion should be made by a radiologist.

Taking into account the above, the authors of the present invention conducted a series of studies and experiments to determine the presence of organ pathology in a medical image using machine learning using the example of the chest organs, during which they found that the approach described below shows the best results.

In particular, the image processing performed in block 250 may be as follows.

The neural network interprets the resulting image in order to identify signs of pathologies, and the result of the work is the probability of the presence of at least one radiological sign from a predetermined set of possible pathologies characteristic of the corresponding organs or cavities. For example, for the chest organs, the probability of having at least one radiological sign from the above list is determined.

A feature of the present invention is that the neural network is pre-trained to take into account additional data about the patient, such as gender and age.

In the process of studying errors in neural networks that are trained only on pathology data and analyze only images, the authors discovered that some errors are associated with the gender characteristics of patients. For example, a neural network can sometimes make mistakes in cases where the size of a woman’s mammary glands differs from the average size. Also, the presence of breast implants in women often causes difficulties for the neural network and can be incorrectly recognized as a pathology. There may be situations where the nipples of the mammary gland are visible in the image, which can also lead to processing difficulties.

In addition, the authors noticed that traditional neural networks do not take into account the natural gender-age changes in anatomy that occur in human bone structures and internal organs. For example, as people age, changes may occur such as fibrosis (especially hilar), enlargement of the heart (common), elongation and sclerosis of the aorta (common), thickening of the cartilage tissue in the ribs, osteoporosis in the bones, decreased density of the bone structure (leading to increased transparency of bones in the images), deforming osteoarthritis in the joints, decreased height of the vertebral bodies, development of subchondral sclerosis, the presence of listhesis, styloid and pommer hernia, etc. Accordingly, the older a person is, the more such changes can become. It is known that, in general, there may be differences in the nature of age-related changes in the body between men and women associated with lifestyle. There are also studies showing that changes in men and women, at least in some age intervals, occur with different intensities.

Many of these changes appear in medical images, and some degree of change may be considered acceptable for a certain age, but traditional neural networks do not take this into account and may erroneously predict that the patient is sick. Since the proposed device, as mentioned above, does not provide a differentiated probability of a specific pathology, but only predicts the probability of the presence of signs of at least one pathological change, an erroneous indication of the presence of pathology in an image can complicate the work of a medical specialist with such an image.

For example, in FIG. Figure 7 shows a sample x-ray image for a patient under 65 years of age whose lungs and spine are in age-appropriate condition.

In FIG. Figure 8 shows a sample x-ray image for a patient aged 72 years. From the side of the cardiovascular system, in this image there is an enlargement of the heart, an enlargement of the superior vena cava, congestive vessels mainly in the lower sections, an enhanced pulmonary pattern; from the skeletal system - arthrosis of the 1st costoclavicular joint on both sides, spondylous growths of the vertebrae on the lateral sides. All these signs are visible in the image, and despite the fact that they are generally acceptable for a person at this age, a neural network that does not take into account the patient’s age may mistakenly consider that he is sick, and the medical specialist who received such a preliminary conclusion will have to spend extra time to study a photo that doesn't really require that much attention.

The state of the art does not propose specific methods for solving these problems, so the authors proposed their own method of how to take into account data on the gender and age of the patient in order to increase the accuracy of predicting pathologies.

According to the present invention, a neural network 300, a simplified example of the architecture of which is shown in FIG. 9 is such that for a predetermined number of layers (preliminary section 310) it analyzes the image in a conventional manner, after which the intermediate data is fed into three subnetworks, each of which is designed to predict a different output.

In particular, it is proposed to use a convolutional neural network in which each layer is sequentially connected to each subsequent layer. The DenseNet-121 (Densely Connected Convolutional Network) architecture is used as an example since it has shown the best results for the purposes of the present invention. For example, as shown in FIG. 9, the preliminary section 310 contains a convolution block 320, a pooling block 330, 4 dense blocks 341-344, 3 transition blocks 351-353. The more detailed architecture of DenseNet is known to those skilled in the art and is not disclosed herein. It should also be understood that in other embodiments, a DenseNet with a different index, such as DenseNet-161, -169, -201, -264, etc., may be used if its performance and accuracy meet the requirements of a particular application. The preliminary section 310 ends with the last dense block, without using post-pooling with a single linear transform known from traditional DenseNet.

After the last dense block (the fourth dense block 344 in the example of FIG. 9), a split occurs and the data obtained from the analysis of the preliminary section 310 is then analyzed separately by each of the three subnets 360, 370, 380. The data format of the output of each subnet may vary depending on the specific implementation.

The first subnetwork 360 predicts the probability of the presence of pathology in the analyzed image. This subnet is implemented as a classifier. In a specific example, the classifier, using a linear function, obtains an intermediate set of output features from a set of input features, performs a sigmoid activation function, and then again, using a linear function, obtains the final set of output features corresponding to the presence or absence of pathology. Specifically, in the example in FIG. 9, the first subnet 360 may have the structure 1024-18-2, where 1024 is the set of input features, 18 is the intermediate set of output features, 2 is the final set of output features (healthy or sick).

Accordingly, in one embodiment, the output of the first subnet 360 may only output a binary number indicating whether a pathology is present in a given image. “0” may indicate the absence of pathology (that is, that the patient is healthy), and “1” may indicate the presence of pathology. The opposite indication is also possible, where “1” means healthy, “0” means probably sick. In another embodiment, the output of the first subnetwork 360 may output a number in some range, for example, from 0 to 1 or from 0 to 100, indicating the probability of the presence of pathology (the structure of the first subnetwork 360 in this case will be different from the above structure 1024-18- 2). A text format for the indication is also possible.

The second subnetwork 370 predicts the expected gender of the patient. This subnetwork, using a predefined function from a set of input features, obtains a set of output features corresponding to the expected gender of the patient. Specifically, in the example in FIG. 9, the second subnetwork 370 may have the structure 1024-3, where 1024 is a set of input features, 3 is a set of output features (male, female, or other).

The presence of the third gradation (other) in this case is due to the fact that in the available training data sets, some of the images were not marked by gender, and the proportion of these images was such that it would be inappropriate to discard them and not use them for training. Thus, as a floor marking for such pictures, it was necessary to indicate “other” and introduce such a gradation at the output of the second subnet 370, so that it eventually also began to classify some of the pictures as this gradation. However, this provided the ability to use incompletely labeled datasets, which significantly increased training capabilities, and the accuracy of pathology detection increased.

Moreover, in modern conditions, when medicine provides many opportunities to change one’s body, and society increasingly recognizes non-binary gender identity and a third gender, a problem may arise with both the physical identification of binary gender by the neural network in patients who have changed their body, and with image markings for patients who independently indicate in the “gender” column a non-binary value other than “man” or “woman”. The presence of a third gradation of gender (other) in the present invention allows all such cases to be taken into account. Accordingly, the present invention improves the accuracy of prediction by providing both the ability to use images of such patients for training a neural network, which increases the volume of available training data sets, and the ability to analyze images of such patients.

Accordingly, in one embodiment, if only fully labeled data containing only 2 genders (male and female) are used for training, then the output of the second subnet 370 may output a binary number indicating the expected gender of the patient, where "0" is male, and “1” is female, or vice versa, “1” is male, and “0” is female. In another embodiment, the output of the first subnet may output text where "m", "man", "male", "m", "male", etc. – male gender, and “f”, “wives”, “female”, “f”, “female”, etc. – female, “d”, “dr”, “other”, “o”, “other”, etc. – other. If not only fully labeled data is used for training and/or the data contains labeling with gender assigned to the “other” category, then the output of the second subnet 370 can also display an additional number or text indicating that the neural network was unable to explicitly classify the patient's gender as male or female, so he is categorized as “other.”

The third subnetwork 380 predicts the expected age of the patient. This subnetwork, using a predefined function from a set of input features, receives a set of output features corresponding to the expected age of the patient. Specifically, in the example in FIG. 9, the third subnetwork 380 may have the structure 1024-10, where 1024 is a set of input features, 10 is a set of output features (0-10 years, 10-20 years, ..., 90 or more years).

Accordingly, in one embodiment, the output of third subnet 380 may output a number indicating the patient's expected age specifically in years—eg, 25 years, 68 years. In another embodiment, the output of the third subnetwork 380 may output an indication of the age range that the patient most likely falls within. For example, if we divide ages into 10 equal ranges of 10 years, then the output of the third subnet 380 indicates which of the 10 ranges the patient most likely belongs to using the range number - for example, “4”, that is, from 40 to 50 years old, or by direct reference - for example, "from 80 to 90 years." Additionally, similar to the second subnet 370, if training uses more than just fully labeled data, the output of the third subnet 380 may output an additional number or text indicating that the patient's age is not determined. This enables the use of incompletely labeled datasets, which increases learning capabilities and accuracy of pathology detection.

Gender and age predictions are used only during the training phase, as will be described later, and pathology prediction is used both during the training phase and during the main operation of the device. Accordingly, if technically possible, during primary operation of the device, the second subnet 370 and the third subnet 380 can be disabled, which can reduce power consumption and computing resources.

As for training data, it is important to note that one of the problems that arises when training a neural network is the fragmentation of training data sets (datasets), since medical and other organizations preparing such data may have different profiles. For example, one organization may specialize in the treatment of tuberculosis, another organization may specialize in pediatrics, another organization may have a broad profile, etc. In addition, medical organizations may be located in different countries. Accordingly, the data supplied by each of these organizations for training may have significantly different approaches to the description of images, the distribution of pathologies found in the images, and the gender and age distribution of the patients examined.

The present invention effectively overcomes these shortcomings so that many different training data sets can be used for training, which significantly improves the quality and accuracy of pathology detection.

In particular, with respect to the varying distribution of pathologies occurring in images, according to the present invention, the device 200 is trained to find signs of any pathology in the image from a predefined set of possible pathologies, but does not provide a differentiated probability of a particular pathology, which eliminates the need for a sufficient number of images from each particular pathology. pathology for each gender and age and thereby eliminates the dependence on the content of the data sets used for training.

With regard to the different distribution of examined patients by age, the authors of the present invention found that if you indicate not a specific age, but divide the ages into equal ranges - for example, 10 years, 15 years, 7 years, then the device demonstrates high efficiency: high prediction accuracy, high operating speed and learning speed, smaller required amount of training data, as well as less dependence on the content of the data sets used for training. The number of ranges and their size may vary depending on the specific application, including the age of the training data. It should be understood that too large ranges can lead to loss of information, since the difference, for example, between a person in the active growth phase and a middle-aged person or between a middle-aged person and a person in the aging phase may be lost. On the other hand, too small ranges can confuse the neural network, since it will have to make distinctions between people of almost equal age, despite the fact that the same age-related changes at the same stage may not occur suddenly in people, but in all at relatively different ages. Accordingly, according to the present invention, the optimal size of the ranges is from 7 to 15 years.

Regarding the different approach to image description, this issue is discussed later in this document.

Next, the preliminary training of the model will be described in more detail. In particular, it is produced using a supervised learning method based on a set of medical images that have been pre-processed and labeled by medical specialists with sufficient qualifications - for example, radiologists. In particular, as a result of visual examination of the images, the doctor indicates whether there is a pathology in the presented image. For example, “0” may indicate the absence of pathology (that is, that the patient is healthy), and “1” may indicate the presence of pathology, or vice versa. In addition, for each image from the training data, the gender and age of the patient are indicated. This is either done manually by the physician, or read automatically from patient data stored in the medical information system 130 of the medical organization 110 or in the central medical information system 150, or from image metadata if it contains this information, etc. Manually adding marks can be done by analyzing the text of the image description by a doctor, on the basis of which a conclusion was drawn up about the presence/absence of pathology in the image. It is also possible to automatically recognize existing description texts for/on images, however, it should be understood that the recognition accuracy in the general case is not 100%, which reduces the degree of confidence in such markup and introduces potential errors into the learning process.

The training process of the neural network 300 is controlled by a training unit 280 by applying a training algorithm to the neural network being trained using the training data. The instructions (markings) from the doctor are used by the trained neural network as ground truth. For example, in one embodiment, the marking may be 3 numbers: "1" (sick), "1" (female), and "0" (a range number indicating that the image is from a patient aged 0-10 years). In another embodiment, the marking may represent other values: "healthy", "m" and "10-20" (years). It is preferable that the markup format matches the output format of the data in the above three subnets, so that there is no need to reduce the data to a single format to compare the prediction result with the true data. However, if there are many different sets of training data from different sources, then there may be differences in markup, and then reduction to a common format may be necessary.

In an illustrative embodiment, 70% of the images taken for training can be used directly for training, and 30% for testing the model. In another embodiment, a different ratio may be used that is more suitable for the purposes of training a particular neural network. For example, to train a neural network, images can be divided into training, validation and test samples. The training set size can be 70%, the validation set size can be 15%, and the test set size can be 15% of the images submitted for training. Specific ratios depend on the number of training images and are intended to test the generalized model - the extent to which the research results are applicable to new data is checked before using the model in operational mode in a medical organization.

During the training process, images are prepared in the same way as described above in relation to block 240, and are fed to the input of the neural network, while images with both the presence and absence of pathologies are used. At each training step, the neural network makes predictions for one or more images. These predictions are compared with the indication of the true presence/absence of pathology, as well as with the indication of the true gender and age of the patient, and the value of the loss function is calculated (how much the neural network made a mistake in identifying the presence of pathology, in determining the gender and age of the patient). Next, using the gradient descent method and the backpropagation algorithm, all weights (weighting coefficients) of the neural network are changed in accordance with the selected learning rate parameter in the direction opposite to the calculated gradient in order to minimize the error in the current image(s). I). This step is repeated many times, and as a result of the training process, the weights of the neural network converge to the optimal ones. These weights are subsequently used by the network during device operation to identify pathologies in input images that have not been marked.

In particular, as stated above, during the main operation of the device, only the pathology prediction is used, that is, the preliminary neural network section 310 and the first subnetwork 360, which predicts the probability of the presence of pathology in the analyzed image.

In one embodiment, the output data of the pathology prediction block 250, that is, an indication of the probability of the presence of a pathology, is the final result of the operation of the device 200 and is further used by other devices - for example, it is stored in the data storage block 220, from where it can later be retrieved on request from other devices , and/or are sent to the data transmission unit 270, which transmits this result to another device - for example, to the central medical information system 150 or to the medical organization 110 that requested data processing.

In another embodiment, the output of the pathology prediction block 250 is also transmitted to the report generation block 260, which generates a report of the examination performed on the device 200. The examination report may contain at least one of the following: a) a pathology imaging image; b) textual study protocol.

A pathology image can be generated for those images that have been analyzed to have a pathology probability greater than a predetermined pathology threshold, indicating that the radiologist should pay attention to the image. The threshold probability of the presence of pathology in the general case is 0.5 (or 50%), but depending on the accuracy indicators of a particular trained model and the degree of confidence in it, it can vary in practical application, either down or up. For example, a medical organization may specialize in the treatment of diseases, the early detection of which is critical in terms of prognosis of treatment, and therefore may request visualizations if the device has detected a probability of the presence of pathology of only 10% or more. In another example, a medical organization may be affiliated with an enterprise, and the majority of its patients may have occupational chronic chest diseases, and some changes in the images of such patients may be considered acceptable, so imaging may be requested only in cases where the device has identified the possibility of the presence pathology 60% or more. It should be noted that many other situations are possible that are not listed herein but are within the scope of this invention. The ability to select the threshold probability of the presence of pathology allows you to adapt the results of the proposed device 200 to various specific applications, for example, to reduce the burden on medical personnel and at the same time ensure early detection of pathologies with the necessary accuracy.

An image with pathology visualization is a copy of the analyzed image, on which an indication of the found signs of pathologies is superimposed. For example, if the DICOM format is used, it can be generated as a DICOM Secondary Capture.

Various visualization options are possible. In one embodiment, the visualization can be performed in the form of a heat map, which provides a more intensive selection of those points and/or areas that are of greater interest from the point of view of the presence of pathologies, a less intensive selection of those points and/or areas that are of less interest , and those areas that presumably do not contain pathology are not highlighted. The principle of selection can be different - for example, selection can be done using a gradient of the same color (for example, shades of red or orange), using a gradient transition from one color to another color (for example, from yellow to red), using precise specifying the color (for example, blue - lower probability, blue - higher probability), using the size of the dot (for example, the lower the probability, the smaller the dot), etc. Various methods for constructing heat maps are applicable - for example, a combination of two methods shows improved performance: Grad-CAM (a weighted combination of gradient-based class activation maps) and Saliency maps (saliency maps). In particular, each of these methods separately analyzes the input image of the neural network 300 and the activations (numbers) within the neural network 300 that were obtained by running this image through it, after which the results of the methods are combined and a common heat map is constructed, an example of which shown in Fig. 10.

Heat maps allow you to adjust the amount of attention a healthcare professional should pay when examining a particular area in an image, eliminating the need for detailed visual inspection of the entire image. This ensures a reduction in the burden on medical personnel.

However, there may be situations where the heat map is not accurate enough, in which case the doctor, on the contrary, has to spend time studying an area that is not of interest. It also turned out that in some implementations, heat maps clutter the image, as a result of which the specialist has to spend time viewing two images simultaneously: both the original image and the image with the heat map applied to it.

Accordingly, in another embodiment of the present invention, the visualization may be in the form of delineating boundaries (or outlines) enclosing one or more areas in the image that are likely to contain pathology, or in other words, that pathology prediction unit 250 has detected evidence of pathology. To implement this approach, various computer vision methods can be used to analyze the state of the neural network of the pathology prediction block 250. For example, as mentioned above, a combination of two methods shows improved performance: the Grad-CAM method (a weighted combination of gradient-based class activation maps) and Saliency maps (saliency maps). A predefined threshold heat value is applied to the heat map constructed on their basis, according to which the contour is drawn (for example, 0.25, 0.3, etc.). In this way, closed loops are formed that enclose regions with heat equal to or greater than the threshold, while regions with heat below the threshold are discarded. The heat maps themselves are not applied to the image; only the contours are visualized on it, as shown in the example in Fig. eleven.

An image with the outlines of the boundaries of possible pathologies marked on it, as before, makes it possible to reduce the burden on medical personnel, while it is enough to study only it, without referring to the original image. In addition, this approach allows us to smooth out inaccuracies in defining areas of interest.

In one embodiment, an indication of the found signs of pathologies in the form of a heat map or boundaries of pathologies can be superimposed on a copy of the image that was directly analyzed by the neural network, that is, on an image that, compared to the original X-ray image obtained in the data to be processed, has been compressed, cropped and reduction to a single size. This simplifies the process of visualizing the pathology, but in this case the resulting image is different from the original one, so it may cause some discomfort for the radiologist comparing the two images.

In another embodiment, an indication of the detected pathology features in the form of a heat map or pathology boundaries may be superimposed on the original x-ray image obtained in the data to be processed. To implement this approach, the parameters of all transformations performed on the original image and associated with resizing, shifting, rotating, cropping and cropping are stored by the image preparation unit 240 - for example, in the storage unit 220. After developing methods for searching for signs of pathologies, a visualization of the pathology is obtained in the form of a heat map or the boundaries of pathologies with reference to the size, shape and position of the image, which was directly analyzed by the neural network. The inverse transformations of the above stored transformations are applied to the resulting visualization to produce a pathology visualization that is referenced to the size, shape, and position of the original x-ray image. This rendering is then superimposed on a copy of the original image. This somewhat complicates the processing, but allows you to reduce the time for image analysis by a radiologist and increase the accuracy of identifying pathologies. This is especially applicable to heat maps, where the comparison of the original image and the pathology image is in many ways an urgent need for the radiologist.

If necessary, the image with visualization of the pathology may contain in text form an indication of the probability of the presence of pathology, obtained as a result of the operation of the first neural network of the pathology prediction block 250. This reduces the need to study any other images or files, including the original image and/or file with a text study protocol.

Turning now to the textual study protocol, it can be noted that it can either complement the above-described pathology image or replace it. The text research protocol contains the results of the work and a description of the operation of the device 200 in the form of text information, for example, in CSV format. If the DICOM format is used, the protocol can be generated in the form of a DICOM Structured Report.

In particular, the textual research protocol may contain:

1) The name of the source image (a link to it in the form of a corresponding identifier - for example, StudyInstanceID).

2) Recommendation (or in other words, prediction). As stated above, a given recommendation may have a binary value (for example, 0 / 1), may be a number in some range indicating the probability of having the pathology (for example, from 0 to 1), or may have a text format (for example, “probably sick " / "probably healthy").

3) Time stamps – time of receiving the study, time of generating the report.

4) Information about device operation errors during the analysis, if any.

The study report obtained as a result of the operation of block 260, containing an image with pathology visualization and/or a text study protocol, is stored in data storage block 220, from where it can later be retrieved upon request from other devices, and/or sent to data transmission block 270, which transmits this report to another device - for example, to a central medical information system 150 or to a medical organization 110 that requested data processing.

In one embodiment, the processing results obtained by the device 200 may optionally be stored in the device for a predetermined time (eg, seven days) and then deleted. This may be done, for example, for backup purposes so that the analysis result can be queried and retransmitted without the need for a full re-analysis.

Thus, a pathology detection device is provided that is capable of automatically determining the likelihood of a pathology with increased accuracy. This allows us to reduce the requirements for the qualifications of medical personnel and reduce the influence of the human factor (attention, fatigue, responsibility). It also provides the ability to use a large number of disparate data sets for training. In addition, the result of the study provides a comprehensive set of information necessary for making the correct medical decision with increased speed and accuracy.

In an illustrative non-limiting example for the purposes of the present invention, the pathology detection device 200 may be implemented as a cloud server having 24 computing cores, 64 GB RAM, 300 GB disk space for the CentOS 7 operating system, 12 TB disk space for data, 1 disk space, 5 TB for database and 2 network adapters.

Method for determining pathology

Next, with reference to Fig. 12, a pathology detection method 400 will be described.

It should be understood that the steps of method 400 correspond to the above-described functions performed by each of the blocks in the device 200, and if any information is not disclosed in connection with the method, but is disclosed in connection with the device, and vice versa, then it is not implied that in the device or method this function or step cannot be performed, but is done only to avoid cluttering the description with repetition of details. It should also be understood that depending on the specific application, some of the described method steps may not be used.

The pathology determination method 400 may comprise step S410 of receiving data to be processed comprising a medical image of a patient being examined, such as an AP chest X-ray image.

The pathology determination method 400 may also include a step S430 in which validation is performed by checking whether the data to be processed is suitable for processing.

The pathology determination method 400 may also comprise a step S440 of preparing an image contained in the data to be processed for use by the neural network by performing one or more preliminary transformations on it.

The pathology detection method 400 comprises step S450, in which the prepared image is analyzed using a pre-trained neural network and the presence or absence of organ pathology in the image is determined.

The pathology determination method 400 may also comprise a step S451 where, if a pathology is present, the most likely boundaries within which it is located are determined using a pre-trained neural network.

The pathology determination method 400 may also comprise step S460, which generates an examination report comprising at least one of the following: a) a pathology imaging image; b) text study protocol.

The pathology determination method 400 may also comprise a step S470 of transmitting the processing result obtained in steps S450, S451 and S460 to the device requesting the data processing.

The pathology detection method 400 may also include a step S480 (not shown) in which the neural network is pre-trained using pre-labeled training images, wherein the label for each image includes an indication of the presence or absence of pathology of the organs in the area of the image for which it is desired to perform training a neural network, indicating the gender and indicating the age of the patient to whom this image belongs.

The pathology determination method 400 may also include a step S490 (not shown), in which, using the processing result obtained in steps S450, S451 and S460, a medical decision is made about the presence or absence of pathology of the organs in the image area and, if necessary, the nature of the pathology.

The pathology determination method 400 may also comprise a step S491 (not shown) in which a patient is prescribed and/or treated based on the processing result obtained in steps S450, S451 and S460 and/or the medical decision made in step S490.

The pathology determination method 400 may also include a step S420 (not shown) in which data received by the data receiving unit 210 and/or obtained as a result of or during execution of other method steps is stored.

The proposed method for determining pathology provides increased accuracy of automatic determination of the probability of pathology, reduces the requirements for the qualifications of medical personnel and reduces the influence of the human factor (attention, fatigue, responsibility). In addition, the result of the study provides a comprehensive set of information necessary for making the correct medical decision with increased speed and accuracy.

Application

The devices and methods of the present invention can be used to process diagnostic medical images of organs and/or cavities to identify signs of pathologies therein.

Additional implementation features

Although this document may indicate that data is being transmitted/sent or received/received by a human (e.g., medical professional, doctor, expert), one skilled in the art should understand that such indication is used solely for the purpose of simplifying the description , when in fact it is implied that the data is transmitted/sent or received/received by the corresponding device used and/or controlled by that person.

One or more transmission blocks or devices described herein (transmitters) and one or more reception blocks or devices (receivers) may be physically implemented in the same block or device or in different blocks or devices.

A transmission device or unit in this document, for the sake of simplicity of description, may be referred to as a device or unit having the functions of not only transmitting, but also receiving data, information and/or signals. Likewise, the receiving device or unit may also include functions for transmitting data, information and/or signals.

The various illustrative blocks and modules described in connection with the disclosure herein may be implemented or executed by a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable logic device (PLD), discrete logic gate or transistor logic, discrete hardware components, or any combination of the foregoing, designed to perform the functions described herein. The general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (eg, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors plus a DSP core, or any other similar configuration).

Some of the units, individually or collectively, may constitute, for example, a computer, and include a processor that is configured to call and execute computer programs from memory to perform method steps or functions of the units in accordance with embodiments of the present invention. According to embodiments, the device may further include memory. The processor may call and execute computer programs from memory to perform the method. Memory may be a separate device, independent of the processor, or may be integrated into the processor. The memory may store code, instructions, commands and/or data for execution on a set of one or more processors of the described device. Codes, instructions, or commands may direct a processor to perform steps of a method or device function.

The functions described herein may be implemented in hardware, software executed by one or more processors, firmware, or any combination thereof. The hardware and software that implement the functions may also be physically located in different locations, including according to a distribution such that portions of the functions are implemented in different physical locations, that is, distributed processing or distributed computing may be performed.

If necessary (for example, if the volume of data and/or calculations that need to be performed on this data is large), multi-threaded data processing can be performed, which in a simple representation can be expressed in the fact that the entire set of data to be processed is divided into a set of subsets , and each processor core performs processing on its assigned subset of data.

The above memory may be volatile or non-volatile memory, or may include both volatile and non-volatile memory. One skilled in the art will also understand that when talking about memory and the storage of data, programs, codes, instructions, commands, etc., it is meant to be a machine-readable (or computer-readable, processor-readable) storage medium. Computer readable media includes both non-transitory computer storage media and communication media, which includes any transmission medium that facilitates movement of a computer program or portion thereof from one location to another. A non-transitory computer-readable storage medium may be any available medium that can be used to carry or store desired program code media in the form of instructions or data structures, and that can be accessed by a computer, processor, or other general purpose or special purpose processing device. appointments.

By way of example, and not limitation, computer-readable media may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), flash memory , random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM) , Enhanced Speed Synchronous Dynamic Random Access Memory (ESDRAM), Synchronous Line DRAM (SLDRAM) and Direct Access Random Access Memory (DR RAM), register, cache memory, semiconductor memories, magnetic media such as internal hard drives and removable drives, magneto-optical media and optical media such as CD-ROMs and digital versatile disks (DVDs), as well as any other storage media known in the art.

The information and signals described herein may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols and chips that may be exemplified in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combinations of the above, as applicable to the present invention.

At least one of the steps in the method or blocks in the device may use an artificial intelligence (AI) model to perform corresponding operations. The AI-related function may be performed through a processor and non-volatile and/or volatile memory.

The processor may include one or more processors. At the same time, one or more processors may be a general purpose processor such as a central processing unit (CPU), an application processor (AP), etc., a graphics processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU) ), and/or a specialized AI processor such as a neural processing unit (NPU).

One or more processors control the processing of input data in accordance with a predefined operating rule or artificial intelligence (AI) model stored in non-volatile memory and/or volatile memory. A predefined operating rule or artificial intelligence model can be obtained through training. In this case, the processor can perform a pre-processing operation to convert the data into a form suitable for use as input to an artificial intelligence model.

“Learned” means that by applying a learning algorithm to a trained AI model using a set of training data, a predefined operating rule or AI model with the desired characteristic is created. The training may be performed on the device itself on which the AI is running according to the embodiment, and/or may be implemented through a separate server/system.

An artificial intelligence model may include many layers of neural network. Each of the multiple layers of the neural network includes a plurality of weights (coefficients) and performs an operation for that layer by computing using the plurality of weights of that layer with respect to the input data or computation result of the previous layer.

Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN) ), generative adversarial networks (GANs), and deep Q-networks.

A learning algorithm is a method of training a predefined target device (such as a GPU or NPU based neural network) using a set of training data to invoke, enable, or control the target device to perform a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

It should be understood that while terms such as “first,” “second,” “third,” and the like may be used herein to describe various blocks, modules, networks, elements, components, regions, layers, and/or sections. ., these blocks, modules, networks, elements, components, areas, layers and/or sections should not be limited to these terms. These terms are used only to distinguish one block, module, network, element, component, area, layer or section from another block, module, network, element, component, area, layer or section. Thus, a first block, module, network, element, component, region, layer or section may be referred to as a second block, module, network, element, component, region, layer or section without departing from the scope of the present invention. As used herein, the term “and/or” includes any and all combinations of one or more of the respective items listed. Elements referred to in the singular do not exclude the plurality of elements unless specifically stated otherwise.

The functionality of an element specified in the description or claims as a single element may be implemented in practice by means of several components of the device, and conversely, the functionality of elements specified in the description or claims as several separate elements may be realized in practice by means of a single component.

In one embodiment, some or all elements/blocks/modules of the proposed device are located in a common housing, can be placed on one frame/structure/printed circuit board/chip and are connected to each other structurally through installation (assembly) operations and functionally through communication lines . The mentioned lines or communication channels, unless otherwise indicated, are typical communication lines known to specialists, the material implementation of which does not require creative efforts. The communication line can be a wire, a set of wires, a bus, a track, a wireless communication line (inductive, radio frequency, infrared, ultrasonic, etc.). Communication protocols over communication lines are known to those skilled in the art and are not disclosed separately.

The functional connection of elements should be understood as a connection that ensures the correct interaction of these elements with each other and the implementation of one or another functionality of the elements. Particular examples of functional communication may be communication with the ability to exchange information, communication with the ability to transmit electric current, communication with the ability to transmit mechanical motion, communication with the ability to transmit light, sound, electromagnetic or mechanical vibrations, etc. The specific type of functional connection is determined by the nature of the interaction of the mentioned elements, and, unless otherwise indicated, is provided by widely known means, using principles widely known in the art.

The design of the elements of the proposed device is known to those skilled in the art and is not described separately in this document unless otherwise indicated. The elements of the device can be made of any suitable material. These component parts can be manufactured using known methods including, by way of example only, machining, investment casting, and crystal growth. The assembly, connection and other operations in accordance with the above description are also within the knowledge of a person skilled in the art and, therefore, will not be explained in more detail here.

Although exemplary embodiments have been described in detail and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative and are not intended to limit the present invention, and that the invention should not be limited to the specific arrangements and structures shown and described, since Various other modifications and embodiments of the invention may be apparent to one skilled in the art based on the information set forth in the specification and knowledge of the prior art without departing from the spirit and scope of the present invention.

Claims (54)

1. A device for analyzing medical images, containing: a data receiving unit configured to receive data containing a medical image to be processed from a device requesting data processing by communicating with it; an image preparation unit configured to prepare, using at least one processor, a medical image contained in the data to be processed by performing one or more preliminary transformations on it for use by a neural network operating on the at least one processor; a pathology prediction block configured to analyze the prepared image using said neural network and determine the presence or absence of pathology in the image; a report generation unit configured to generate, using at least one processor, a report on a study conducted in this device, wherein the study report contains at least one file containing an indication of the result of the processing performed in the pathology prediction block; And a data transmission unit configured to transmit a research report to a device that requested data processing by communicating with it, wherein said neural network is pre-trained using at least one data set containing labeled medical images to predict the presence or absence of pathology, as well as the gender and age of the patient, wherein the labeled medical images contained in the at least one dataset used to train the neural network contain: images having a first label associated with an explicit indication of the male gender of the patient, images having a second label associated with an explicit indication of the patient's female gender, and images having a third label associated with an implicit indication of the patient's gender, wherein the third label is assigned to images that initially did not have a label indicating the patient's gender, and images that initially had a label in which the patient's gender was specified as data other than the explicit one indications of male or female, wherein said neural network is pre-trained to predict the expected gender of the patient in the form of an indication of the first mark or the second mark, if the neural network was able to explicitly determine the male or female gender of the patient, or in the form of an indication of the third mark, if the neural network was unable to clearly determine the gender of the patient, wherein said neural network is configured to analyze during operation only the prepared image without additional input data in the form of data on the gender and age of the patient depicted in this image. 2. The device according to claim 1, in which said processing result is intended to support medical decision-making about the condition of at least one organ and/or cavity of the examined patient recorded in said image and/or about the treatment of the examined patient. 3. The device according to claim 1, in which said neural network contains: - a preliminary section containing a first neural network, in which each layer is sequentially connected to each subsequent layer, and the preliminary section is configured to receive said prepared image and output intermediate data; - a first subnetwork containing a second neural network and configured to receive said intermediate data from the preliminary section and predict the probability of the presence of pathology in the analyzed image; - a second subnetwork containing a third neural network and configured to receive said intermediate data from the preliminary site and predict the expected gender of the patient; - a third subnetwork containing a fourth neural network and configured to receive said intermediate data from the preliminary section and predict the expected age of the patient. 4. The device according to claim 3, in which the preliminary section is based on the DenseNet architecture. 5. The apparatus of claim 1, wherein said neural network is pre-trained to predict the expected age of the patient as an indication of one of the predefined age ranges. 6. The apparatus of claim 5, wherein the size of each age range is from 7 to 15 years. 7. The device according to claim 5, wherein the number of preset age ranges is 10, the size of each range is 10 years. 8. The device according to claim 1, in which said neural network is pre-trained to predict the presence or absence of at least one sign from a predetermined set of possible pathologies characteristic of at least one organ and/or cavity recorded in said image or for a given type diagnostic study. 9. The device of claim 8, wherein the predetermined plurality of possible pathologies for the chest x-ray comprises at least one of the following pathologies: pleural effusion, pneumothorax, atelectasis, focus of opacification, infiltration/consolidation, dissemination, cavity with collapse, cavity with fluid level, calcification/calcified shadow in the lungs, violation of the integrity of the cortical layer. 10. The device according to claim 1, in which, if it is determined that there is a pathology in the image, the report generation unit is additionally configured to: identifying possible areas in which signs of pathologies were found, using computer vision methods that analyze the original image and activation within the neural network of the pathology prediction block, which were obtained by running this image through it; generating an image with visualization of pathology in the form of a copy of the original image, on which an indication of the found areas with signs of pathologies is superimposed; And generating a research report containing an image with pathology visualization. 11. Device according to claim 10, in which: to identify possible areas with signs of pathologies, a combination of Grad-CAM (weighted combination of gradient-based class activation maps) and Saliency maps (saliency maps) methods is used, and a heat map or outline of the boundaries of the areas is used to indicate the found areas with signs of pathologies. 12. The device according to claim 1, in which the study report contains a text study protocol containing in text form at least one of the following: - source image identifier; - indication of the probability of the presence of pathology; - image acquisition time; - time of receiving the processing result from the pathology prediction block; - time of report generation; - information about device operation errors that occurred during the analysis. 13. A method for analyzing medical images, containing the steps of: using a communication device, receiving data to be processed containing a medical image of the patient being examined; preparing, using at least one processor, a medical image contained in the data to be processed by performing one or more preliminary transformations on it for use by a neural network operating on the at least one processor; using said neural network, the prepared image is analyzed and the presence or absence of pathology in the image is determined; wherein said neural network is pre-trained using at least one data set containing labeled medical images to predict the presence or absence of pathology, as well as the gender and age of the patient, wherein the labeled medical images contained in the at least one dataset used to train the neural network contain: images having a first label associated with an explicit indication of the male gender of the patient, images having a second label associated with an explicit indication of the patient's female gender, and images having a third label associated with an implicit indication of the patient's gender, wherein the third label is assigned to images that initially did not have a label indicating the patient's gender, and images that initially had a label in which the patient's gender was specified as data other than the explicit one indications of male or female, wherein said neural network is pre-trained to predict the expected gender of the patient in the form of an indication of the first mark or the second mark, if the neural network was able to explicitly determine the male or female gender of the patient, or in the form of an indication of the third mark, if the neural network was unable to clearly determine the gender of the patient, Moreover, during the operation of the said neural network, only the prepared image is analyzed without additional input data in the form of data on the gender and age of the patient depicted in this image, generating, using at least one processor, a report that contains at least one file containing an indication of the presence or absence of pathology; transmitting a report to the doctor’s device using a communication device, receive a report on the doctor’s device using a communication device, using the doctor's device and using the report, a doctor's decision is made on the condition of at least one organ and/or cavity of the examined patient recorded in said image and/or on the treatment of the examined patient.