RU2845119C2 - System and method for determining pulmonary pathologies from x-ray images - Google Patents

Abstract

FIELD: image processing.

SUBSTANCE: group of inventions relates to image processing and, more particularly, to supporting medical decision-making based on medical image analysis. Disclosed is a system for implementing the method for determining lung pathologies, which comprises a unit for obtaining an X-ray image of the lungs; a pathology diagnosis unit which, using a neural network based on segmentation, detects areas of pulmonary pathologies in the obtained image; anatomical structures segmentation unit which determines using the neural network based on the segmentation of the heart region, the right lung region and the left lung region along their outer contours, as well as the region from the diaphragm and below it; pathology localization unit, which determines the detected pathology sites localization by comparing the output data from the pathology diagnostic unit and output data from the anatomical structures segmentation unit and removes the suspected areas of pathologies extending beyond the area to which they correspond; and a unit for transmitting a report to the device requesting the diagnosis, wherein the pathology diagnosis unit is further configured to perform post-processing of segmentation masks output by the neural network it comprises.

EFFECT: group of inventions provides higher accuracy of diagnostics and higher automation of processing.

7 cl, 1 dwg

Description

Field of technology to which the invention relates

The present invention relates to the field of image processing, and more particularly to supporting medical decision making based on the analysis of medical images.

State of the art

X-ray examinations of the chest organs account for 24% or more than 76 million examinations per year in the structure of all radiation examinations in the Russian Federation. Currently, the main method of active detection of latent pathological processes in the lungs are fluorographic examinations.

Algorithms and software that enable automated analysis of radiological data are being widely tested. The use of such algorithms suggests more efficient detection of pathological changes.

The use of artificial intelligence (AI) in various fields of medicine has grown significantly in recent years, largely due to advances in high-performance computing and the development of deep learning (DL), a subset of AI algorithms that use neural networks to learn how to solve specific problems from databases labeled by experts.

Early methods of pulmonary disease diagnosis using AI used the following methods: combining a neural network classifier with a pathology segmentation network, combining a neural network pathology segmentation network with a multi-class detector, multi-class segmentation with thresholding, and multi-class segmentation with two thresholds. All these methods have limited accuracy, increased requirements for the amount of training data, increased processing complexity, and limited operating speed. In addition, the prior art does not provide effective solutions to overcome the problem of small class sizes and class imbalance.

The essence of the invention

In order to eliminate at least some of the above-mentioned disadvantages of the prior art, the present invention is aimed at improving the efficiency of systems and methods for diagnosing lung pathologies using artificial intelligence. In the modern world, where technology plays a key role in improving the quality of medical care, the presented invention allows improving the accuracy parameters of diagnosing lung diseases, the speed of image processing by automatic lung diagnostic systems and reducing the requirements for the amount of training data for AI systems.

According to a first aspect of the present invention, a system for determining lung pathologies from X-ray images is proposed, comprising:

an imaging unit configured to obtain a two-dimensional X-ray image of a region of the lungs;

a pathology diagnostic unit, configured to detect areas of lung pathologies on the obtained image using at least one processor and memory using a neural network based on segmentation;

a segmentation unit for anatomical structures, configured to determine, using at least one processor and memory, using a neural network based on segmentation, the region of the heart, the region of the right lung and the region of the left lung along their outer contours, as well as the region from the diaphragm and below it;

a pathology localization unit configured to, using at least one processor and memory, determine the localization of the identified pathology areas by comparing the output data from the pathology diagnostic unit and the output data from the anatomical structure segmentation unit, and remove the suspected pathology areas that extend beyond the area to which they correspond; and

a report transmission unit configured to generate a report containing information on the presence or absence of lung pathologies in the received image based on output data from at least one of the pathology diagnostic unit, the anatomical structure segmentation unit, and the pathology localization unit,

wherein the pathology diagnostics unit is additionally configured with the ability to perform post-processing of segmentation masks output by the neural network contained therein, the post-processing consists of preliminary binarization of the mask of each class, selection of contours, representation of each contour as a separate detection object, calculation of the average probability of pixels of the initial mask within each contour with subsequent comparison of this probability with the detection threshold calculated on the basis of the AUROC curve for the entire training sample.

In one embodiment, the report transmission unit is additionally configured to form a copy of the input image with contours applied to it, limiting the area of pathology, and also to indicate text information about the localization of the pathology and numerical values of the area of lung damage.

In one embodiment, the pathology localization unit is additionally configured to make a conclusion about the presence or absence of disseminated damage depending on the localization of the pathology in one or two lungs, the number of lesions, and the area of damage to each lung.

In one embodiment, the pathology localization unit is additionally configured to calculate the transverse size of the lungs and heart, calculate the cardiothoracic index (CTI) on their basis, and reset the probability of the presence of cardiomegaly determined by the pathology diagnostic unit if the calculated CTI is less than the critical threshold.

In one embodiment, each neural network is pre-trained to determine a mask for each class corresponding to it, which contains the probabilities of each pixel belonging to this class, wherein during training a focal loss function and pixel weights are used, which represent weights for positive pixels of the true class mask and negative pixels of the true class mask, based on the frequency of pixels belonging to a specific class in the entire sample, wherein the focal loss function is calculated for each pixel of the mask after determining the class mask, and if the corresponding pixel of the true class mask is positive, the error obtained on this pixel of the class mask is multiplied by the weight of the pixel of a positive example of this class, and if the corresponding pixel of the true class mask is negative, the error obtained on this pixel of the class mask is multiplied by the weight of the pixel of a negative example of this class.

In one embodiment, after each training epoch, the ratio between the average precision and average recall metrics is calculated, and if it is outside the range from 0.9 to 1.1, then the pixel weights for the focal loss function are adjusted as follows: the pixel weight for a positive pixel is multiplied by the result of dividing the average F1 metric by the average recall, for a negative pixel - by the result of dividing the average F1 metric by the average precision.

In one embodiment, each neural network is pre-trained to determine a mask for each class corresponding to it, which contains the probabilities of each pixel belonging to this class, wherein during training a loss function is used based on the Jaccard coefficient and the class weights of the positive and negative class labels, which are determined based on the inverse frequency of positive and negative class labels for all images in the sample for each class, wherein if for a given image in the true mask of a given class there are positive elements, then the entire instance is assigned a positive class label, if absent - a negative class label, wherein the loss function based on the Jaccard coefficient is calculated for each class mask after determining the class mask for each class, and if the true class label corresponding to the image is positive, then the error of the corresponding class is multiplied by the class weight of the positive instance of the class, and if negative - by the class weight of the negative instance of the class, respectively.

According to a second aspect of the present invention, a method for determining lung pathologies from X-ray images is provided, comprising the steps of:

Using an imaging unit, a two-dimensional X-ray image is obtained containing an area of the lungs;

using a pathology diagnostics unit containing a neural network for segmentation, areas of lung pathologies are determined on the obtained image, while post-processing of the segmentation masks output by the neural network is performed, post-processing consists of preliminary binarization of the mask of each class, selection of contours, presentation of each contour as a separate detection object, calculation of the average probability of pixels of the original mask within each contour with subsequent comparison of this probability with the detection threshold calculated on the basis of the AUROC curve for the entire training sample;

using a block for segmentation of anatomical structures containing a neural network for segmentation, the region of the heart, the region of the right lung and the region of the left lung are determined on the resulting image along their outer contours, as well as the region from the diaphragm and below it;

using the pathology localization unit, determine the localization of the identified pathology areas based on a comparison of the output data from the pathology diagnostics unit and the output data from the anatomical structure segmentation unit and remove the suspected pathology areas that extend beyond the area to which they correspond; and

using a report transmission unit, a report is generated and transmitted containing information about the presence or absence of lung pathologies in the obtained image based on output data from at least one of the pathology diagnostic unit, the anatomical structure segmentation unit and the pathology localization unit, to the device that requested the diagnostics.

Technical result

The present invention makes it possible to increase the efficiency of systems and methods for diagnosing lung pathologies using two-dimensional X-ray images. This ensures:

- increasing the accuracy of diagnostics;

- increasing the speed of processing two-dimensional images;

- simplification of interpretation and description of images;

- the ability to perform diagnostics of lung diseases in screening mode;

- freeing medical personnel from routine processes of analyzing patient images;

- reducing the influence of the human factor (attentiveness, fatigue, responsibility);

- increasing the ratio between the number of images used for training and the obtained classification and detection metrics.

These and other advantages of the present invention will become apparent from reading the following detailed description with reference to the accompanying drawings.

Brief description of the drawings

Fig. 1 shows a block diagram of a system for diagnosing lung pathologies according to the present invention.

It should be understood that the figures may be diagrammatic and not to scale and are intended primarily to facilitate an understanding of the present invention.

Detailed description

In practice, lung diseases are usually diagnosed by doctors by manual analysis of images. The present invention provides a method and a system that automate the diagnostic process, determining the presence of signs of pathologies from the list, their size and location, heart size, cardiothoracic index.

The present authors collected an anonymous database of patients with and without signs of lung pathologies. Each patient case consists of a lung image in a direct projection. On each X-ray image, a qualified radiologist manually outlined the area of each lung, the area from the domes of the diaphragm and below, the heart area, and each of the signs of pathologies, if present, with a named contour. As will be shown below, such a database will be used to train machine learning models.

The prior art discloses systems in which networks of the detector and classifier type were used to detect lung pathologies, with subsequent segmentation based on a separate segmenter network in a multi-class version or an ensemble of segmenters in a mono-class version or based on classifier attention maps. Examples of networks combining segmentation of the target lung region are also known, with subsequent transfer of the target region to the classifier with the exclusion of regions that are not related to the lungs. The prior art discloses systems in which pathology was segmented as a single class with subsequent classification of the selected regions. The prior art discloses systems in which detection and classification were performed by a segmenter network based on a segmentation threshold or two thresholds, the first of which served to determine the presence or absence of pathology, and the second to determine the optimal segmentation region.

Of the presented approaches, the automatic lung pathology diagnostics systems based on classifiers, classifiers with preliminary segmentation and target area allocation, and detector networks require a significant number of images for training. Automatic lung pathology diagnostics systems based on segmentation networks are prone to false positives due to network activation in individual small areas and require additional filtering by the size of the detected potential pathology areas. In turn, during the training process, previously trained segmentation networks face the problem of both class imbalance and class representation imbalance in the image, which leads to deterioration of the results when training using the error function (loss function) based on crossentropy, especially for such pathological signs as calcifications, focal shadows, focal fibrosis, rib fractures.

The present invention effectively overcomes this obstacle by using segmentation-based detection with the training and post-processing algorithm described in the present invention, eliminating false positive results based on an analysis of the intersection of masks of normal anatomy and pathology, and increasing accuracy, including through the assessment of the cardiothoracic index.

General description of the system

As shown in the block diagram in Fig. 1, the system 100 for diagnosing lung pathologies according to the present invention comprises an image acquisition unit 110, an anatomical structure segmentation unit 120, a pathology diagnosis unit 130, a pathology localization unit 140, and a report transmission unit 150.

In one embodiment, the image acquisition unit may be a network card or other data receiving means. In another embodiment, the image acquisition unit may be the X-ray apparatus itself.

The pathology diagnostics unit, the pathology localization unit, and the anatomical structure segmentation unit are implemented on the basis of a processing device containing a computing module (e.g., a central processor), a memory module (e.g., RAM and ROM), and, if necessary, an additional computing module (e.g., a graphic processor or a neural processor). Thus, the pathology diagnostics unit, the pathology localization unit, and the anatomical structure segmentation unit can be implemented on the basis of a computer or a server.

The report transmission unit may be a network card, a hardware data/video signal transmission interface, or other data exchange means.

The method of diagnosing lung pathologies is performed by the pathology diagnostic system and is as follows.

First, a two-dimensional image of the lungs is obtained using an imaging unit.

As indicated above, in one embodiment, if the image acquisition unit is an X-ray machine, it can acquire an image independently (or under the control of an operator). In another embodiment, the image acquisition unit, being part of a pathology diagnostic service, can receive from outside (e.g., from a medical organization, from a doctor, from a regional medical data storage system, etc.) a diagnostic request or input data containing one or a plurality (e.g., a batch) of images, and extract the desired image from the received input data or request.

If necessary, the resulting image is pre-processed in a pre-processing block (not shown in Fig. 1) so that it is suitable for processing by further blocks. For example, if the original image has insufficient or excessive resolution in any direction, then the image can be transformed or reconstructed to the required resolution. Brightness, contrast, and other image parameters can also be adjusted.

The image is then fed to the anatomical structure segmentation unit and the pathology diagnostic unit.

In the anatomical structure segmentation block, the probability of class presence and binary segmentation masks for each of the following classes are determined using a pre-trained neural network: "Right lung", "Left lung", "Domes of the diaphragm and underlying region", "Heart" (as shown above, the training dataset contains markings for each such class). The output of the anatomical structure segmentation block is a binary mask of the right lung, left lung, heart, and domes of the diaphragm.

In the pathology diagnostics block, binary pathology masks and probabilities of pathology class presence are generated for each lung pathology class using a pre-trained neural network. The pathology classes are the following classes: focal shadow, infiltration, pleural effusion, atelectasis, pneumothorax, calcifications, fibrosis, rib fracture, consolidated rib fracture, pulmonary edema, and cardiac dilation (as shown above, the training dataset contains named marking of areas with pathology in those images where pathologies are present). The output of the pathology diagnostics block is binary pathology masks and a set of values from 0 to 1 for each pathology, interpreted as the probability of its presence.

The output data from the anatomical structure segmentation unit and the pathology diagnostics unit are sent to the pathology localization unit. In the pathology localization unit, image areas marked as containing pathology are checked for intersection with image areas selected as corresponding to the right lung, left lung, or heart to exclude false positive results. Pathology masks outside the lung area are reset to zero, and heart pathology masks outside the heart mask are reset to zero. After that, the transverse size of the lungs (chest) and heart is calculated (e.g., by the number of pixels that make up the corresponding transverse line) and the cardiothoracic index (CTI) is calculated, which is the ratio of the heart shadow diameter measured at the most distant points of the right and left heart contours to the internal chest diameter measured above the domes of the diaphragm during direct radiography.

In one implementation, to eliminate false negative results in the class "Cardiomegaly", the probability of the class presence is reset to zero if the calculated CTI is less than the critical threshold of 0.6.

In one implementation, the area of the pathology "Infiltration" and "Focal shadow" is calculated in comparison with the area of the lungs. If the infiltration or focal shadows occupy more than a critical threshold of area in each of the lungs, a conclusion is made about the presence of dissemination.

In one implementation, isolated contours of the "Focal shadow" class are counted. If the number of shadows exceeds the threshold, a conclusion is made about the presence of disseminated lung damage.

In fact, both the anatomical structure segmentation block and the pathology diagnostics block first perform segmentation, and then, based on this segmentation, form a binary mask and calculate probabilities. The difference is that in the anatomical structure segmentation block, segmentation is intended to classify anatomical structures, while in the pathology diagnostics block, segmentation is intended to detect pathologies. The data from these blocks are then combined in the pathology localization block to improve the diagnostic accuracy and eliminate false positives that could occur if only one of the two approaches using segmentation were used, rather than the proposed combination. This compares favorably with known solutions from the state of the art, which either do not use segmentation at all or use segmentation only to detect anatomical structures.

Next, the output from the pathology localization unit is transmitted to the report transmission unit. In one embodiment, a report on the state of the lungs can be generated for each image. In another embodiment, a diagnostic request or diagnostic system settings can provide for the need to generate a report only for cases in which pathology is detected. For example, if the system has detected one case of pathology in the received batch of 10 images, it can generate a report for the corresponding image (or patient) containing the calculated clinical indicators. The report can also contain a copy of the input image generated by the report transmission unit with pathology contours applied to it. The report can also contain information in text form on the localization of each sign in one of the two lungs, the CTI, and the area of damage to each lung.

The report on the health status of the lungs is then displayed on the monitor screen connected to the server, or transmitted via a report transmission unit to the device that requested the diagnostics – for example, to a remotely located automated workstation (AWS) of the doctor.

Using the image with the contours applied to it and other data contained in the report, the doctor makes a decision about the patient's health condition.

Detection and classification based on segmentation masks

Below, the process performed in the blocks of segmentation of anatomical structures and diagnostics of pathologies will be described in detail.

For each mask of each class, the output of the neural network is a class mask (respectively, the class of anatomical structure or the class of pathology) with probabilities of pixel belonging to the class in the range 0-1. For each class mask, image areas are determined - candidates for belonging to the class. Determination of candidates means simple segmentation by a threshold calculated on the basis of maximization of the pixel-by-pixel F1-score over the entire training dataset, hereinafter referred to as the segmentation threshold. In another implementation, segmentation is performed by a threshold determined within a single image using the ADAPTIVE_THRESH_GAUSSIAN method (an adaptive threshold equal to the sum of the neighborhood values weighted based on a Gaussian window). The result of segmentation is a binary class mask, hereinafter referred to as an object mask. Next (for example, using the opencv library), a search for contours for segments contained in the object mask is performed. Next, for each contour on the object mask, a region of interest mask is generated from the contour, which means that all mask elements lying inside this contour are considered equal to one, and outside this contour - equal to zero. The region of interest mask is multiplied element by element by the original class mask. Next, all non-zero pixels of the region of interest mask (N) and the sum of the probabilities for all pixels of the mask (M) are counted. The probability of belonging of the considered area (region of interest) to the class is defined as P = M / N. P is compared with the detection threshold. The detection threshold is understood to be the threshold calculated on the basis of maximization of average F1 for the object mask. If P is greater than the detection threshold, the area is considered to belong to the desired class. If P is less than the detection threshold, the candidate area is considered to be an area with a false positive response, and the object mask in the area of the corresponding area is reset to zero. For each isolated area of the object mask, the obtained average probability for the area is saved. The maximum probability over all isolated areas is considered the probability of the presence of the class as a whole.

In another embodiment of the pathology diagnostics block, post-processing of the output data of the neural network contained therein can be performed by preliminary binarization of the mask of each class, extraction of contours, representation of each contour as a separate detection object, calculation of the average probability of the pixels of the initial mask within each contour, followed by comparison of this probability with the detection threshold calculated on the basis of the AUROC curve for the entire training sample.

The indicated approaches allow to increase the accuracy of diagnostics by increasing the accuracy of the operation of the blocks included in the proposed system.

The neural network performing the segmentation may be based on various suitable architectures known to the person skilled in the art. As non-limiting examples, the neural network may be based on the UNet or FPN architecture.

UNet as a source of basic segmentation

The UNet architecture was developed for image segmentation and is currently widely used in medical imaging due to its versatility and high-precision segmentation results. It consists of a downsampling encoder path that extracts image features at different scales and complexities, and an upsampling decoder path that generates segmentation results using the extracted features. The basic concept of UNet is to use skip connections that run from the encoder to the decoder, going from the early layers of the encoder path, where fine image details are still preserved, to the later layers of the decoder path, which contain the coarse feature information needed for object recognition and segmentation. The encoder path consists of four blocks, each consisting of two 3×3×3 convolutional layers followed by a ReLU activation layer and a 2×2×2 max-pooling layer. The decoder path consists of four blocks, each consisting of an upsampling layer, a 2×2×2 transpose convolution layer, and two 3×3×3 convolution layers, each followed by a ReLU layer.

UNet can be trained using the Adam optimizer with a learning rate of 0.0002 and a weight decay of 0.0001.

FPN as a source of basic segmentation

The FPN architecture with RPN is based on the principle of efficient multi-scale feature extraction, which allows combining semantically robust features with low resolution and semantically weak features with high resolution. This approach is effective for the task of object detection. The FPN component extracts features from the input image, and during this process, it has some similarities with UNet. The decoder part creates higher-resolution feature maps by upsampling the results of previous layers, and each block in the decoder path generates a separate output for training the network. Trailing connections consist of 1×1 convolution layers applied to encoder blocks, which are then added to the corresponding decoder blocks. As a result, FPN aims to generate variable-resolution features that can describe the target object. In a specific example, FPN can be trained using the Adam optimizer with a learning rate of 0.0002 and a weight decay of 0.0001.

Training neural networks

In neural network training, it is proposed to use the mean value of the focal loss function as a loss function - this is a variation of crossentropy designed to solve the problem of class imbalance, and the loss function based on the Jaccard coefficient (Jaccard loss), the peculiarity of which is the independence of the error value from the size of objects in the image, and for each error, the proper weights of each class were used.

Metrics used in the learning process

The following metrics are proposed to be used in the training process: IOU (intersection relative to union), average recall, precision, F1, and per-pixel recall, precision, F1. The following rules are proposed to be used to calculate the metrics. For each neural network class, the output of the segmentation mask greater than 0.5 is estimated as 1, less than 0.5 is estimated as 0. The search for contours is performed for the resulting binary segmentation mask and for the true segmentation mask (for example, using the cvcontours library). Each contour found is estimated as a separate object, similar to bb in the detection task. After that, the corresponding metrics are calculated based on the critical value of IOU = 0.5.

Pixel weight combined with Focal loss to increase segmentation accuracy

In one embodiment, the focal loss weights may be designed to correct for systematic bias caused by the different number of pixels that belong, on average across the sample, to each class and background.

For this purpose, the focal loss can use weights for positive elements of the true class mask and negative elements of the class mask based on the frequency of pixels belonging to a specific class in the entire sample (hereinafter pixel-weight, pixel weights). After obtaining the segmentation mask, the focal loss is calculated for each mask element. If the corresponding element of the true segmentation mask is positive, the error obtained on this segmentation mask element is multiplied by the pixel-weight of the positive example of this class. If the segmentation mask element is negative, the error obtained on this segmentation mask element is multiplied by the pixel-weight of the negative example of this class. After multiplying by the weights, the error is averaged over all elements of the weighted error. Thus, the segmentation accuracy increases.

Class-weight combined with Jaccard loss to improve segmentation accuracy

In one embodiment, the Jaccard loss weights may be designed to correct for the systematic imbalance caused by the fact that not all images contain pathological classes, as opposed to normal anatomy classes.

For this purpose, for each error in Jaccard loss, it is proposed to use the proper weights of each class. The pathology classes in the present invention are not opposed to each other. The normal anatomy classes in the present invention are not opposed to each other.

If for some image in the true mask of some class there are positive elements, then the whole instance is assigned a positive class label, if absent - a negative class label. Based on the frequency of positive and negative class labels for all images in the sample for each class, based on the inverse frequency of class labels, class weights (class-weight) of the positive and negative class labels are determined. After obtaining the segmentation masks for each class, Jaccard loss is calculated for each mask. If the true class label corresponding to the image is positive, then the error of the corresponding class is multiplied by the class-weight for the positive instance of the class, if negative - by the class-weight for the negative instance of the class, respectively. The resulting error is averaged. Thus, the accuracy of segmentation increases.

Dynamic hyperplane of network parameter optimization

In the above implementation with pixel-weight for focal loss, the starting value of the weights was based on the pixel frequency in the entire sample. Using this approach even for a small representation of a class in an image provides a high value of per-pixel recall and average recall. The disadvantage of this choice of weights is that in some cases the neural network can quickly converge to the state of highlighting a large number of false positive areas of the image. To solve this problem, it is proposed to modify the pixel-weight every epoch to achieve a balance between the average recall and average precision according to the following principle: the pixel-weight for a positive pixel is multiplied by the result of dividing (average F1)/(average recall), for a negative pixel - by the result of dividing (average F1)/(average precision), if the ratio between average precision and average recall is outside the range from 0.9 to 1.1. These ratios can be represented as follows:

Positive_PW _{n +1} = Positive_PW _n * (average_F1/average_recall)

Negative_PW _n+1 = Negative_PW _n * (average_F1/average_precision)

where Positive_PW is the pixel-weight for a positive pixel, Negative_PW is the pixel-weight for a negative pixel, n is the epoch number.

This ensures an increase in the accuracy of the neural network by eliminating false positive results while maintaining high metrics provided by the pixel-weight approach for focal loss. Accordingly, the accuracy of diagnostics (definition, detection) of pathologies in images also increases.

Example of implementation

The proposed system was tested on a database containing 6139 lung X-ray images. Of this set, 75% of the images were used for training and 25% for testing.

The age of patients was not taken into account. The most common diagnoses were cardiomegaly, pneumonia, hydrothorax, corresponding to the presence of signs from the list: cardiomegaly, pleural effusion, infiltration.

The parameters of individual components of the system were determined as follows.

The network was trained in the multilabel-segmentation mode, sigmoid activation was applied to the network output using the error (loss) function Jaccard loss and focal loss with subsequent calculation of their average, with preliminary application of pixel-weight and class-weight, with correction of pixel-weight each epoch to optimize detection labels.

The learning rate for the models was chosen to be 0.00001. The framework was trained on a server equipped with a Tesla V100-SXM2-80GB graphics processing unit (GPU), an 80-core Intel(R) Xeon(R) E5-2698 v4 central processing unit (CPU) @ 2.20 GHz, and 503GB of RAM.

The efficiency of the system was assessed by assessing the accuracy of the IOU indicator for segmentation masks and class labels calculated based on segmentation, according to the method described earlier, using such classification metrics as recall, precision, F1-score. The results were compared with other systems and with human assessments, and it was found that the proposed system can successfully identify lung pathologies, demonstrating increased accuracy and speed of diagnosis for almost all indicators. With the presence of an external correction mechanism, it is possible to further increase the accuracy of diagnosis, while the help of an external specialist is required only in cases where there is a possibility of an incorrect automatic diagnosis.

Application

The systems and methods according to the present invention can be used to process diagnostic medical images containing the lung area in order to detect signs of tuberculosis, neoplasms, atelectasis, pneumothorax, rib fractures, pulmonary edema, residual changes after tuberculosis and other diseases.

Additional implementation features

Although this document may indicate that data is transmitted/sent or received/received by a person (e.g., a medical professional, a doctor, an expert), one skilled in the art should understand that such indication is used solely for the purpose of simplifying the description, whereas in fact it is meant that the data is transmitted/sent or received/received by the corresponding device used and/or controlled by such person.

One or more of the transmission units or devices (transmitters) and one or more reception units or devices (receivers) described herein may be physically implemented in the same transmission/reception unit or device or in different units or devices.

In this document, a device or unit for transmitting may be referred to as a device or unit having functions of not only transmitting but also receiving data, information and/or signals for ease of description. Similarly, a device or unit for receiving may also include functions of transmitting data, information and/or signals.

The various illustrative blocks and modules described in connection with the disclosure herein may be implemented or executed by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete logic element or transistor logic, discrete hardware components, or any combination of the foregoing designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration).

Some units individually or together may represent, for example, a computer, and include a processor that is configured to call and execute computer programs from the memory to perform the steps of the method or the functions of the units according to the embodiments of the present invention. According to the embodiments, the device may further include a memory. The processor may call and execute computer programs from the memory to perform the method. The memory may be a separate device independent of the processor, or may be integrated into the processor. The memory may store code, instructions, commands and/or data for execution on a set of one or more processors of the described device. The codes, instructions, commands may cause the processor to perform the steps of the method or the functions of the device.

The functions described in this document may be implemented in hardware, software executed by one or more processors, firmware, or any combination thereof. The hardware and software implementing the functions may also be physically located in different locations, including in a distribution such that parts of the functions are implemented in different physical locations, i.e., distributed processing or distributed computing may be performed.

If necessary (for example, if the volume of data and/or calculations that need to be performed on this data is large), multi-threaded data processing can be performed, which in a simple representation can be expressed in the fact that the entire set of data to be processed is divided into a set of subsets, and each processor core performs processing on the subset of data assigned to it.

The above-mentioned memory may be volatile or nonvolatile memory or may include both volatile and nonvolatile memory. It should also be clear to a person skilled in the art that when talking about memory and storing data, programs, codes, instructions, commands, etc., it is implied that there is a machine-readable (or computer-readable, processor-readable) storage medium. Machine-readable storage media include both non-transitory computer storage media and communication media, including any transmission medium that facilitates the movement of a computer program or a portion thereof from one place to another. A non-transitory machine-readable storage medium may be any available medium that can be used to carry or store the desired program code means in the form of instructions or data structures and that can be accessed by a computer, processor or other general-purpose or special-purpose processing device.

By way of example, and not limitation, computer-readable media may comprise read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), flash memory, random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), increased speed synchronous dynamic random access memory (ESDRAM), synchronous line DRAM (SLDRAM), and direct access bus random access memory (DR RAM), register, cache memory, semiconductor memories, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROMs and digital versatile discs (DVD), as well as any other data carriers known in the prior art.

The information and signals described herein may be represented by any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, and signal elements, which may be exemplified in the above description, may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof, as applicable to the present invention.

At least one of the steps in the method or units in the device may use an artificial intelligence (AI) model to perform the corresponding operations. The function associated with AI may be performed via a processor and non-volatile and/or volatile memory.

The processor may include one or more processors. At the same time, the one or more processors may be a general-purpose processor, such as a central processing unit (CPU), an application processor (AP), etc., a graphics-only processing unit, such as a graphics processing unit (GPU), a visual processing unit (VPU), and/or a specialized AI processor, such as a neural processing unit (NPU).

One or more processors control the processing of input data in accordance with a predetermined operating rule or an artificial intelligence (AI) model stored in non-volatile memory and/or volatile memory. The predetermined operating rule or an AI model may be obtained by training. In this case, the processor may perform a pre-processing operation of the data to transform it into a form suitable for use as input data for the AI model.

"Obtained by training" means that by applying a training algorithm to a trainable artificial intelligence model using a plurality of training data, a predetermined operating rule or AI model with a desired characteristic is created. The training may be performed on the device itself, in which the AI is executed according to the embodiment, and/or may be implemented through a separate server/system.

An artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values (coefficients) and performs a work operation for that layer by calculating using the multiple weight values of that layer with respect to the input data or the calculation result of the previous layer.

Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GANs), and deep Q-networks.

A learning algorithm is a method for training a predefined target device (e.g., a GPU-based neural network or NPU) using a set of training data to cause, enable, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

It should be understood that although terms such as "first", "second", "third", etc. may be used herein to describe various blocks, modules, networks, elements, components, regions, layers and/or sections, these blocks, modules, networks, elements, components, regions, layers and/or sections should not be limited by these terms. These terms are used only to distinguish one block, module, network, element, component, region, layer or section from another block, module, network, element, component, region, layer or section. Thus, a first block, module, network, element, component, region, layer or section may be referred to as a second block, module, network, element, component, region, layer or section without departing from the scope of the present invention. As used herein, the term "and/or" includes any and all combinations of one or more of the corresponding listed items. Elements mentioned in the singular do not exclude a plurality of elements unless otherwise specifically indicated.

The functionality of an element indicated in the description or claims as a single element may be implemented in practice by means of several components of the device, and vice versa, the functionality of elements indicated in the description or claims as several separate elements may be implemented in practice by means of a single component.

In one embodiment, some or all elements/blocks/modules of the proposed device are in a common housing, can be placed on a single frame/structure/printed circuit board/crystal and are structurally connected to each other by means of assembly operations and functionally by means of communication lines. The said communication lines or channels, unless otherwise specified, are typical communication lines known to specialists, the material implementation of which does not require creative efforts. A communication line can be a wire, a set of wires, a bus, a track, a wireless communication line (inductive, radio frequency, infrared, ultrasonic, etc.). Communication protocols over communication lines are known to specialists and are not disclosed separately.

The functional connection of elements should be understood as a connection that ensures the correct interaction of these elements with each other and the implementation of one or another functionality of the elements. Particular examples of a functional connection may be a connection with the ability to exchange information, a connection with the ability to transmit electric current, a connection with the ability to transmit mechanical movement, a connection with the ability to transmit light, sound, electromagnetic or mechanical vibrations, etc. The specific type of functional connection is determined by the nature of the interaction of the mentioned elements, and, unless otherwise specified, is ensured by widely known means, using widely known principles in technology.

Although exemplary embodiments have been described in detail and shown in the accompanying drawings, it should be understood that such embodiments are illustrative only and are not intended to limit the present invention, and that the present invention should not be limited to the specific arrangements and structures shown and described, since various other modifications and embodiments of the invention may be apparent to one skilled in the art from the information set forth in the description and knowledge of the prior art without departing from the spirit and scope of the present invention.

Claims (8)

1. A system for identifying lung pathologies using X-ray images, comprising: an image acquisition unit configured to obtain a two-dimensional X-ray image of a lung region; a pathology diagnostic unit configured to, using at least one processor and memory, detect areas of lung pathologies in the obtained image using a segmentation-based neural network; an anatomical structure segmentation unit configured to, using at least one processor and memory, determine, using a segmentation-based neural network, the region of the heart, the region of the right lung, and the region of the left lung based on their outer contours; a pathology localization unit configured to, using at least one processor and memory, determine the localization of the detected areas of pathology by comparing the output data from the pathology diagnostic unit and the output data from the anatomical structure segmentation unit, and delete the suspected areas of pathology that extend beyond the region to which they correspond; and a report transmission unit configured to generate a report containing information on the presence or absence of lung pathologies in the obtained image based on output data from at least one of the pathology diagnostic unit, the anatomical structure segmentation unit and the pathology localization unit, wherein the pathology diagnostic unit is additionally configured to perform post-processing of the segmentation masks output by the neural network contained therein, the post-processing consisting of preliminary binarization of the mask of each class, contour extraction, representation of each contour as a separate detection object, calculation of the average probability of the pixels of the initial mask within each contour, followed by comparison of this probability with a detection threshold calculated based on the AUROC curve over the entire training sample, wherein each neural network is pre-trained to determine a mask for each class corresponding to it, which contains the probabilities of each pixel belonging to this class, wherein during training a focal loss function and pixel weights are used, which represent weights for the pixels of a positive example of the true mask of the class and the pixels of a negative example of the true mask of the class, based on the frequency of the pixels belonging to a specific class in the entire sample, wherein the focal loss function is calculated for each pixel of the mask after determining the class mask, and if the corresponding pixel of the true class mask belongs to a positive example, the error obtained on this pixel of the class mask is multiplied by the weight of the pixel of the positive example of the given class, and if the corresponding pixel of the true class mask belongs to a negative example, the error obtained on this pixel of the class mask is multiplied by the weight of the pixel of the negative example of the given class. 2. The system according to item 1, in which the report transmission unit is additionally configured to form a copy of the input image with contours applied to it, limiting the area of pathology, and also to indicate text information about the localization of the pathology and numerical values of the area of lung damage. 3. The system according to item 1, in which the pathology localization unit is additionally configured to make a conclusion about the presence or absence of disseminated damage depending on the localization of the pathology in one or two lungs, the number of lesions and the area of damage in each lung. 4. The system according to item 1, in which the pathology localization unit is additionally configured to calculate the transverse size of the lungs and heart, calculate the cardiothoracic index (CTI) on their basis, and reset the probability of the presence of cardiomegaly determined by the pathology diagnostic unit if the calculated CTI is less than the critical threshold. 5. The system of claim 1, wherein after each training epoch the ratio between the average precision and average recall metrics is calculated, and if it goes beyond the range from 0.9 to 1.1, then the pixel weights for the focal loss function are adjusted as follows: the pixel weight for a positive pixel is multiplied by the result of dividing the average F1 metric by the average recall, for a negative pixel - by the result of dividing the average F1 metric by the average precision. 6. The system of claim 1, wherein each neural network is pre-trained to determine a mask for each class corresponding to it, which contains the probabilities of each pixel belonging to the given class, wherein during training a loss function is used based on the Jaccard coefficient and the class weights of the positive and negative class labels, which are determined based on the inverse frequency of positive and negative class labels for all images in the sample for each class, wherein if for a given image in the true mask of a given class there are positive elements, then the entire instance is assigned a positive class label, if absent - a negative class label, wherein the loss function based on the Jaccard coefficient is calculated for each class mask after determining the class mask for each class, and if the true class label corresponding to the image is positive, then the error of the corresponding class is multiplied by the class weight of the positive instance of the class, and if negative - by the class weight of the negative instance of the class, respectively. 7. A method for determining lung pathologies from images, comprising the following stages: using an image acquisition unit, a two-dimensional X-ray image is obtained containing a region of the lungs; using a pathology diagnostic unit containing a neural network for segmentation, areas of lung pathologies are determined on the obtained image, while post-processing the segmentation masks output by the neural network is performed, the post-processing consists of preliminary binarization of the mask of each class, contour extraction, representation of each contour as a separate detection object, calculation of the average probability of the pixels of the initial mask within each contour, followed by comparison of this probability with the detection threshold calculated based on the AUROC curve for the entire training sample; using a block for segmentation of anatomical structures containing a neural network for segmentation, the region of the heart, the region of the right lung and the region of the left lung are determined on the obtained image based on their external contours, using a block for localization of pathologies, the localization of the identified areas of pathology is determined based on a comparison of the output data from the block for diagnostics of pathologies and the output data from the block for segmentation of anatomical structures and the supposed areas of pathologies that go beyond the area to which they correspond are removed; and using the report transmission unit, forming and transmitting a report containing information on the presence or absence of lung pathologies in the obtained image based on output data from at least one of the pathology diagnostic unit, the anatomical structure segmentation unit and the pathology localization unit, to the device that requested the diagnosis, wherein each neural network is pre-trained to determine a mask for each class corresponding to it, which contains the probabilities of each pixel belonging to this class, wherein during training a focal loss function and pixel weights are used, which represent weights for the pixels of the positive example of the true class mask and the pixels of the negative example of the true class mask, based on the frequency of pixels belonging to a specific class in the entire sample, wherein the focal loss function is calculated for each pixel of the mask after determining the class mask, and if the corresponding pixel of the true class mask belongs to a positive example, the error obtained on this pixel of the class mask is multiplied by the weight of the pixel of the positive example of this class, and if the corresponding pixel of the true class mask belongs to a negative example, the error obtained on this pixel of the class mask is multiplied by the pixel weight of a negative example of a given class.