FR3159699A1 - device and method for annotating medical images - Google Patents

Abstract

Device and method for annotating medical images A method for annotating a target image obtained by medical imaging is described, comprising: - an application, on the reference image tiles, of a foundation model trained to obtain a vector representation of a medical image decomposed into tiles, - an annotation of medical data on said reference images, - an application, of the foundation model, to obtain a vector representation of the tiles of said target image, - a comparison of the vector representations of the tiles of the target image and the reference images, - a selection, for each tile of said target image, of at least one tile of a reference image, according to the distance between their vector representation, - obtaining a medical annotation by assigning to each tile of the target image, the annotation of medical data associated with the selected tile of the reference image. Figure for abstract: Fig. 1

Description

device and method for annotating medical images

The present invention relates to the field of annotation of radiological images.

In the field of medical imaging, it is important to note that interpreting images from radiology devices, such as CT scans and MRIs, requires considerable expertise and experience on the part of the radiologist. They may also refer to previous images of the patient or request additional studies, such as thinner slices or contrast-enhanced examinations, to obtain a better visualization or a more precise assessment.

The radiologist methodically examines the different parts of the body or specific regions that require special attention based on an indication. He evaluates the bone structures, internal organs, blood vessels, soft tissues, etc. He examines the images in search of normal anatomical structures and evaluates their appearance, size, shape and position. This involves recognizing the organs, bones, blood vessels and surrounding tissues. These examinations are often triggered when looking for pathologies, anomalies, such as fractures, lesions (e.g. tumors or coronary artery disease), infections, thrombi, congenital anomalies, signs of disease, etc. He compares the suspicious structures to those that are considered normal. Finally, once the anomalies have been detected or not, the radiologist summarizes his observations in a detailed written report, which is then sent to the requesting physician. The report typically includes a description of normal structures, detected anomalies, their location, size, appearance, and other relevant characteristics.

All these steps can make the disease identification process imprecise and time-consuming for a practitioner, and there is a risk of not identifying a pathology. In addition, the interpretation of the same image by two or more different practitioners, or even the interpretation of an image at different time intervals by the same radiologist, can lead to different interpretations of the image. This is, for example, illustrated in the article by Daniela Muenzel et al entitled: “Intra- and inter-observer variability in measurement of target lesions: implication on response evaluation according to RECIST 1.1” published on January 2, 2012. There is therefore a need to improve the detection of pathologies during the interpretation of medical images.

The present invention aims to overcome at least one of the drawbacks of the prior art.

To this end, the present invention proposes a method for annotating at least one target image obtained by medical imaging representing at least one structure of interest:
- an application, on a tile decomposition of a plurality of unannotated reference images, of a foundation model previously trained to obtain a vector representation of a medical image decomposed into tiles,
- an annotation of a plurality of medical data on said reference images, at least one medical data being annotated on the tiles of said at least one structure of interest present in said reference images,
- an application, on a decomposition into tiles of said target image, of said foundation model, to obtain a vector representation of the tiles of said target image
- a comparison of the vector representations of the tiles of the target image and the vector representations of the tiles of at least one of said reference images,
- a selection, for each tile of a structure of interest of said target image, of at least one tile of at least one reference image, the distance between their vector representation of which is less than a threshold or the distance of which is minimal,
- obtaining an annotation of at least one medical data item for the tiles of said structure of interest of said target image, by assigning to each tile of said structure of interest of the target image, the annotation of a medical data item associated with the selected tile of the reference image.

Thus, advantageously, the present invention can make it possible to annotate medical images without re-training neural networks on new images when it is desired to benefit from the content of new images during the analysis of target medical images. The vectors of the reference images obtained are associated, or even matched to one or more annotations. The annotations are transferred to the target image based on a comparison of the reference vectors with the vectors of the target image.

According to some embodiments, said annotation of a plurality of medical data comprises
- an association of one or more labels with a tile of a reference image, said labels belonging to one or more atlases according to a determined classification.

According to certain embodiments, said classification of said labels is chosen from one or more of:
- a hierarchical classification of an organ of the human body, the hierarchical classification corresponding to a level of detail of the components of said organ
- a pathological classification,
- a classification relating to a structure of the human body.

According to certain embodiments, the method comprises, prior to comparing the vector representations of the tiles of the target image and the vector representations of the tiles of at least one of said reference images,
- a selection from said plurality of reference images, reference images representing at least said structure of interest,
- said comparison being carried out between the tiles of the target image and the tiles of the selected reference images.

According to some embodiments, the method comprises,
- a segmentation of at least a portion of said reference images from among said plurality of reference images, to identify one or more structures of interest in said images
- a segmentation of the target image, to identify one or more structures of interest in the target image.

According to some embodiments, the method comprises,
- a segmentation of at least a portion of said reference images from among said plurality of reference images, to identify one or more structures of interest in said images
- an E2' segmentation of the target image, to identify one or more structures of interest in the target image.

According to some embodiments, the method comprises,
- obtaining a vector representation for said segmented structures of interest, from said vector representations of the tiles composing said segmented structures of interest, for the reference images and for the target image
- a comparison of the vector representations of the tiles of the target image and the vector representations of the tiles of at least one of said reference images being obtained by a comparison of the vector representations of the segmented structures of interest.

According to some embodiments, the method comprises
- a selection, for said structures of interest of the target image, of at least one structure of interest of a reference image whose distance between their vector representation is less than a threshold or whose distance is minimal
- obtaining an annotation of at least one medical data item for said structure of interest of said target image, by assigning to said structure of interest of the target image, the annotation of a medical data item associated with the structure of interest of the selected reference image.

According to certain embodiments, the method comprises, following the assignment of said annotation to each tile of said structure of interest of the target image,
- a grouping of tiles associated with the same annotation,
- a determination of the outline of said structure of interest, said outline encompassing said tiles having the same annotation,
- an association of said annotation of said tiles of said structure of interest to said structure of interest.

According to some embodiments, the method further comprises
- smoothing of the contours of said annotated structure of interest.

According to certain embodiments, the method further comprises, prior to said annotation of the target image,
- a selection of an atlas based on one or more criteria chosen from:
- a hierarchical level of detail relating to a pathology, or a malformation, or a suspected anomaly, or a structure,
-one or more types of structures chosen from vessels, and/or arteries, and/or muscles, and/or bones,
- an area of the body, associated with one or more hierarchical levels of detail of this area,
said annotation containing one or more labels of the atlas selected for the at least one structure of interest.

According to some embodiments, the method further comprises,
- a display of said target image with said annotation.

According to some embodiments, the method comprises, following the display of said target image with said annotation,
- the selection of a new atlas different from a previously selected atlas, the new selected atlas including more precise annotations than the previously selected atlas,
- the annotation of at least one medical data item for the tiles of said structure of interest of said target image, by assigning to each tile of said structure of interest of the target image, the annotation of a medical data item associated with the selected tile of the reference image from the new selected atlas.

According to some embodiments, the method comprises
- a correction of the annotations of at least one medical data for the tiles of said structure of interest of said target image, when said annotation is incorrect,
- an enrichment of the set of reference images with said target image on which the annotations have been corrected and an addition of said at least one corrected annotation to said labels.

According to certain embodiments, following the obtaining of an annotation of at least one medical data item for said structure of interest of said target image, said target image is added to the set of reference images, said at least one annotation of at least one medical data item for said structure of interest of said target image being added to said labels.

According to certain embodiments, said at least one target image and/or said at least one reference image are images obtained by computed tomography or by magnetic resonance imaging.

According to some embodiments, said foundation model is trained by self-supervised learning.

According to some embodiments, said structure may be one or more of
- a part of the human body,
- a pathology of a part of the human body.

According to some embodiments, said human body part may be one or more of
- bone structures,
- internal organs,
- blood vessels,
- soft tissues.

According to certain embodiments, said medical data relates to one or more of:
- a naming of an organ, and/or a bone structure, and/or a blood vessel, and/or soft tissue, and/or
- a naming of a pathology.

The present invention also relates to a computer program comprising instructions for executing the steps of the method according to any of its embodiments when said program is executed by a computer.

The present invention also relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to any of its embodiments.

The invention also relates to a device for annotating at least one target image obtained by medical imaging representing at least one structure of interest, the device comprising one or more processors configured to
- applying, on a tile decomposition of a plurality of unannotated reference images, a previously trained foundation model to obtain a vector representation of a medical image decomposed into tiles,
- annotating a plurality of medical data on said reference images, at least one medical data being annotated on the tiles of said at least one structure of interest present in said reference images,
- applying, on a tile decomposition of said target image, said foundation model, to obtain a vector representation of the tiles of said target image
-comparing the vector representations of the tiles of the target image and the vector representations of the tiles of at least one of said reference images,
- select, for each tile of a structure of interest of said target image, at least one tile of at least one reference image, the distance between their vector representation of which is less than a threshold or the distance of which is minimal,
- obtaining an annotation of at least one medical data item for the tiles of said structure of interest of said target image, by assigning to each tile of said structure of interest of the target image, the annotation of a medical data item associated with the selected tile of the reference image.

FIG. 1 There FIG. 1 represents a method according to certain embodiments of the invention.

FIG. 2 There FIG. 2 represents a schematic view of a medical image comprising several structures on which a grid of patches is positioned,

FIG. 3 There FIG. 3 represents an example of segmentation of structures on a medical image,

FIG. 4 There FIG. 4 represents an example of an annotation according to a first classification,

FIG. 4 There FIG. 4 represents an example of an annotation according to a second classification,

FIG. 5 There FIG. 5 represents an example of a patch annotation of a medical image,

FIG. 6 There FIG. 6 represents an example of the use of segmentation during annotation,

FIG. 7 There FIG. 7 represents an example of a device according to an embodiment of the invention.

The present invention relates mainly to the field of medical imaging. Medical imaging may include, but is not limited to, images such as X-rays, CT scans or computed tomography (computed tomography), MRI (magnetic resonance imaging), ultrasound, mammography, angiography, endoscopy, etc.

The present invention can help improve the process of diagnosing pathologies based on the observation of medical images. In particular, it can help to automatically identify anomalies observed in medical images.

In particular, it can help to annotate medical images from reference images and in particular from structures identified in reference images.

A reference image can, for example, be a representative image of an individual identified as a representative individual for a population category. For example, we can take a reference image by gender, by age or age group, by origin. We can thus determine a plurality of reference images representing a human liver according to different ages, genders, etc. We can also determine a plurality of images by pathology. We can thus obtain a representative image of a particular pathology. We can also obtain, for a particular pathology, of a determined organ, a plurality of images representative of subgroups in this particular pathology.

Finally, it is also possible to acquire reference images for a plurality of different acquisition methods; for example, the reference image database can contain different MRI sequences or different images associated with different injection times in CT scan.

By structure, we mean in the following description, bone structures, internal organs, blood vessels, soft tissues, pathologies, anomalies, such as fractures, lesions (e.g. tumors or coronary diseases), infections, thrombi, congenital anomalies, signs of diseases, etc. Thus a structure can be defined as a part of the human body or a pathology relating to a part of the human body or a pathological area relating to a part of the human body.

There FIG. 1 represents an embodiment of a method according to the present disclosure.

A foundation model (known by the English acronym "foundation model") or also called basic model, is trained on a set of medical images, step E1. Preferably, to obtain remarkable performances, this set of medical images comprises at least 100 million images. However, this set of medical images can be reduced to a lower number. By image we mean here either a 2D radiograph or a slice of a 3D acquisition (CT scan or MRI), which typically contains 100 to 1000 of these images. These images are representative of the diversity of radiological practice and preferably:
- cover many different organs/indications,
- represent a plurality of varied examples for the same pathology,
- are obtained with a plurality of acquisition techniques: CT-Scanner (with or without injection)/MRI (via the different available sequences, which are very numerous),
- are obtained using a plurality of imaging systems (CT-scanner, MRI, etc.) supplied by a plurality of different manufacturers,
- are representative of individuals of different genders, ages, origins,
- can come from a plurality of acquisition centers.

This step E1 is optional in the sense that the method as disclosed uses the foundation model as obtained during this step E1, which can be obtained well before the method steps described later.

This step involves adjusting the weights and parameters of the foundation model using the training data. This is typically done by optimizing a loss function using optimization algorithms such as stochastic gradient descent (SGD) or its variants. The objective is to minimize the error between the model's predictions and the actual values. The foundation model used may include from a few hundred million parameters to at least one billion parameters to be relevant to the application of the present disclosure. In some embodiments, the number of parameters may be as high as 20 billion.

Several foundation model training techniques are described in the literature and are not the subject of this disclosure. The techniques themselves, typically Self-Supervised Learning, as described in the paper titled “DINOv2: https://ai.meta.com/blog/dino-v2-computer-vision-self-supervised-learning/)” are public but the radiological data is not, which means that radiology-specific foundation models are not readily available.

The foundation model used is preferably based on visual transformers, as proposed for example by the ViT (https://arxiv.org/abs/2010.11929) or SWIN (https://arxiv.org/abs/2103.14030) models, which define the patches used for the similarity measurement in the method of the present disclosure.

The foundation model thus trained can be used in the process steps as described below.

The foundation model used may be a self-supervised learning (SSL) type model, which is a machine learning approach that allows the model to learn from unlabeled, unannotated data by creating artificial supervised tasks from this data. The foundation model provides as output a vector or latent representation, or a latent vector (known as embedding) of the input data, in this case the medical image received as input or a part of the medical image received as input such as one or more structures of interest.

More precisely, the image is cut into patches, also called tiles, of intermediate size (typically but not exclusively 16x16 pixels) and the foundation model associates with each of the patches a latent vector or representative vector.

There FIG. 2 represents a schematic view of a medical image and its patch grid or on which a grid of patches is placed.

The patches can constitute a grid defined by the foundation model before training, and statically placed on the image. Advantageously, the size of a patch can be defined according to the size of the smallest structure. In other words, the size of a patch can be such that it is not larger than the size of the smallest medical structure to be studied. Thus, a structure as defined above can be covered by one or more patches, or parts of patches. Similarly, a patch can be positioned entirely on a structure or overlap one or more structures. The size of the patches, when composed of a square of 16*16 pixels remains relatively small compared to the largest part of the structures and thus, the edge phenomenon induced by patches only partially included in a structure, can be considered negligible. According to certain embodiments, the method according to the invention may comprise a step of zooming or enlarging the image or a step of increasing the resolution of the image, prior to its coding by the foundation model, so as to adapt the size of the structures according to the size of the patches. This may be advantageous when the size of the patches used by the foundation model is too large compared to the size of the structures to be studied (structures of interest coded by the foundation model).

In some embodiments, the size of the vector representation is fixed upon training the foundation model. For example, the vector representation may be a vector of size 2048.

Different foundation model training architectures and algorithms can be used to create a foundation model dedicated to medical images. For example, the “CLIP”, “DINO” algorithms are known, which can be used in the present disclosure.

When we want to characterize a structure on a target medical image, for example to detect a pathology on a target medical image (by detecting a pathological structure), for example during a new medical examination, at least one piece of data from this target image or this target image is entered into the foundation model.

As indicated on the FIG. 1 , the target image data used by the foundation model can be one or more target images.

Optionally, the method may comprise a step E2 and a step E2’. Steps E2 and E2’ are segmentation steps for segmenting the images into structures of interest present in the images, or part of the structures of interest, to identify one or more structures of interest. This segmentation may be performed using the method as described in the publication “segment anything” by Alexander Kirillov et al published on April 5, 2023. The segmentation makes it possible to group together parts of the image having visual similarities. The segmentation further groups together patches that are located in the same structure.

Such segmentation methods can, for example, be based on contour detection methods. Parts of the image with visual similarities are generally parts of the image that are part of the same structure. This can then advantageously make it possible to automatically obtain the contours of the annotated structures.

At the end of steps E2 and E2', we thus obtain reference images, respectively target images, in which one or more structures are identified or segmented. FIG. 3 represents an example of segmentation of a medical image, on which two different structures, structure 1 and structure 2, are identified. It can be noted that the segmentation steps consist of isolating or even detecting or determining areas of the medical image according to their characteristics, but does not aim to recognize them anatomically.

It can of course be noted that steps E2, E3, E4 and E5 (E5 applied to the data relating to the reference images), are carried out upstream of steps E2’ and E5 (E5 applied to the data relating to the target images), so as to have vector representations of the patches of the reference images, or even segmented images, during step E6. Of course, steps E2, E3, E4 and E5 (E5 applied to the data relating to the reference images) can be carried out as the database of reference images is enriched with other images.

In step E3, annotations are determined that can be used to obtain one or more atlases. An atlas can be described as a database of annotated reference patches. In other words, an atlas represents a visual map.

During step E3, an annotation of at least one medical data item is obtained on said reference images, at least one medical data item being annotated on the patches present in the structures of interest of the reference images.

The medical data relates to one or more of:
- a naming of an organ, and/or a bone structure, and/or a blood vessel, and/or soft tissue, and/or another anatomical structure or substructure and/or
- a naming of a pathology

These annotations are performed on at least one reference image and allow classes associated with patches to be defined, which are called reference patches. An annotation can therefore be performed at the patch level, each patch being able to be annotated, or in other words labeled. A reference patch can correspond to at least one structure of interest, or part of a structure of interest, which can be identified, and is annotated with the label(s) associated with the structure of interest. Thus, a reference patch can be associated with several labels, for example in a hierarchical manner. A patch can be associated with one or more anatomical data, such as a label of a structure (liver, radius, aorta, etc.) and in addition with one or more pathological data (hepatic steatosis). A patch can for example be associated with the liver label and the hepatic steatosis label.

In other words, medical data annotation involves the association of one or more labels with a patch of a reference image, the labels belonging to one or more atlases according to a specific classification.

The classification of labels can be chosen from one or more of:
- a hierarchical classification of an organ of the human body, the hierarchical classification corresponding to a level of detail of the components of said organ
- a pathological classification,
- a classification relating to a structure of the human body,

According to certain embodiments, the annotations of the reference images are carried out manually by an operator such as a radiologist, for example using a graphical interface on a computer, a tablet, etc.

According to certain embodiments, the annotations can be carried out more automatically, by software. For example, the “IMAIOS” software can be used, which annotates a reference image and more particularly the structures present in this reference image.

According to certain embodiments, the annotations comprise at least one anatomical class. By anatomical class, we can mean an organ name, such as liver, heart, lung, etc.

According to some embodiments, the annotations comprise at least one pathology relating to an anatomical class. For example, an annotation may consist of a cancerous lesion (e.g., a lung tumor), metabolic (fatty liver disease), infectious, etc.

According to some embodiments, the annotations made on the reference images are made by structure, in order to facilitate the association of an annotation with a patch. The annotations of the structure are then automatically associated with the patches composing this structure.

When step E2 is not present, structures of interest are also determined during the annotation step E3, for example by tracing the contours of the structures of interest on the image to determine them geographically. The annotations are then applied to these determined geographical areas (corresponding to anatomical structures).

For example, when an image contains a representation of a fatty liver, the contours of this are traced and annotations indicating at least that this structure is a fatty liver are associated with this structure.

When step E2 is present, the annotations made in step E3 can be performed on the reference images segmented during step E2. Segmentation or even the visual identification of reference structures in the reference images allows for simpler annotation of structures than when this does not exist. Indeed, according to certain methods, by selecting a segmented structure of interest, for example via a graphical interface, for example by clicking on the structure with the mouse, a window can open in which the radiologist can enter the name of the structure.

The method may also include an optional step E4 comprising both a customization, E41, and a selection, E42, of the atlas. Indeed, if the target image is obtained following a suspicion of a certain pathology or malformation, a customization of the type of annotations may help to refine the detection of the anomaly or pathology. The customization may consist of
- determine an atlas class based on a hierarchical level of detail relating to a pathology, a malformation, a suspected anomaly, a structure,
- determine a plurality of atlas classes for one or more structures of interest, each linked to a hierarchical level,
- determine a specialized atlas class by type of structure, for example a class for vessels, a class for arteries, for muscles, for bones, etc.
- determine an atlas class for areas of the body, for example an atlas class for the upper back, for the lower back, for the leg, the arm... this can also be combined at a plurality of hierarchical levels.

These personalized atlas classes are saved, like the reference images and the vectors of the associated patches, and can be used and enriched collaboratively by a group of practitioners.

Thus, personalization can have the technical effect of helping with diagnosis by providing a level of detail adapted to a pathology, malformation, suspected anomaly, or structure. Furthermore, depending on their medical specialty, radiologists may wish to obtain annotations in a different way from their colleagues.

Thus, without the need to re-train the foundation model, practitioners can then, while benefiting from the trained and shared foundation model, use their own annotation (atlas) for similarity measures and annotations of the target image and customize the atlases.

This customization can allow defining one or more versions of the atlas. Figures 4a and 4b illustrate, for example, different levels of granularity of the brain analyzed as a structure. A version of the atlas can correspond to a different naming of the structures, to a particular level of detail for a given structure. More generally, a version of the atlas can correspond to a hierarchical level of detail associated with a structure as illustrated in Figures 4a and 4b. Thus, customization has the advantage of improving the detection of pathologies or anomalies of a structure by providing a significant level of detail on this structure.

For example, an oncologist may focus on pathologies, sometimes even pathologies related to a specific organ. A version of an atlas customized for this practitioner may consist of an atlas containing a plurality of very precise annotations of this organ. By increasing the level of detail, one can obtain more precision and/or robustness in the analysis of a medical image.

A rheumatologist will prefer to obtain a version of the atlas containing details on bone structures and such an atlas can allow better detection of diseases such as rheumatism and osteoarthritis.

The atlas customization can be done or redone multiple times and is not static over time. It can also evolve based on the discovery of new pathologies. This can therefore allow for continually better annotating of medical images, without the need for training of the foundation model.

An atlas can be selected from the custom atlases (step E42) when several atlases are available. The selection can be made based on a pathology being sought or a structure of the target image or based on a practitioner's preferences or specialty. The selection can be made based on:
- a hierarchical level of detail relating to a pathology, a malformation, a suspected anomaly, a structure,
- a type of structure, for example a class for vessels, a class for arteries, for muscles, for bones…
- of a body area, for example an atlas class for the upper back, for the lower back, for the leg, the arm...this can also be combined at a plurality of hierarchical levels.

It can be noted that the selection step E42 can be iterated several times (following step E8 described later) if the previously selected atlas does not allow a sufficiently precise annotation of the said structure of interest. By sufficiently precise analysis we can mean that the labels used are too generic or too high level, for example at the level of a structure corresponding to an organ (see FIG. 4 ) and not to the detail of this organ (see FIG. 4 ). It can also be heard that the initially selected atlas was linked to a classification by structure while a pathological classification is more appropriate.

Thus, with a single trained foundation model and a single similarity measurement step, multiple types or levels of annotation can be obtained.

In a step E5, the foundation model receives as input, on the one hand, reference data and, on the other hand, target data. As mentioned previously, step E5 is performed in a decorrelated manner on the reference images and on the target images. The application of the foundation model to the reference images can be performed in the background from a database of reference images, for example, and when new reference images are added to this database. The method requires at least one reference image to have been used by the foundation model in order to perform, in a step E6, the similarity measurement. The greater the number of reference images, the more diverse or adapted to the target image, the more relevant the result of the similarity measurement. By adapted, we can understand that the method allows for better precision when a large quantity of annotated reference images, which have been coded by the foundation model, are available, which contain the structures of interest present in the target image.

As indicated on the FIG. 1 , the data of at least one reference image used by the foundation model can be so-called arbitrary reference images, i.e. not selected (or in other words not pre-sorted) by type of structure.

As mentioned earlier, the foundation model associates a vector representation with a patch of an image. The target images, the reference images, are therefore decomposed or even gridded into patches, (as can be seen in the FIG. 2 ) of a determined size, and for example of size 16*16 pixels prior to their use by the foundation model. The size of the patches is determined by the parameterized patch size of the foundation model. In other words, the foundation model works on patches of a determined size of the reference image and the target image. In other words, the foundation model is applied to a decomposition into patches of the data of a plurality of reference images and the data of the target image. The vector representation obtained for each patch is saved so as to be used in subsequent steps

The method therefore comprises an application, on a decomposition into tiles or patches, of said plurality of reference images, of the trained foundation model to obtain a vector representation of an image decomposed into patches.

The method therefore comprises an application, on a decomposition into tiles or patches, of the target image, of the trained foundation model to obtain a vector representation of an image decomposed into patches.

According to certain embodiments, the foundation model receives as input one or more previously selected reference images focusing, for example, on a structure of interest. This embodiment may be of interest when there is no access to a database of reference images.

In such an embodiment, it can be considered that the vector representations of the patches of the reference images have not been generated by the foundation model beforehand nor recorded in a database beforehand but are generated at the same time as the generation for the target image.

As described later, new reference patches can enrich the method when new pathologies are detected, or when new patches are considered particularly relevant to serve as reference patches.

At the end of step E5, we can therefore have a plurality of reference patches with which a reference vector is associated.

It is also possible, according to certain embodiments, to have reference vectors per reference structure and target vectors per structure of the target image. Indeed, when the patches are associated with a structure, for example following the segmentation carried out during step E2 (or E2'), a vector can be determined per structure from the vectors of patches associated with the structure obtained by the foundation model. The determination of a vector per structure can be done according to different methods, following the obtaining of a vector per patch. According to one embodiment, the vector associated with the structure can be obtained by taking an arithmetic mean of the vectors of the patches identified as patches of the structure. According to another embodiment, all of the vectors of the structure are all compared to the reference vectors in step E6.

The application of the foundation model to the reference images therefore generates a plurality of reference vectors, associated with patches, these vectors associated with patches being able to be grouped to determine a vector associated with a structure common to the patches. The reference images, as well as the vectors, associated with the patches or the structures are recorded, for example in a database so as to be used during the similarity measurement step E6.

The vector description of the target image is compared to the vector description of the reference image or images, step E6. More specifically, the vector representation of a target patch of a target image is compared to the vector representation of the reference patches. According to certain embodiments, when vectors per structure are obtained from the vectors per patch, the similarity measurement can be made on the vectors obtained per structure.

Step E6 includes:
- the comparison between the vector representation of the patches of the target image and the vector representations obtained for the patches of at least one reference image,
- the selection, for each tile of the target image, of at least one tile of at least one reference image, the distance between their vector representation of which is less than a threshold or the distance of which is minimal.

As mentioned previously, to shorten the calculation times, it may be advantageous to select the reference images used during the similarity measurement according to structures of interest sought in the target image. Thus, prior to the similarity measurement, the method may comprise a step (not shown) of selecting a plurality of reference images comprising the structure(s) of interest present in the target image or assumed to be present in the target image, in order to carry out the similarity measurement with the vectors of the patches of the selected images. Advantageously, the selection consists of selecting the reference images comprising the anatomical structure(s) present in the target image. This selection may advantageously be based solely on anatomical structures and not on pathologies. This selection may be carried out according to received medical data, the medical data being able to relate to a pathology or an anomaly. This data can be considered as statistically potentially present in at least one structure of the target image, for example by a practitioner having carried out a clinical assessment on the patient to whom the target image belongs prior to the creation of the medical image. This selection can consist of selecting reference images containing structures present in the target image, healthy and pathological. The selection can be even more precise by selecting reference images containing the structure(s) present in the target image showing a suspected pathology in the target image.

Thus, for example, a target liver image can be compared to all or at least a plurality of reference liver images including both pathological and healthy livers. By selecting only liver images, the similarity step is faster since the number of images is reduced and in particular reduced to relevant images.

More specifically, the structures of interest can be related to a pathology or an anatomical abnormality. For example, when the radiologist suspects that a patient has NASH disease, the radiologist can select reference images including a structure corresponding to a liver annotated as a liver with NASH disease and use the reference vectors of these images as reference vectors to be used for the similarity measurement. This advantageously avoids comparing the vectors of the target image with all the vectors of the reference images, most of which are very far apart, thus shortening the computation time.

The comparison can be carried out by a similarity measurement between the vector representation of the target patch (respectively of the target structure) and each of the vector representations of the reference patches (respectively of the reference structures). The similarity measurement can for example be carried out by measuring the Euclidean distance between vector representations.

One or more reference vector representations can be selected whose distance is the smallest with the vector representation of the target patch. When selecting several vector representations, one can select the vector representations whose distance is less than a predetermined or parameterized distance. This can improve robustness because if several reference patches representative of a pathology, for example Nash disease, are selected, this can confirm the diagnosis of this pathology on the target liver.

We therefore obtain the reference patch associated with the selected vector representation, the reference patch and the associated vector representation being for example linked by an index, a pointer, or linked in a classic manner in a database.

Since the foundation model produces a vector per patch, a grouping of patches forming a structure of interest can be carried out and the similarity measurement can be carried out at the structure scale and not at the patch scale. As mentioned above, when the target image and the reference image are segmented (steps E2’ and E2) one vector per structure can be determined. Thus, the similarity measurement can be carried out by comparing a vector of a segmented structure of the target image to the vectors of the segmented structures of the reference images. This can advantageously be faster than comparing the patches one by one.

Once the reference patch or the reference structure has been selected during step E6, the annotations of this selected reference patch or of this selected reference structure are transferred to the target patch, respectively to the structure of interest of the target image, step E7. FIG. 5 represents an example of annotated patches.

Following the obtaining of an annotation of at least one medical data for the structure of interest of the target image, the target image is added to the set of reference images, the at least one annotation of at least one medical data for the structure of interest of the target image being added to the labels.

Advantageously, the method may comprise, following the assignment of the annotation to each patch of the structure of interest,
- obtaining the outline of the structure of interest, the outline encompassing the patches having the same annotation,
- an association of the annotation of the patches of the structure of interest to the structure of interest.

This allows for clearer reading of the annotations. Indeed, an annotation by patch can be less readable than an annotation by structure. We can note, as in the FIG. 5 that annotations can be indicated as color per patch, with a color being associated with an annotation. Here, for example, liver patches are colored light gray, and light gray is associated with the liver.

Structures are generally of a different geometric shape than the shape of patches. Indeed, the shape of structures is closer to an ovoid shape while patches as defined are square (or possibly rectangular). Thus, it is easy to understand that when decomposing an image or a structure into patches, at least some of the patches may be on the edge of a structure and include both pixels belonging to the structure and pixels not belonging to the structure creating edge effects. As indicated in FIG. 5 , the representation of patches without them being grouped into a structure offers less visibility than when the patches are grouped into a structure, FIG. 6 .

Indeed, the patches included on the edges of the structures of interest have representative vectors that are probably far from the representative vectors of the reference patches of this structure of interest since they include both pixels of the structure and pixels outside the structure. A vote can be carried out to determine whether a patch comprising both pixels of the structure and pixels outside the structure is associated with the structure. This vote can be done manually by an operator or automatically. This vote can for example be based on a proportion of pixels that are part of the structure, and if this proportion is greater than a first threshold, the patch is selected as belonging to the structure. This determination can be carried out using the segmentation information obtained during step E2’.

There FIG. 6 illustrates an embodiment of obtaining the annotation of the structure of interest from the annotation of the patches and the segmentation. According to this embodiment, the contours of structures (bottom right image) are obtained by the segmentation obtained during step E2'. The classification of the vectors by tile obtained during the similarity measurement of step E6 is illustrated on the top right image. The left image is obtained on which a liver and spleen annotation is obtained, by combining the contours of the bottom right image and the classified tiles of the top left image. Thus the tiles of the edges of the structure are associated with the structure according to the contours of the structure.

According to other embodiments, when the segmentation into the structure of interest and therefore the image at the bottom right is not available (absence of steps E2 and E2'), the annotated image is obtained (left image of the FIG. 6 ) from the classification of patches (top right image of the FIG. 6 ) on which a vote can be taken to determine whether a patch comprising both pixels in the structure and pixels outside the structure is associated with the structure. The association of a patch comprising only a portion of these pixels in a structure with that structure can be established from anatomical considerations and thus the outlines of the structures can be determined.

So, we group the tiles associated with the same annotation, as mentioned above,
- we determine the outline of the structure of interest, the outline encompassing the tiles having the same annotation,
- we associate, with the structure of interest, the annotation of the tiles of the structure of interest.

An example is given in FIG. 5 where we can observe the problems of structure edge effects. Since the reference annotations are moved to each of the patches independently, this is a source of noise: neighboring patches may be associated with different annotations. It may therefore be decided that some of these patches are liver patches and others are not. Smoothing methods can be used to adjust the edges or inconsistencies. The smoothing method may consist of using convolution kernels to attenuate this noise

In some embodiments, it may be necessary to adjust the annotations to accommodate certain morphological constraints of the morphology of the patient whose image is the target image. For example, it may be necessary to adjust contours of the organ of interest.

In order to be used by the radiologist, the annotated target image can be displayed, for example on a screen, step E8. The radiologist can thus, simply and directly, view the analyzed image of the patient. The method therefore advantageously allows him to benefit from a report created automatically from the reference images. In addition, the level of detail of the annotations may have been personalized during step E41, the annotations are therefore adapted to the personalized level of detail to allow the detection of anomalies or pathologies. If the displayed annotations are not sufficient for the interpretation of a pathology or anomaly, it may be considered to select a new atlas, step E42, whose annotations are more precise or more suitable. For example, an atlas corresponding to the arm may have been selected. When analyzing the displayed image including the annotations present in the atlas relating to the arm, the radiologist may wish to display details on the wrist if he suspects a problem at the wrist level. The method may include a new selection of an atlas relating to the wrist. Steps E7 and E8 are repeated following this new selection.

Following the selection of a new atlas, the method comprises steps E7 to E9. During step E7, at least one medical data item is annotated for the tiles of the structure of interest of the target image, by assigning to each tile of the structure of interest of the target image, the annotation of a medical data item associated with the selected tile of the reference image from the new selected atlas.

In certain embodiments, the method comprises a correction step, E9. This step can advantageously be implemented, for example when the annotations viewed during step E8 are incorrect, in particular at the scale of the structure. This correction can consist of a modification of the annotation or annotations made during step E7, for example by receiving, through a user interface, corrective annotation information. For example, it may happen that new target images are analyzed, containing structures different from those present in the reference images. The measurement of similarity between the vectors representative of these new images and the vectors representative of the reference images can lead to the selection of reference vectors distant from the reference vectors of the target images, even if they are the closest, which can lead to an annotation that is not very precise, or even very distant.

For example, when there is ascites around the liver, in some liver pathologies, the appearance of the liver may be altered and a target image containing such a liver may be difficult to annotate as a liver image. Step E9 can correct this annotation error.

Thus, for example, when a new target image showing a diseased liver bathed in ascites is analyzed, its reference vectors are close to the reference vectors of the previously analyzed and corrected reference image containing such a liver. The annotations of this reference image containing the liver surrounded by ascites are therefore used to be transferred during step E7 to this target image.

Thus, advantageously, step E9 can allow the method to improve when the corrected images are added to, or even enriched, the database of reference images. These new reference images added to the reference image database are injected into the foundation model to obtain their reference vector. Thus, the model has a new reference image, its reference vector and the associated annotations. The method as described therefore advantageously makes it possible, without re-training the foundation model, to benefit from the medical information of new reference images.

According to certain embodiments, during step E9, the radiologist, in addition to correcting, can also adjust the annotations obtained from the annotations of the selected reference image. Thus, it may be possible to customize the annotations to the target image, for example if the radiologist detects new pathologies. Thus, following step E9, the method can move on to step E3 in which the existing atlases can be modified or supplemented from the information identified during the display, step E8. This can make the annotations more robust. In certain cases, new atlases can also be created.

This customization can also be linked to the radiologist's preferences.

According to certain embodiments, it may be possible to modify the contours by smoothing methods during step E9. This is particularly relevant when the images have not been segmented as described in steps E2 and E2’. The smoothing method may consist of using convolution kernels to attenuate this noise.

According to certain embodiments, a collaborative platform can be made available allowing practitioners to use such a method by enriching it with their own medical images and/or their own reference images. Thus, when certain practitioners or care centers are specialized in a pathology, an organ, etc., they can annotate reference images targeted on this pathology and/or this organ of interest, these new annotations being able to constitute new semantic classes of the atlas. The present disclosure can thus advantageously make it possible to detect pathologies that are not present in the training images of the model but which are discovered during examinations of the target images. These new target images become reference images and provide new annotations.

To this end, new reference images can be added by practitioners in order to enrich the process and make the similarity measurement more relevant. The higher the number of reference images, the more efficient the process is in terms of annotation and the less adjustment it requires by the practitioner in the formatting of a report.

The present disclosure can be used for the identification of characteristic “signatures” of pathologies. For example, there are several subgroups of hepatocellular carcinomas (HCC), the most common liver cancer: typically there are subgroups of pseudo-progressors, progressors, and hyper-progressors. Representative images of a liver affected by each of the pathology subgroups can be annotated by a practitioner and the foundation model can be applied to these new images constituting reference images. By comparing these reference images with that of a new patient suffering from HCC, it is very simple to identify the subgroup associated with this HCC. The foundation model can thus be used to define a signature of the HCC subgroups.

The present disclosure can in particular be applied to the detection of Nash disease (an acronym for “non-alcoholic steatosis-hepatitis”). This disease potentially affects a significant number of people in the general population, between 1.5 and 6% of the population, and currently remains underdiagnosed. There are therefore potentially many patients suffering from this disease who are unaware of it in the cohorts of images already collected and which have, for example, been used to train the foundation model. Thanks to the present disclosure, practitioners can annotate images comprising livers presenting this pathology. These images are reference images which are entered into the foundation model and each produce a vector representation (per patch) as output. These vector representations are then also used, with all the vector representations of the patches of the reference images already collected, for the similarity measurements carried out during step E6. Thus, the present disclosure does not require retraining the foundation model with images representative of the NASH pathology. A similarity measure of vector representations of NASH pathological livers from the foundation model with vector representations of reference images with NASH livers from the foundation model and annotated can allow for rapid and automatic annotation of NASH liver images. The foundation model can thus allow for the identification of a signature of livers with NASH disease.

The present invention also relates to an annotation device implementing the method as described previously in at least one of these embodiments. To this end, the present disclosure relates to a device for annotating at least one target image obtained by medical imaging representing at least one structure of interest, the device comprising one or more processors configured to
- applying, on a tile decomposition of a plurality of unannotated reference images, a previously trained foundation model to obtain a vector representation of a medical image decomposed into tiles,
- annotating a plurality of medical data on the reference images, at least one medical data being annotated on the tiles of the at least one structure of interest present in the reference images,
- apply, on a tile decomposition of the target image, the foundation model, to obtain a vector representation of the tiles of the target image
- compare the vector representations of the tiles of the target image and the vector representations of the tiles of at least one of the reference images,
- select, for each tile of a structure of interest of the target image, at least one tile of at least one reference image, the distance between their vector representation of which is less than a threshold or the distance of which is minimal,
- obtaining an annotation of at least one medical data item for the tiles of the structure of interest of the target image, by assigning to each tile of the structure of interest of the target image, the annotation of a medical data item associated with the selected tile of the reference image.

There FIG. 7 represents an example of hardware architecture 10 of an annotation device allowing the implementation of a method according to the present disclosure and as represented for example in FIG. 1 . This hardware architecture is that of a computer. Other hardware architecture elements are present in the device 10 and not shown here.

The hardware architecture 10 comprises one or more processors 11 (only one is shown in the FIG. 2 ) implementing a method according to the present disclosure, a read-only memory 12 (of the “ROM” type), a rewritable non-volatile memory 13 (of the “EEPROM” or “NAND Flash” type for example), a rewritable volatile memory 14 (of the “RAM” type) a communication interface 15 with an external network (for example a cellular network for accessing external databases capable of recording the reference data, the databases being able to be shared and fed collaboratively) and a communication interface 16 (or several communication interfaces) with radiology equipment for example. The read-only memory 12 constitutes a recording medium in accordance with an exemplary embodiment of the invention, readable by the processor or processors 11 and on which is recorded a computer program Prog in accordance with an exemplary embodiment of the invention comprising instructions for executing steps of the method according to one or more of the embodiments of the invention. Alternatively, the computer program Prog is stored in the rewritable non-volatile memory 13. In certain embodiments, the hardware architecture 10 may also include a graphics card of the GPU type (for “graphic processor unit” in English), not shown.

The computer program Prog may enable the device 10 to implement at least part of the method in accordance with the present disclosure and as illustrated for example in FIG. 1 .

This computer program Prog can thus define functional and software modules, configured to implement the steps of an annotation method in accordance with an exemplary embodiment of the invention, or at least part of these steps. These functional modules rely on or control the hardware elements 11, 12, 13, 14, 15 cited above.

Claims (19)

Method of annotating at least one target image obtained by medical imaging representing at least one structure of interest:
- an application (E5), on a decomposition into tiles of a plurality of unannotated reference images, of a foundation model previously trained to obtain a vector representation of a medical image decomposed into tiles,
- an annotation (E3) of a plurality of medical data on said reference images, at least one medical data being annotated on the tiles of said at least one structure of interest present in said reference images,
- an application (E5), on a decomposition into tiles of said target image, of said foundation model, to obtain a vector representation of the tiles of said target image
- a comparison (E6) of the vector representations of the tiles of the target image and the vector representations of the tiles of at least one of said reference images,
- a selection (E6), for each tile of a structure of interest of said target image, of at least one tile of at least one reference image, the distance between their vector representation of which is less than a threshold or the distance of which is minimal,
- obtaining (E7) an annotation of at least one medical data item for the tiles of said structure of interest of said target image, by assigning to each tile of said structure of interest of the target image, the annotation of a medical data item associated with the selected tile of the reference image. Method according to claim 1 wherein said annotation (E3) of a plurality of medical data comprises
- an association of one or more labels with a tile of a reference image, said labels belonging to one or more atlases according to a determined classification. The method of claim 2 wherein said classification of said labels is chosen from one or more of:
- a hierarchical classification of an organ of the human body, the hierarchical classification corresponding to a level of detail of the components of said organ
- a pathological classification,
- a classification relating to a structure of the human body,
Method according to one of the preceding claims comprising, prior to the comparison of the vector representations of the tiles of the target image and the vector representations of the tiles of at least one of said reference images,
- a selection from said plurality of reference images, reference images representing at least said structure of interest,
- said comparison being carried out between the tiles of the target image and the tiles of the selected reference images. Method according to one of the preceding claims comprising,
- a segmentation (E2) of at least a part of said reference images among said plurality of reference images, to identify one or more structures of interest in said images
- a segmentation (E2') of the target image, to identify one or more structures of interest in the target image. The method of claim 5 comprising,
- obtaining a vector representation for said segmented structures of interest, from said vector representations of the tiles composing said segmented structures of interest, for the reference images and for the target image
- a comparison (E6) of the vector representations of the tiles of the target image and the vector representations of the tiles of at least one of said reference images being obtained by a comparison of the vector representations of the segmented structures of interest,
- a selection, for said structures of interest of the target image, of at least one structure of interest of a reference image whose distance between their vector representation is less than a threshold or whose distance is minimal
- obtaining an annotation of at least one medical data item for said structure of interest of said target image, by assigning to said structure of interest of the target image, the annotation of a medical data item associated with the structure of interest of the selected reference image. Method according to one of claims 1 to 5 comprising, following the assignment of said annotation to each tile of said structure of interest of the target image,
- a grouping of tiles associated with the same annotation,
- a determination of the outline of said structure of interest, said outline encompassing said tiles having the same annotation,
- an association of said annotation of said tiles of said structure of interest to said structure of interest. The method of claim 7 further comprising
- smoothing of the contours of said annotated structure of interest. Method according to one of claims 2 to 8 comprising, prior to said annotation of the target image,
- a selection (E42) of an atlas based on one or more criteria chosen from:
- a hierarchical level of detail relating to a pathology, or a malformation, or a suspected anomaly, or a structure,
-one or more types of structures chosen from vessels, and/or arteries, and/or muscles, and/or bones,
- an area of the body, associated with one or more hierarchical levels of detail of this area,
said annotation containing one or more labels of the atlas selected for the at least one structure of interest. Method according to one of the preceding claims further comprising
- a display (E8) of said target image with said annotation. A method according to claims 9 and 10 comprising, following the display of said target image with said annotation,
- the selection (E42) of a new atlas different from a previously selected atlas, the new selected atlas including more precise annotations than the previously selected atlas,
- the annotation of at least one medical data item for the tiles of said structure of interest of said target image, by assigning to each tile of said structure of interest of the target image, the annotation of a medical data item associated with the selected tile of the reference image from the new selected atlas. Method according to one of claims 10 or 11 comprising
- a correction (E9) of the annotations of at least one medical data for the tiles of said structure of interest of said target image, when said annotation is incorrect,
- an enrichment of the set of reference images with said target image on which the annotations have been corrected and an addition of said at least one corrected annotation to said labels. Method according to one of the preceding claims in which, following the obtaining of an annotation of at least one medical data item for said structure of interest of said target image, said target image is added to the set of reference images, said at least one annotation of at least one medical data item for said structure of interest of said target image being added to said labels. A method according to any preceding claim wherein said structure may be one or more of
- a part of the human body,
- a pathology of a part of the human body. A method according to any preceding claim wherein said human body part may be one or more of
- bone structures,
- internal organs,
- blood vessels,
- soft tissues. Method according to one of claims 2 or 3 in which said medical data relates to one or other or more of:
- a naming of an organ, and/or a bone structure, and/or a blood vessel, and/or soft tissue, and/or
- a naming of a pathology Computer program comprising instructions for executing the steps of the method according to one of claims 1 to 16 when said program is executed by a computer. Computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to one of claims 1 to 16. Device for annotating at least one target image obtained by medical imaging representing at least one structure of interest, the device comprising one or more processors configured to
- applying, on a tile decomposition of a plurality of unannotated reference images, a previously trained foundation model to obtain a vector representation of a medical image decomposed into tiles,
- annotating a plurality of medical data on said reference images, at least one medical data being annotated on the tiles of said at least one structure of interest present in said reference images,
- applying, on a tile decomposition of said target image, said foundation model, to obtain a vector representation of the tiles of said target image
-comparing the vector representations of the tiles of the target image and the vector representations of the tiles of at least one of said reference images,
- select, for each tile of a structure of interest of said target image, at least one tile of at least one reference image, the distance between their vector representation of which is less than a threshold or the distance of which is minimal,
- obtaining an annotation of at least one medical data item for the tiles of said structure of interest of said target image, by assigning to each tile of said structure of interest of the target image, the annotation of a medical data item associated with the selected tile of the reference image.