FR3156302A1 - Method for acquiring data by endoscopy from biological tissues, and corresponding endoscopic imaging system. - Google Patents

Abstract

The invention relates to a method for analyzing images acquired endoscopically from biological tissues using a light-generating device associated with an endoscope with a camera. [Fig. 4]

Description

Method for acquiring data by endoscopy from biological tissues, and corresponding endoscopic imaging system. Field of invention

The field of the invention is that of medical imaging.

More specifically, the invention relates to the acquisition of data from imaging resulting from endoscopic exploration of biological tissues, and more specifically, of hollow organs or cavities of humans or animals.

Prior art

In the medical field, endoscopes are of crucial importance in enabling visual observation of hollow organs and cavities in the human body, and remain essential for patient monitoring, diagnosis and interventional procedures.

Endoscopes are particularly useful for detecting lesions or signs of inflammation present on the surface of tissues. Indeed, various hollow organ cancers are the consequence of a carcinological process resulting from chronic inflammation. The pre-cancerous lesions at the origin of these cancers are difficult to detect due to their appearance, which is not very different from healthy mucosa, and their flat nature. Therefore, these lesions are often not detected and the diagnosis is made at the cancer stage, which leads to heavy treatment and a decrease in patient survival rates.

In recent years, medical imaging techniques have been developed to attempt to detect tissue changes early, and in particular pre-cancerous or cancerous lesions.

There are methods of illuminating the surface of the organ to be studied with a light source, while moving the endoscope in the human cavity, and acquiring a video with a color camera. Such a method makes it possible to obtain a video reflecting the reflectance of light by the illuminated tissue.

The color camera records the light signal as three bands (red, green, and blue) covering the visible spectrum, while the light source is either white light or light composed of two or four wavelength bands.

The resulting visuals vary depending on the light source used. Video acquisition under white light produces images similar to those perceived by the human visual system. In contrast, video acquisition under light composed of two or four wavelength bands produces false-color images by combining the images.

However, current systems provide true or false color images whose textures, particularly of the mucous membranes, are insufficiently defined and recognizable, which increases the risk for the operator of not detecting lesions. In addition, such systems do not provide any assistance in interpreting the images.

There is therefore an unmet need to provide a method and device for acquiring endoscopic imaging making it possible to obtain images of improved quality and to provide data which can subsequently be exploited, in particular by artificial intelligence.

The present invention solves the problems raised by the prior art.

• la génération d’une lumière sur une zone de tissu biologique par un dispositif lumineux apte à générer n bandes spectrales de longueurs d’onde différentes successives de manière à couvrir le spectre visible et le proche infrarouge et générer une vidéo multispectrale dans laquelle chaque image correspond à l’une des bandes spectrales ;
• le découpage de la vidéo endoscopique ainsi obtenue de manière à extraire des images de bandes spectrales de longueurs d’onde différentes ;
• le recalage des images de bandes spectrales pour obtenir au moins une image multispectrale recalée comprenant une image pour chaque bande spectrale de longueurs d’onde ;
• la reconstruction spectrale d’un cube de réflectance, à partir de l’image multispectrale recalée, dans lequel chaque pixel correspond à un vecteur reflétant la réflectance de la zone de tissu biologique.

For this purpose, the present invention proposes a method for acquiring data by endoscopy from biological tissues using a light generating device associated with an endoscope with camera, said method comprising:

• the generation of light on an area of biological tissue by a light device capable of generating n successive spectral bands of different wavelengths so as to cover the visible spectrum and the near infrared and generate a multispectral video in which each image corresponds to one of the spectral bands;
• cutting the endoscopic video thus obtained so as to extract images from spectral bands of different wavelengths;
• registering the spectral band images to obtain at least one registered multispectral image comprising an image for each spectral band of wavelengths;
• the spectral reconstruction of a reflectance cube, from the registered multispectral image, in which each pixel corresponds to a vector reflecting the reflectance of the biological tissue area.

Images obtained in the spectral bands covering the near infrared are advantageously used to characterize tissues in a more absolute manner by complementarity with the spectral bands covering the visible spectrum.

This multiplicity of spectral bands covering the visible spectrum and the near infrared allows, firstly, the obtaining of a multispectral image having an improved and extended spectral resolution with respect to the color images, and secondly, to generate a reflectance cube in which each pixel corresponds to a vector reflecting only the reflectance of the corresponding tissue area. In comparison with the images acquired by the n spectral bands, such reflectance cubes also include a better spectral resolution over the visible and near infrared range in each pixel specific to the reflectance of the tissue after having freed itself from the radiance of the illuminant and the sensitivity of the camera.

Advantageously, the wavelength bands may be between 380 and 1000 nm, preferably between 400 and 850 nm.

Advantageously, n can be equal to six or a multiple of six.

Advantageously, at least four of the spectral bands may have wavelengths spanning the visible spectrum and at least one of said spectral bands has wavelengths spanning the near infrared.

According to a particular characteristic, at least four of said spectral bands may have wavelengths covering the visible spectrum and at least two of the spectral bands may have wavelengths covering the near infrared.

The use of a second spectral band covering the infrared makes it possible to increase the quality of the multispectral image obtained.

Advantageously, the camera can capture 20 to 30 frames per second during the acquisition of the multispectral video, and only every other frame can be alternatively selected for the production of a multispectral image when cutting the obtained multispectral video.

According to a first advantageous embodiment of the method according to the invention, the spectral reconstruction of a reflectance cube from the multispectral image can comprise processing of the multispectral image by a deep neural network.

• un deuxième recalage (ou « mosaïquage ») des images multispectrales recalées de façon à obtenir une mosaïque d’images multispectrales ;

• un traitement de la mosaïque d’images multispectrales par un réseau de neurones profond ou par un modèle d’interaction lumière-tissus.

According to a second advantageous embodiment of the method according to the invention, the spectral reconstruction of a reflectance cube from the recalibrated multispectral images can comprise:

• a second registration (or “mosaicing”) of the registered multispectral images so as to obtain a mosaic of multispectral images;

• processing of the multispectral image mosaic by a deep neural network or by a light-tissue interaction model.

Mosaicking a set of images consists of positioning the different images in a common reference frame, thus covering a field of view larger than that of the images taken individually.

The implementation of the method according to the second embodiment makes it possible to obtain several multispectral images representing a common area of the observed biological tissue. Thus, the generation of a reflectance spectrum can be carried out from several reflectance data vectors of the same area but acquired with a different positioning of the endoscope, which makes it possible to improve the estimation of the reflectance of the biological tissue.

Preferably, whether for the first or second embodiment, the deep neural network can be pre-trained from reflectance spectra data characteristic of healthy and pathological tissues.

The method may further comprise a histological analysis associated with the area of the tissue, in particular by microscopic analysis. This histological analysis provides ground truth potentially necessary for learning for the purposes of automatic analysis or characterization of tissues.

Advantageously, the method may further comprise analyzing the reflectance cube by artificial intelligence.

It should be noted that the method according to the present description can be implemented indifferently in real time, that is to say during the endoscopic examination, or in a delayed time, that is to say after the endoscopic examination.

• un endoscope configuré pour capter un flux vidéo,
• au moins un boitier générant de la lumière à partir d’une source LED, et
• un écran de visualisation configuré pour afficher des images acquises par l’endoscope et/ou tout ou partie de l’information contenue dans les cubes de réflectance obtenus avec un procédé d’acquisition de données tel que décrit précédemment.

The present invention also relates to an endoscopic imaging system which comprises:

• an endoscope configured to capture a video stream,
• at least one box generating light from an LED source, and
• a display screen configured to display images acquired by the endoscope and/or all or part of the information contained in the reflectance cubes obtained with a data acquisition method as described previously.

List of figures

• FIG. 1illustre les principales étapes d’un procédé d’acquisition de données par endoscopie à partir de tissus biologiques selon l’invention ;
• FIG. 2illustre un exemple de recalage interbandes non rigide selon la présente description ;
• FIG. 3illustre les principales étapes du procédé de laFIG. 1, selon un premier mode de réalisation ;
• FIG. 4illustre les principales étapes du procédé de laFIG. 1, selon un deuxième mode de réalisation ;
• FIG. 5illustre les principales étapes du procédé de laFIG. 4, selon une variante ; et
• FIG. 6illustre les spectres d'absorption de l'hémoglobine.

Other characteristics and advantages of the invention will appear more clearly on reading the following description of preferred embodiments, given as illustrative and non-limiting examples, and the appended figures among which:

• FIG. 1 illustrates the main steps of a method for acquiring data by endoscopy from biological tissues according to the invention;
• FIG. 2 illustrates an example of non-rigid interband registration according to the present description;
• FIG. 3 illustrates the main stages of the process of the FIG. 1 , according to a first embodiment;
• FIG. 4 illustrates the main stages of the process of the FIG. 1 , according to a second embodiment;
• FIG. 5 illustrates the main stages of the process of the FIG. 4 , according to a variant; and
• FIG. 6 illustrates the absorption spectra of hemoglobin.

Detailed description of the invention

The general principle of the method of data acquisition by endoscopy from biological tissues is described in more detail in relation to the FIG. 1 .

During a step 11, a light device associated with an endoscope generates light with n spectral bands of different and successive wavelengths so as to generate a multispectral video in which each image corresponds to one of the spectral bands.

More specifically, this step 11 comprises the generation of a light 111 having wavelength bands covering the visible spectrum and the near infrared, that is to say so as to respectively cover wavelength ranges between 380-780 nanometers (nm) and 780-1000 nm.

The light device may illuminate the tissue to be observed successively with the different spectral bands, or alternatively, may illuminate the tissue to be observed with a plurality of spectral bands.

Each of the spectral bands can cover a wavelength range from a few nanometers to several tens of nanometers, for example a wavelength range of 50 nanometers, preferably a wavelength range of 40 nanometers.

The spectral bands of the generated light are preferably six in number, or a multiple of six. Thus, it is possible to select at least four bands covering the visible spectrum and at least one spectral band covering the infrared.

The four spectral bands covering the visible spectrum make it possible to obtain images, each comprising different absorption peaks.

The accuracy of these results is refined by images obtained with the infrared spectral band(s). Indeed, the spectral bands covering the infrared make it possible to obtain additional absorption peaks, and thus offer the possibility of obtaining information that cannot be perceived with spectral bands covering the visible spectrum.

For example, the following six wavelength bands can be used: a first band centered on 415 nm, a second band centered on 450 nm, a third band centered on 540 nm, a fourth band centered on 605 nm, a fifth band between 650 and 700 nm, a sixth band between 700 and 750 nm.

According to a first alternative, the light emitted by the light device replaces the light emitted by the endoscope, the latter being consequently switched off during the implementation of the present method. In other words, in this alternative, the white light can be switched on and used to position the camera of the endoscope in the organ and the area to be observed, and switched off during the recording of the video.

In a second alternative, the light emitted by the light device complements the light emitted by the endoscope. In other words, the white light of the endoscope remains on during video acquisition.

This second alternative can be implemented in particular when the power of the LED source is sufficiently high to illuminate the observed tissue area and acquire characteristic images of the spectral bands. Indeed, the LED source must ensure that the relevant details of the spectral band of a given wavelength range are captured despite the presence of the white light of the endoscope.

Step 11 further comprises the acquisition of a multispectral video 112 by the endoscope camera, as well as successive illumination 113 of the observed area by the different spectral bands. This results in an endoscopic video in which each image corresponds to one of the spectral bands with which the tissue area was illuminated.

This successive illumination is carried out in a synchronized manner with the camera's frame rate. To do this, frame rate information from the endoscopic camera coming from the video stream is recovered in order to generate a timer based on an electronic device that allows the triggering of the spectral bands emitted by the light source.

During a step 12, a cutting 121 of the endoscopic video obtained is carried out so as to extract images of spectral bands of different wavelengths,

The video needs to be cut in order to eliminate so-called "transition" images. "Transition" images are images obtained between two successive illuminations by different spectral bands, and which generally present unusable data resulting from the change in spectral bands.

For example, the camera can capture 20 to 30 images per second during video acquisition. When the camera has this latter rate, a cutting step can be performed to select every other image from the endoscopic video. This cutting results in one or more multispectral images. Each multispectral image includes at least one image representative of each of the spectral bands.

The image size can be chosen so that the images can encompass one or more lesions present on the surface of the tissue. The dimensions of the images can vary depending on various parameters such as the acquisition conditions, the nature of the organ studied and the pathological areas likely to develop there.

During a step 13, the images resulting from the cutting undergo a recalibration 132 so as to obtain one or more multispectral image(s) comprising an image for each spectral band of wavelengths.

The spectral band images of the same multispectral image are superimposed on each other so that the characteristic elements of the analyzed tissues, present in each of them, are aligned. The characteristic elements are all the tissue variations that can be on the surface of the observed organs, and can be, for example, lesions, signs of inflammation, blood vessels or variations in tissue granulosity.

In particular, a so-called “non-rigid” interband registration is carried out. In this description, “non-rigid interband registration” means an elastic and anisotropic registration for all pixels to respond to the elastic deformations of the observed tissues.

Thus, unlike rigid interband registration which consists of aligning all the characteristic elements of an image by a single geometric transformation composed of a rotation and a translation, non-rigid interband registration consists of aligning each characteristic element obtained in the different spectral bands independently in one or more of the images of the multispectral image. Indeed, the movements of the observed organ coupled with those of the endoscope, particularly when it is a stomach, are likely to cause a displacement in space of the observed characteristic element, and thus, the characteristic element has a different position and shape in each image of the multispectral image.

For illustration purposes, the FIG. 2 represents an example of non-rigid interband registration of a point A in a multispectral image. The FIG. 2 shows that before re-alignment, one of the points A (corresponding to a characteristic element) is not aligned with the other corresponding points A, despite the alignment of the different images constituting the multispectral image. This type of shift in the positioning of point A can in particular be the result of a movement of the internal wall of the organ observed. In order to take this movement into account, the unaligned point A is recentered so as to be aligned with the corresponding points A.

This type of registration can be achieved by a geometric approach. The latter is based on the extraction from each of the images of geometric primitives which are paired in order to determine the transformation between the images two by two, for example in the form of a transformation matrix. It can also be carried out by a calculation providing an optical flow between the images two by two in order to determine a displacement vector field at each pixel.

This step results in a realigned multispectral image in which each pixel corresponds to a vector with n components and preferably corresponding to the number of spectral bands used. Each of the components then corresponds to one of the spectral bands.

Thus, each pixel of a multispectral image presents n reflectance values for the same point of the observed tissue. However, although these values partly integrate the reflectance of the observed tissue, they also depend on the characteristics of the lighting device used and the sensitivity of the endoscope camera.

During a step 13, a spectral reconstruction of a reflectance cube from the obtained realigned multispectral image is implemented. More precisely, a reflectance cube is obtained in which each pixel corresponds to a vector reflecting only the reflectance of an area of the observed tissue or organ.

In this description, reflectance is defined by the quantity of energy re-emitted relative to the quantity of energy received, and this for each of the wavelengths covered by the spectral bands.

This vector corresponds to the reflectance spectrum of the tissue for the pixel in question over a given wide wavelength range, i.e. between 380-780 nm and 780-1000 nm. Each of its values corresponds to the reflectance for a narrow band of the wavelength range which can be from a few nanometers to several tens of nanometers.

This spectral reconstruction can be carried out from one or more multispectral images as described previously, according to one of the embodiments presented below.

According to a first embodiment of the method for acquiring data by endoscopy from biological tissues represented by the FIG. 3 , spectral reconstruction includes processing of the multispectral image by a deep neural network.

The deep neural network is trained on a large amount of data so that the neural network generates reflectance spectra from multispectral images. The amount of data is preferably at least a few hundred.

Training a deep neural network involves providing it with a set of data pairs. Each pair is composed of a vector similar to those present in the multispectral images (input) and a vector similar to those present in the reflectance cubes (output) and representing the theoretical reflectance spectrum corresponding to the input data.

Alternatively, or in addition, the deep neural network can be pre-trained using pairs of data from healthy tissues and pathological or abnormal tissues, whose reflectance spectrum comes from an additional device.

According to a second embodiment of the method of acquiring data by endoscopy from biological tissues represented by the FIG. 4 , spectral reconstruction includes a registration of the multispectral images in order to obtain a corresponding multispectral image mosaic, and a processing of the mosaic by a deep neural network.

The spectral reconstruction in this second embodiment requires the use of a plurality of multispectral images, which can be obtained successively over time. Preferably, the images forming the different multispectral images are obtained by the same spectral bands.

The multispectral image mosaic is processed by a new neural network. The neural network has been previously trained to provide a reflectance cube (output) from a multispectral image mosaic (input).

As described previously, the deep neural network can be pre-trained using pairs of data from healthy tissues and pathological or abnormal tissues, whose reflectance spectrum comes from an additional device.

There FIG. 5 represents a variant of the second embodiment of the method previously presented. According to this variant, the spectral reconstruction comprises the registration of the multispectral images so as to obtain a corresponding mosaic of multispectral images, and a processing of the mosaic by a light-tissue interaction model.

The light-tissue interaction model is an analytical model described by equations. It is obtained after characterizing each of the tissues from their optical properties such as light absorption and scattering.

In particular, the equations of the light-tissue interaction model include characteristic values for each tissue.

For example, it is possible to determine the amount of hemoglobin present in an area of the observed tissue using specific spectral bands of specific wavelengths. This information regarding the amount of hemoglobin can provide an indication of the presence of an anomaly, a cancerous or precancerous lesion.

The propagation of light in a tissue layer can be modeled via two optical processes: scattering (defined by its reduced diffusion coefficient µ' _s ) and absorption (defined by its coefficient µ _a ).

With this model, the reflectance R can be described according to the following formula:

R0 is based on the theory of light transport (scattering theory or Monte Carlo simulations). The absorption spectra of hemoglobin are shown in the FIG. 6 .

Preferably, the spectral reconstruction according to the first or second embodiment results in a reflectance cube of spatial size identical to the acquired images, but each pixel of which corresponds to a spectral vector of dimension greater than the number of initial bands.

It is also possible to obtain a reflectance cube of a spatial size smaller than the acquired images. Indeed, the captured images can represent the same area with a more or less significant shift, and consequently one or more parts of an image may be absent from the other images intended to form the multispectral image. The registration of the images between them may thus require reducing the sizes of the images so that they have identical sizes between them and represent the same and unique area of biological tissue.

On the contrary, when several multispectral images are acquired, the spatial size of the reflectance cube can be larger than the size of the acquired images. The multispectral images constituting the mosaic comprise a plurality of images covering the observed tissue area, which makes it possible to construct a reflectance cube whose size results from the combination of the different multispectral images.

According to another aspect, an endoscopic imaging system for implementing the method according to the present disclosure comprises an endoscope, one or more housings generating light from an LED source, and a viewing screen.

In particular, endoscopic video may be acquired using an endoscopic imaging system, comprising an endoscope and a viewing screen.

The endoscope typically comprises an operating channel equipped with a light beam (which can be conducted by optical fibers), and an optical or video vision system (such as a camera) positioned at the end present in the organ to be inspected.

The endoscope is intended to analyze human or animal biological tissues, and to be inserted into cavities or hollow organs such as a stomach, a bladder, or even a colon.

The LED source can be introduced into the working channel of the endoscope via an optical fiber. Thus, the LED source can be added to a large number of endoscopes on the market.

Preferably, the LED source used in the endoscopic imaging system generates n spectral bands having different wavelengths and preferably located between 380 and 1000 nanometers.

Each LED source preferentially generates six spectral bands, with at least four spectral bands covering the visible spectrum and at least one spectral band covering the near infrared spectrum. Thus, for example, it is possible to assemble two boxes to the endoscope so as to have twelve spectral bands covering the visible spectrum and the near infrared.

The box is positioned next to the endoscopic column of the endoscope, or alternatively on the endoscopic column, and allows the programming and emission of light with one or more spectral bands covering defined wavelength ranges. In addition, the box offers the possibility of generating one or more spectral bands with a band change frequency that can be high.

During or at the end of the implementation of the data acquisition method according to the present description, the display screen can display the images acquired by the endoscope, the reflectance cubes obtained, and all the information obtained from the method described above. The display screen can also display results of an artificial intelligence processing of the information from the reflectance cubes obtained.

Such a system then makes it possible to alert the endoscope user to the possible presence of one or more anomalies, particularly early ones, in the biological tissues observed.

Claims (11)

Method for analyzing a multispectral video previously acquired by endoscopy from biological tissues, said multispectral video having been previously acquired using a light generating device associated with an endoscope with camera and comprising:

• a light device capable of generating n successive spectral bands of different wavelengths so as to cover the visible spectrum and the near infrared and generate a multispectral video in which each image corresponds to one of said spectral bands;

said method comprising:

• the cutting of said multispectral video thus obtained, so as to extract images from spectral bands of different wavelengths;
• the registration of said spectral band images to obtain at least one registered multispectral image comprising an image for each spectral band of wavelengths;
• the spectral reconstruction of a reflectance cube, from said registered multispectral image, in which each pixel corresponds to a vector reflecting the reflectance of said area of biological tissue.

Method according to claim 1, wherein the wavelength bands are located between 380 and 1000 nm, preferably between 400 and 850 nm. A method according to any preceding claim, wherein n is six or a multiple of six. The method of claim 3, wherein at least four of said spectral bands have wavelengths spanning the visible spectrum and at least one of said spectral bands has wavelengths spanning the near infrared. The method of claim 4, wherein at least four of said spectral bands have wavelengths spanning the visible spectrum and at least two of said spectral bands have wavelengths spanning the near infrared. Method according to any one of the preceding claims, in which the camera captures 20 to 30 images per second during the acquisition of the multispectral video, and in which only one image out of two is alternately selected for the production of a multispectral image during the cutting of said obtained multispectral video. A method according to any one of claims 1 to 6, wherein the spectral reconstruction of a reflectance cube from said multispectral image comprises processing said multispectral image by a deep neural network. A method according to any one of claims 1 to 6, wherein the spectral reconstruction of a reflectance cube from said recalibrated multispectral images comprises:

• a second recalibration of said recalibrated multispectral images so as to obtain a mosaic of multispectral images;
• processing of said multispectral image mosaic by a deep neural network or by a light-tissue interaction model.

A method according to any one of claims 7 and 8, wherein the deep neural network is pre-trained from reflectance spectra data characteristic of healthy and pathological tissues. A method according to any one of claims 1 to 9, wherein it further comprises analyzing the reflectance cube by artificial intelligence. Endoscopic imaging system characterized in that it comprises:

• an endoscope configured to capture a video stream,
• at least one box generating light from an LED source, and
• a display screen configured to display images acquired by the endoscope and/or all or part of the information contained in reflectance cubes obtained with a data acquisition method as defined according to any one of claims 1 to 10.