EP4000004A1 - Method and system for automatically detecting, locating and identifying objects in a 3d volume - Google Patents

Abstract

The method comprises the following steps: - from a 3D volume in voxels (11) of the complex scene, obtaining k 2D sections (15) in the 3D volume (11); - for each input 2D section (21) obtained in this way, automatically detecting, locating and identifying objects (25) of interest by means of a specialised artificial intelligence method (23) arranged to deliver, as output: - a label corresponding to each object identified in the current input section (k) (21); - a 2D bounding box (24) for each object (25) labelled in this way; - a 2D icon defined by the 2D bounding box (24) extracted in this way; - for each output 2D section (23), semantically segmenting each 2D icon defined by a 2D bounding box (24), and - concatenating the 3D results of all the output 2D sections (23) in order to generate the consolidated labels of the objects of interest (25), generate 3D bounding boxes and generate 3D icons segmented in this way.

Description

DESCRIPTION

Title of the invention: Method and system for the detection, location and automatic identification of objects in a 3D volume

FIELD OF THE INVENTION

[0001] The invention relates to the detection, location and automatic identification of objects in a 3D volume.

[0002] It applies in general to the field of target detection, to the medical field, to the field of microelectronics as well as to similar fields. In particular, it makes it possible to respond to application questions encountered in many situations, such as the automatic detection of small fossils (2-3 microns) in large 3D volumes reconstructed by scanning oil exploration cores, the identification of objects camouflaged in a complex 3D scene; the identification of a benign pigment disorder likely to progress to carcinoma or melanoma from a three-dimensional skin reconstruction; the identification of "carcinogenic anomalies" in OCT (Optical Cohérence Tomography) slices, or the detection, localization and automatic identification of carcinogenic tumors / lacunar areas resulting from three-dimensional reconstructions of tomographic scans or MRI (Magnetic Resonance Imaging) .

BACKGROUND OF THE INVENTION

[0003] Currently, the application queries mentioned above are most often handled by the professional expert (geophysicist, physicist, radiologist, dermatologist, etc.) who identifies the objects of interest in 3D volumes using 'visualization tools such as MIP (Maximum Intensity Projection) for example.

[0004] However, the processing by the business expert is difficult to set up because the mass of data to be processed is very large. In addition, the success rate of the identification by the business expert is limited and rarely exceeds 90%.

SUMMARY OF THE INVENTION

[0005] The aim of the present invention is to improve the situation. [0006] To this end, the present invention provides a method for detecting, locating and identifying objects contained in a complex scene.

[0007] According to a general definition of the invention, the method comprises the following steps:

- from a 3D volume in voxels of the complex scene, obtain k 2D sections in the 3D volume;

- for each 2D input cut thus obtained, automatically detect, locate, and identify objects of interest by a specialized artificial intelligence method designed to deliver:

- a label corresponding to each object identified in the current 2D input section;

- a bounding box of each object thus labeled;

- a 2D icon defined by the 2D bounding box thus extracted;

- for each output 2D cut, semantically segment each 2D icon defined by a 2D bounding box, and

- concatenate the results of all the k 2D output sections in order to generate the consolidated labels of the objects of interest, to generate 3D bounding boxes as well as to generate 3D icons thus segmented.

[0008] Thus the invention makes it possible to significantly improve the quality of the detection, localization, and identification of objects in a complex scene thanks, from the 3D volume of the scene to be processed, to obtaining k 2D sections in each from which we proceed to a detection, localization and identification of the objects to be processed by artificial intelligence and semantic segmentation and the concatenation of the results in 3D. This results in the ability to detect, locate and identify objects automatically with very high precision even when the objects are hidden from each other.

[0009] According to preferred embodiments, the invention comprises one or more of the following characteristics which can be used separately or in partial combination with one another or in total combination with one another: - the complex scene is previously transformed by 3D imaging in volume the method further comprises a step of indexing the labels for all the objects of interest in the complex scene;

- the resolution of the process depends on the number and the nature of the 2D sections;

- the resolution of the process depends on the size of the 2D bounding boxes;

- the resolution of the process depends on the size of the 3D bounding boxes.

Advantageously, the method comprises, in order to concatenate in 3D the results of all the k 2D output sections, for an object of interest among said objects of interest, the following steps: define, for each output 2D section, a reference mark local three-dimensional, one of the dimensions of which is perpendicular to the plane defined by the 2D section, and associate said reference with said 2D section; identify, in the output 2D sections, subsets or slices of the object of interest; transform each identified subset or slice of the object of interest by changing the coordinate system, from the local three-dimensional frame of the 2D section to which it belongs to a predetermined absolute Cartesian frame of reference; concatenate the transformed subsets or slices into a 3D icon.

[001 1] The invention also relates to a system for implementing the method defined above.

[0012] The invention further relates to a computer program comprising program instructions for the execution of a method as defined above, when said program is executed on a computer.

[0013] Other features and advantages of the invention will become apparent on reading the following description of a preferred embodiment of the invention, given by way of example and with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 schematically illustrates the main steps of the process according to the invention;

[0015] Figure 1A is an index table of classes of objects of interest;

FIG. 2 schematically represents the steps for obtaining 2D sections; [0017] FIG. 3 diagrammatically represents the steps of detection, location, and automatic identification of objects of interest in each 2D section by specialized artificial intelligence;

[0018] Figure 4 shows a bounding box of the object detected according to the method according to the invention;

[0019] Figure 5 shows a segmented icon in accordance with the invention;

[0020] Figure 6 shows a 3D volume reconstructed in voxels in accordance with the invention;

[0021] Figure 7 shows principal sections in a 3D volume reconstructed by receptive tomography in accordance with the invention;

[0022] FIG. 8 represents an example of an OCT (Optical Coherence Tomography) section;

[0023] FIG. 9 represents an example of a complex 3D scene containing a camouflaged object reconstructed by reflective tomography from the 2D images;

[0024] Figure 10 shows an example of a 2D section in the 3D scene;

[0025] Figure 11 shows the automatic detection and generation of the bounding box of the object camouflaged in the 2D section of the 3D scene;

[0026] Figure 12 shows the identification of the camouflaged object in the 2D section of the 3D scene; and

[0027] Figure 13 shows the generation of the bounding box and the identification of the object in the 3D scene.

DETAILED DESCRIPTION OF THE INVENTION

The invention relates to the detection, location and automatic identification of objects in 3D three-dimensional imagery forming a 3D three-dimensional volume in voxels (pixel volumetry).

[0029] For example and in a nonlimiting manner; 3D imagery corresponds to a complex scene in which objects can hide from each other as illustrated in figure 9.

In practice, the three-dimensional volume can be obtained by means of a reconstruction process by transmission or by fluorescence (Optical Projection Tomography, nuclear imaging or X-Ray Computed Tomography) or by reflection (reflection of a laser wave or by solar reflection in the case of the visible band (between 0.4 pm and 0.7 pm) or near infrared (between 0.7 pm and 1 pm or SWIR (Small Wave InfraRed between 1 pm and 3 pm) or taking into account the thermal emission of the object (thermal imaging between 3 pm and 5 pm and between 8 pm and 12 pm), this process three-dimensional reconstruction is described in the patent “Optronic system and method for producing three-dimensional images dedicated to identification” (US8836762B2, EP2333481 B1).

All of the voxels resulting from a three-dimensional reconstruction with the associated intensity are used, this reconstruction having preferably been obtained by reflection.

[0032] First with reference to FIG. 1A, the classes of objects of interest are indexed.

A table TAB of correspondence "Index Class x Label Class" for all the classes of objects of interest is thus created. For example, at the end of the indexing, the following elements are obtained: Class (n) (Index (n), Label (n)}, n = [1, 2, .., N}, N being the number of object classes of interest.

[0034] For example, the index (n) is at the value "n", and the index (background) is at the value "0".

The detection, location and identification method according to the invention comprises the following general steps described with reference to Figure 1.

In a first step referenced 10, k 2D sections are made in the reconstructed 3D volume. 3D volume (Section (k)}, k = [1, 2, .., K}, K being the number of 2D sections made.

[0037] With reference to step 20, for each input 2D section thus obtained, one proceeds to an automatic detection, location, and identification of the objects of interest by a specialized AI artificial intelligence method.

The AI artificial intelligence method outputs the following elements: - Detection of Objects (k, m) in Section (k), m = {1, 2, .., M (k)}, M (k) being the number of objects detected in Section (k);

- Generation of the Label (k, m) corresponding to each Object (k, m) identified in the Section (k);

- Generation of 2D bounding boxes, also called Boundingbox2D (k, m) of each Object (k, m) in Section (k);

- Extraction, from Section (k), of lcones2D (k, m) defined by 2D bounding boxes Boundingbox2D (k, m).

Thus we obtain the following elements: Cut (k) {Object (k, m), Label (k, m), Boundingbox2D (k, m), lcone2D (k, m)}.

[0039] For example, the AI Artificial Intelligence method is based on deep learning, also called Deep Learning, of the "Faster R-CNN (Regions with Convolutional Neural Network features) object classification" type.

Then, the method applies a semantic segmentation of each lcone2D defined by a bounding box Boundingbox2D.

In practice, the generation of a segmented lcone2D (k, m) of the same size as the lcone2D (k, m) has for each pixel that is the value of the index (k, m) of the Object ( k, m) identified in Section (k), or the value of the index (background). At the output we therefore have lcone2D (k, m) lcone2Dsegmented (k, m).

For example, the semantic segmentation is performed by Deep Learning, for example an R-CNN Mask (Regions with Convolutional Neural Network) designated for the semantic segmentation of the images.

Referring to step 30, we finally proceed to the 3D concatenation of the results of all the 2D sections.

In a set of embodiments of the invention, the 3D concatenation of the results of the 2D sections is carried out by the following steps: Definition of a local three-dimensional coordinate system associated with each 2D section, including one of dimension is perpendicular to the plane defined by the 2D section, o Identification of 2D sections in which subsets or sections of the object of interest have been identified, o Mathematical transformation (translation and / or rotation) of all the local three-dimensional reference frames of the subsets or slices retained in a determined absolute Cartesian three-dimensional reference frame.

[0045] This allows the reconstruction of the three-dimensional object by ensuring continuity at the limits of the subassemblies or slices of the object.

At the output, we then have the following elements:

Concatenation of Labels (k, m) Generation of consolidated Labels (n)

Concatenation of Boundingbox2D (k, m) Generation of Boundingbox3D (n)

Concatenation of segmented 2D (k, m) lcones Generation of segmented 3D (n) lcones

(Object (n), Label (n), Boundingbox3D (n), segmented lcone3D (k, n)}, (n) belonging to [1, 2, .., N}, N being the number of object classes to interest.

The method according to the invention presents several details.

The first precision, also called resolution relates to the number and angle of 2D sections, for example 2D sections belong to the group formed by main sections, horizontal sections, vertical sections, oblique sections. The higher the number of 2D slices, the better the detection resolution. In addition, 2D sections at different angles can provide better detection results and will be used in the 3D concatenation of results which will be described in more detail below.

The second precision concerns the 2D bounding boxes, Boundingbox2D = [(x1, x2), (y1, y2)]. The smaller the size of 2D bounding boxes, the better the detection resolution.

The third precision concerns the 3D bounding boxes, Boundingbox3D = [(x1, x2), (y1, y2), (z1, z2)]. The smaller the size of the 3D bounding boxes, the better the detection resolution.

Referring to Figure 2, there is shown the modules relating to the realization of 2D sections.

As seen above, the choice and the number of 2D sections will act on the resolution of the detection. From the 3D volume 1 1 (reconstructed in voxels), the cutting module 12 generates 2D sections 15 (in pixels) in response to the command from the choice module 13. The 2D sections 15 (in pixels) are managed and indexed by the management module 14 in accordance with the indexing table TAB (FIG. 1A).

With reference to FIG. 3, the modules relating to the artificial intelligence method IA 22 applied to each input 2D section 21 thus generated and indexed have been shown.

The output 2D section 23 generated by the IA method 22 comprises a 2D bounding box 24 surrounding an object of interest 25.

Referring to Figure 4, the size of the 2D bounding box 24 surrounding the object 25 is defined by its coordinates on the abscissa X (x1 and x2) and on the ordinate Y (y1 and y2).

Referring to Figure 5, there is shown a 2D icon 50 semantically segmented in accordance with the invention. For example, object 25 is indexed with the value "1" while the background is indexed with the value "0".

Referring to Figure 6, there is shown a 3D volume reconstructed in voxels according to the invention in which the object of interest 25 has the index "1" while another object of interest has the 'index "2"; in an index background volume "0".

According to Figure 7, there is shown 2D main sections in a 3D volume reconstructed by reflective tomography in accordance with the invention, here along XY, XZ and ZY.

[0060] According to FIG. 8, an example of an OCT (Optical Coherence Tomography) section has been shown; in which a gap area is to be identified.

[0061] According to Figure 9, there is shown an example of a complex 3D scene containing a camouflaged object reconstructed by reflective tomography from the 2D images.

[0062] For example, the complex scene includes a vehicle camouflaged in the bushes. The 2D shooting is of the air ground type with 2D images of 415x693 pixels. According to Figure 10, there is shown an example of a 2D section (YZ section) in the 3D scene of Figure 9.

According to Figure 1 1, there is shown the automatic detection and generation of the bounding box of the object camouflaged in the 2D section of the 3D scene illustrated in Figures 9 and 10.

[0065] According to Figure 12, the identification of the camouflaged object (index 1) is shown in the 2D section of the 3D scene with the coordinates of the 2D bounding box.

[0066] According to Figure 13, the generation of the 3D bounding box (with its coordinates) and the identification of the object in the 3D scene have been shown.

[0067] The fields of application of the invention are wide, covering the detection, classification, recognition and identification of objects of interest.

Claims

1. A method of detecting, locating and identifying objects (25) contained in a complex scene, characterized in that it comprises the following steps: - from a 3D voxel volume (1 1) of the complex scene, obtain k 2D sections (15) in the 3D volume (1 1); - for each 2D input cut (21) thus obtained, automatically detect, locate, and identify objects (25) of interest by a specialized artificial intelligence method (23) and arranged to deliver as output: - a label corresponding to each object identified in the current entry cut (21); - a 2D bounding box (24) of each object (25) thus labeled; - a 2D icon defined by the 2D bounding box (24) thus extracted; - for each output 2D cut (23), semantically segment each 2D icon defined by a 2D bounding box (24), and - concatenate in 3D the results of all the k 2D output sections (23) in order to generate the consolidated labels of the objects of interest (25), to generate bounding boxes in 3D as well as to generate 3D icons thus segmented. 2. Method according to claim 1, characterized in that the complex scene is previously transformed by 3D imaging into a 3D volume. 3. Method according to claim 1, characterized in that it further comprises a step of indexing the labels for all the objects of interest in the complex scene. 4. Method according to any one of claims 1 to 3, characterized in that the resolution of the method depends on the number and the nature of the 2D sections. 5. Method according to any one of claims 1 to 3, characterized in that the resolution of the method depends on the size of the 2D bounding boxes. 6. Method according to any one of claims 1 to 3, characterized in that the resolution of the method depends on the size of the 3D bounding boxes. 7. Method according to any one of claims 1 to 6, wherein the AI Artificial Intelligence method is based on deep learning called again Deep Learnining of the Faster R-CNN type (Regions with Convolutional Neural Network). 8. Method according to any one of claims 1 to 7, in which the semantic segmentation is carried out by a convolutional neural network of Mask R-CNN (Regions with Convolutional Neural Network) type. 9. Method according to one of claims 1 to 8, comprising, in order to concatenate in 3D the results of all the k output 2D slices, for an object of interest among said objects of interest, the following steps: - define, for each output 2D section, a local three-dimensional frame of reference, one of the dimensions of which is perpendicular to the plane defined by the 2D section, and associate said frame with said 2D section; - identify, in the output 2D sections, subsets or sections of the object of interest; - transform each identified subset or slice of the object of interest by changing the coordinate system, from the local three-dimensional frame of the 2D section to which it belongs to a predetermined absolute Cartesian frame of reference; - concatenate the sub-assemblies or slices transformed into a 3D icon. 10. System for the detection, location and identification of objects (25) contained in a complex scene, characterized in that it comprises: - from a 3D voxel volume (1 1) of the complex scene, a module (12) intended to obtain k 2D sections (15) in the 3D volume (1 1); - an artificial intelligence module (23) capable, for each input 2D section (21) thus obtained, to automatically detect, locate, and identify objects (25) of interest and arranged to deliver at the output: - a label corresponding to each object identified in the current entry cut (21); - a 2D bounding box (24) of each object (25) (k, m) thus labeled; - a 2D icon defined by the 2D bounding box (24) thus extracted; - a semantic segmentation module able, for each output 2D cut (23), to semantically segment each 2D icon defined by a 2D bounding box (24), and - a processing module capable of concatenating the 3D results of all the k 2D output slices (23) in order to generate the consolidated labels of the objects of interest, to generate 3D bounding boxes as well as to generate 3D icons thus segmented . 1 1. A computer product program comprising program instructions for carrying out a method according to one of claims 1 to 9, when said program is executed on a computer.