WO2020017213A1 - Endoscope image recognition apparatus, endoscope image learning apparatus, endoscope image learning method and program - Google Patents

Abstract

Provided are an endoscope image recognition apparatus, an endoscope image learning apparatus, and an endoscope image learning method and program, which appropriately perform learning of a learning model for image recognition which recognizes endoscope images of light with a specific wavelength band. This endoscope image learning apparatus comprises an image generation unit which uses a generation model to generate an endoscope image of light having a first wavelength band; and a machine learning unit which uses the endoscope image generated by the image generation unit to perform learning of a learning model for image recognition. The endoscope image learning apparatus is used to appropriately perform learning of a learning model for image recognition which recognizes an endoscope image of light having a first wavelength band, with a small number of data.

Description

Endoscope image recognition device, endoscope image learning device, endoscope image learning method and program

The present invention relates to an endoscope image recognition device, an endoscope image learning device, an endoscope image learning method and a program, and more particularly to a technique for learning a learning model for image recognition.

In endoscopy, not only ordinary light, but also observation using special light emitted using a special wavelength balance is expected to improve the accuracy of discrimination and detection of lesions and reduce oversight of lesions. I have.

On the other hand, there is known a technique for recognizing a lesion on an endoscopic image by a computer. Particularly, by using an image recognizer created by machine learning, highly accurate diagnosis can be performed. In the image recognition technology using such an image recognizer, improvement in recognition accuracy of an endoscope image using special light is expected.

精度 The accuracy of an image recognizer created by machine learning largely depends on the image data used for learning. Therefore, it is necessary to collect a large amount of endoscope images captured by various light sources. However, at the actual site, inspection is usually performed using white light, and special light is selectively used depending on the purpose of the inspection or the like. Therefore, it is difficult to collect images captured using special light. Was. Therefore, in order to improve the recognition accuracy of the special light image, it is necessary to generate a special light image suitable for machine learning and use it as learning data.

Patent Literature 1 discloses that in generating a spectral estimation image of an endoscope first observation image and an endoscope second observation image, the endoscope first observation image has a spectral estimation image corresponding to a predetermined wavelength set. Generate one spectral estimation image, set an approximate wavelength set that approximately generates the spectrum of the irradiation light from the first light source for the second observation image of the endoscope, and according to the set approximate wavelength set. A technique for generating a second spectral estimation image has been disclosed.

Patent Document 2 discloses a technique for performing affine deformation on a part image to create a deformed image.

Patent Document 3 further describes a technique of learning a plurality of combinations of a low-resolution image and a high-resolution image, performing inference based on the learning, and performing resolution conversion.

JP 2011-194082 A JP 2012-043151 A JP 2018-005841 A

However, the techniques described in Patent Documents 1 to 3 aim at improving the visibility of an image presented to a user. The image conversion methods described in Patent Literatures 1 and 2 are so-called manual conversion methods, and faithfully reproduce complicated conversion such as conversion of a white light image into a wavelength band light such as a special light image. Not easy. Therefore, an image suitable for machine learning cannot be generated, and a learning model for image recognition cannot be appropriately learned.

The present invention has been made in view of such circumstances, and an endoscope image recognition device, an endoscope image learning device, and an endoscope that appropriately learn a learning model for image recognition that recognizes an endoscope image. It is an object to provide a mirror image learning method and a program.

In order to achieve the above object, one mode of the endoscope image learning device includes an image generation unit that generates an endoscope image of the first wavelength band light using a generation model, and an endoscope generated by the image generation unit. An endoscope image learning apparatus comprising: a machine learning unit that learns a learning model for image recognition using a mirror image.

According to this aspect, the endoscope image of the first wavelength band light is generated by the generation model, and the learning model for image recognition is learned using the generated endoscope image. Learning of a learning model for image recognition for recognizing an endoscope image can be appropriately performed.

The image generation unit receives a second endoscope image captured with a second wavelength band light different from the first wavelength band light as an input, and converts the second endoscope image into a first wavelength band light. Preferably, it is converted to a first endoscopic image. This makes it possible to appropriately generate the first endoscope image of the first wavelength band light.

１ The first wavelength band light is preferably special light, and the second wavelength band light is preferably white light. This makes it possible to generate a special light image having a small number of data from a white light image having a large number of data.

正 It is preferable that the first endoscope image and the second endoscope image have the same correct answer data indicating the correct answer of the image recognition. As a result, it is possible to use the answer data without creating new answer data.

The correct answer data indicating the correct answer of the image recognition of the first endoscope image may be correct answer data generated using the correct answer data indicating the correct answer of the second endoscope image. Even using such correct answer data, learning of the learning model can be appropriately performed.

It is preferable that the learning model detects a lesion area from the input endoscopic image. Thereby, a lesion area can be appropriately detected from the input endoscope image.

It is preferable that the learning model classifies the lesion area as either benign or malignant. This makes it possible to appropriately classify whether the lesion area is benign or malignant.

The generation model is preferably a generation model obtained by learning using an endoscope image captured with the first wavelength band light and the second endoscope image. Thereby, an endoscope image of the first wavelength band light can be appropriately generated.

Preferably, the generation model is learned using a convolutional neural network. Thereby, an endoscope image of the first wavelength band light can be appropriately generated.

The generation model is preferably GAN (Generative Adversarial Networks). Thereby, an endoscope image of the first wavelength band light can be appropriately generated.

It is preferable that the machine learning unit learns a learning model using a convolutional neural network. Thereby, the learning of the learning model for image recognition can be appropriately performed.

One embodiment of the endoscope image recognition device for achieving the above object is an image acquisition unit that acquires an endoscope image photographed with the first wavelength band light, and the endoscope image learning device described above. An endoscope image recognition device including: an image recognition unit that performs image recognition of an endoscope image acquired using a learned learning model.

According to this aspect, the endoscope image captured by the first wavelength band light is acquired, and the image recognition of the endoscopic image acquired by using the learning model is performed. Image recognition can be appropriately performed.

In order to achieve the above object, one embodiment of an endoscope image learning method includes an image generation step of generating an endoscope image of a first wavelength band light using a generation model, and an endoscope generated in the image generation step. And a machine learning step of learning a learning model for image recognition using a mirror image.

According to this aspect, the endoscope image of the first wavelength band light is generated by the generation model, and the learning model for image recognition is learned using the generated endoscope image. Learning of a learning model for image recognition for recognizing an endoscope image can be appropriately performed.

One aspect of a program for causing a computer to execute the above-described object is a program for causing a computer to execute the above-described endoscopic image learning method.

According to this aspect, the endoscope image of the first wavelength band light is generated by the generation model, and the learning model for image recognition is learned using the generated endoscope image. Learning of a learning model for image recognition for recognizing an endoscope image can be appropriately performed.

According to the present invention, a learning model for image recognition for recognizing an endoscopic image can be appropriately learned.

FIG. 1 is a block diagram illustrating an example of a hardware configuration of the endoscope image learning device. FIG. 2 is a functional block diagram illustrating main functions of the endoscope image learning device. FIG. 3 is a functional block diagram illustrating main functions of the machine learning unit. FIG. 4 is a functional block diagram illustrating main functions of the image generation unit. FIG. 5 is a flowchart illustrating an example of an endoscope image learning method. FIG. 6 is a functional block diagram illustrating main functions of the endoscope image learning device. FIG. 7 is a functional block diagram illustrating main functions of the image generation unit. FIG. 8 is an external view showing the endoscope system. FIG. 9 is a block diagram illustrating functions of the endoscope system. FIG. 10 is a graph showing the intensity distribution of the light L1 and the light L2.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<Hardware configuration of medical image learning device>
FIG. 1 is a block diagram showing an example of a hardware configuration of an endoscope image learning device according to the present invention.

The endoscope image learning device 10 is configured by a personal computer or a workstation. The endoscope image learning device 10 mainly includes a communication unit 12, a first database 14, a second database 16, an operation unit 18, a CPU (Central Processing Unit) 20, a RAM (Random Access Memory) 22, A ROM (Read Only Memory) 24 and a display unit 26 are provided.

The communication unit 12 is an interface that performs communication processing with an external device by wire or wirelessly and exchanges information with the external device.

The first database 14 is learning first data composed of a normal light image group including a plurality of normal light images 14A photographed with normal light, and correct data 14B indicating a correct image recognition result of each normal light image 14A. A large-capacity storage device for storing sets. The second database 16 is learning second data composed of a special light image group including a plurality of special light images 16A captured with special light and correct data 16B indicating a correct image recognition result of each special light image 16A. A large-capacity storage device for storing sets.

Here, the normal light image and the special light image are color images photographed by endoscope devices (not shown) under different light sources.

(4) The normal light is light (white light) in which light in all wavelength bands of visible light is almost uniformly mixed, and the normal light image is used for normal observation. Therefore, a relatively large number of ordinary light image groups can be collected.

On the other hand, the special light is light of various wavelength bands according to the observation purpose, which is light of one specific wavelength band or light of a plurality of specific wavelength bands, and is a band narrower than the white wavelength band. And used for narrow band observation (NBI (registered trademark), FICE (Flexible spectral imaging color enhancement, registered trademark)).

The first example of the specific wavelength band is, for example, a blue band or a green band in a visible region. The wavelength band of the first example includes a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and the light of the first example has a peak wavelength in the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. .

第 A second example of the specific wavelength band is, for example, a red band in a visible region. The wavelength band of the second example includes a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light of the second example has a peak wavelength in the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm. .

The third example of the specific wavelength band includes a wavelength band in which the absorption coefficient differs between oxyhemoglobin and reduced hemoglobin, and the light of the third example has a peak wavelength in a wavelength band in which the absorption coefficient differs between oxyhemoglobin and reduced hemoglobin. Having. The wavelength band of the third example includes 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example includes the above 400 ± 10 nm, 440 ± 10 nm, 470 nm. It has a peak wavelength in a wavelength band of ± 10 nm, or 600 nm to 750 nm.

The fourth example of the specific wavelength band is a wavelength band (390 nm to 470 nm) of excitation light used for observing fluorescence emitted from a fluorescent substance in a living body (fluorescence observation) and for exciting this fluorescent substance.

The fifth example of the specific wavelength band is a wavelength band of infrared light. The wavelength band of the fifth example includes a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and the light of the fifth example has a peak wavelength in a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.

The special light image captured under such special light having a specific wavelength band is difficult to see, so it is used only for an observation purpose such as a desire to observe a surface structure, and the number of data is not large.

In this example, the first data set of the normal light image group stored in the first database 14 is prepared more than the second data set of the special light image group stored in the second database 16. And

In the first database 14 and the second database 16, the correct answer data 14B and 16B stored in association with each normal light image 14A and each special light image 16A are reflected in the normal light image and the special light image, for example. The type of the lesion, the data indicating the location of the lesion, identification information unique to the case, and the like can be considered. The classification of lesions includes two classifications, neoplastic and non-neoplastic, and NICE classification. The data indicating the position of the lesion may be rectangular information surrounding the lesion or mask data that covers the lesion.

The first database 14 and the second database 16 may be the same storage device. Further, the first database 14 and the second database 16 may be provided outside the endoscope image learning device 10 instead of being provided inside the endoscope image learning device 10. In this case, a data set for learning can be acquired from an external database via the communication unit 12.

The operation unit 18 is an input interface that receives various operation inputs to the endoscope image learning device 10. As the operation unit 18, a keyboard, a mouse, or the like that is connected to the computer by wire or wirelessly is used.

The CPU 20 reads various programs stored in the ROM 24 or a hard disk device (not shown) and executes various processes. The RAM 22 is used as a work area of the CPU 20. The RAM 22 is used as a storage unit for temporarily storing the read program and various data.

The display unit 26 is an output interface on which necessary information of the endoscopic image learning device 10 is displayed. As the display unit 26, various monitors such as a liquid crystal monitor connectable to a computer are used.

In the endoscope image learning device 10, the CPU 20 reads the endoscope image learning program stored in the ROM 24 or the hard disk device in response to an instruction input from the operation unit 18, and executes the endoscope image learning program. Thereby, as described later, generation of an endoscope image using the generation model and learning of a learning model using the endoscope image are performed.

<First embodiment>
(Endoscope image learning device)
FIG. 2 is a functional block diagram illustrating main functions of the endoscope image learning device 10 according to the first embodiment. The endoscope image learning device 10 includes a machine learning unit 30 and an image generation unit 40.

The machine learning unit 30 learns using a large number of learning images. The machine learning unit 30 uses the data set of the normal light image 14A and the correct data 14B stored in the first database 14 and the data set of the special light image 16A and the correct data 16B stored in the second database 16. By learning, a learning model for image recognition is generated. The machine learning unit 30 constructs a convolutional neural network (CNN: Convolution Neural Network) which is one of the learning models.

FIG. 3 is a functional block diagram showing main functions of the machine learning unit 30. The machine learning unit 30 mainly includes a CNN 32, an error calculating unit 34, and a parameter updating unit 36.

The CNN 32 is a recognizer for recognizing the type of lesion shown in the endoscope image. The CNN 32 has a plurality of layer structures and holds a plurality of weight parameters. The CNN 32 may change from an unlearned model to a learned model by updating the weight parameter from an initial value to an optimal value.

This CNN 32 includes an input layer 32A, an intermediate layer 32B, and an output layer 32C. Each of the input layer 32A, the intermediate layer 32B, and the output layer 32C has a structure in which a plurality of “nodes” are connected by “edges”.

The normal light image 14A and the special light image 16A to be learned are input to the input layer 32A.

The intermediate layer 32B is a layer for extracting features from an image input from the input layer. The intermediate layer 32B has a plurality of sets each including a convolutional layer and a pooling layer, and a fully connected layer. The convolution layer performs a convolution operation using a filter on a nearby node in the previous layer to obtain a feature map. The pooling layer reduces the feature map output from the convolutional layer to a new feature map. The full connection layer connects all the nodes of the immediately preceding layer (here, the pooling layer). The convolutional layer plays a role in extracting features such as edge extraction from an image, and the pooling layer plays a role in providing robustness so that the extracted features are not affected by translation or the like. The intermediate layer 32B is not limited to the case where the convolution layer and the pooling layer are set as one set, but also includes the case where the convolution layer is continuous and the normalization layer.

The output layer 32C is a layer that outputs a recognition result for detecting a lesion (an example of a lesion region) in an endoscopic image based on the features extracted by the intermediate layer 32B. Further, a recognition result for classifying a detected lesion as either benign or malignant may be output, or a recognition result for classifying the type of lesion may be output.

When classifying the type of lesion, the learned CNN 32 classifies the endoscopic images into, for example, three categories of “neoplastic”, “non-neoplastic”, and “other”, and the recognition result is “neoplastic”. Output as three scores corresponding to “non-neoplastic” and “other”. The sum of the three scores is 100%.

初期 An arbitrary initial value is set for the filter coefficient applied to each convolutional layer of the CNN 32 before learning, the offset value, and the weight of connection with the next layer in the fully connected layer.

The error calculator 34 obtains the recognition result output from the output layer 32C of the CNN 32 and the correct answer data for the input image, and calculates an error between the two. As a method of calculating the error, for example, soft max cross entropy or least square error (MSE: MeanMSquared Error) can be considered.

The parameter updating unit 36 adjusts the weight parameter of the CNN 32 by the error back propagation method based on the error calculated by the error calculating unit 34.

調整 The parameter adjustment processing is repeatedly performed, and learning is repeatedly performed until the difference between the output of the CNN 32 and the correct data becomes small.

The machine learning unit 30 optimizes each parameter of the CNN 32 using all data sets of the normal light image group stored in the first database 14 and the special light image group stored in the second database 16. That is, the machine learning unit 30 learns a learning model using a convolutional neural network.

The learning by the machine learning unit 30 may use a mini-batch method in which a fixed number of data sets are extracted from all the data sets, a batch processing of learning is performed by the extracted data sets, and this is repeated.

The machine learning unit 30 may not use the correct answer data depending on the recognition processing to be realized. Alternatively, the machine learning unit 30 may extract a feature using a previously designed algorithm such as edge extraction or the like, and may learn using a support vector machine or the like using the information.

The image generation unit 40 generates a special light image using a generation model. In the generation model, a conditional distribution p (x | C) of data x for a certain class C is modeled. Here, the class corresponds to the type of the light source, and the data corresponds to the image. Using this generation model, pseudo data belonging to a certain class can be generated.

Examples of the generation model include a variational self-encoder (VAE: Variational Auto Encoder) and a hostile generation network (GAN: Generative Adversarial Networks).

画像 The image generation unit 40 of this example constructs a GAN as a generation model. GAN is a generation model formed by learning “Generator” for creating data and “Discriminator” for identifying data alternately.

FIG. 4 is a functional block diagram showing main functions of the image generation unit 40. The image generation unit 40 mainly includes a generator 42 and a classifier 44.

The generator 42 corresponds to “Generator”, and the discriminator 44 corresponds to “Discriminator”. The generator 42 receives the noise z as an input and generates an image x1 that is a special light generation image. The classifier 44 receives as input the generated image x1 and the special light image 16A prepared as the learning image x0, and classifies whether the input image is generated data or learning data.

The generator 42 and the classifier 44 are learned using a convolutional neural network.

The learning of the discriminator 44 is performed in the same flow as the machine learning unit 30. Specifically, an image is input, and classification is performed to determine whether the input image is an image generated by the generator 42 or a real endoscope image.

On the other hand, the learning of the generator 42 receives the noise z and converts the data to the form of an image in such a way as to follow a general CNN in reverse. The learning of the generator 42 uses an upsampling method such as “fractionally-strided convolution” for increasing the size of an image.

As the learning progresses, the generator 42 and the classifier 44 increase the accuracy, and the generator 42 is not recognized by the classifier 44 and is a special light generation image (first wavelength band light of the first wavelength band light) closer to a real special light image. An example of an endoscope image) can be generated. The generator 42 obtained by this learning is a generation model.

(Endoscope image learning method)
FIG. 5 is a flowchart illustrating an example of an endoscope image learning method by the endoscope image learning device 10. The endoscope image learning method includes an image generation step S1 and a machine learning step S2.

In the image generation step S1, a special light generation image equivalent to the special light image is generated by the generation model of the image generation unit 40. For example, 100 special light images are generated.

In the machine learning step S2, the learning model of the machine learning unit 30 is learned using the special light image generated in the image generation step S1. Here, learning is performed using 100 special light images.

As described above, by generating the special light image by the generation model, the special light image with a small number of data can be inflated and collected. Further, by learning a learning model for endoscopic image recognition using the generated special light image, it is possible to appropriately learn a special light image having a small number of data. Therefore, the performance of image recognition using a special light image can be improved.

<Second embodiment>
(Endoscope image learning device)
The hardware configuration of the endoscopic image learning device according to the second embodiment is the same as the block diagram shown in FIG.

FIG. 6 is a functional block diagram showing main functions of the endoscope image learning device 50 according to the second embodiment. Parts common to those in the block diagram shown in FIG. 2 are denoted by the same reference numerals, and detailed description thereof will be omitted.

The endoscope image learning device 50 includes an image generation unit 60. The image generation unit 60 is connected to the first database 14 and the second database 16.

In the second embodiment, a model obtained by applying a generated model to image conversion is used as an image generating unit. The image generation unit 60 constructs CycleGAN as a generation model. In CycleGAN, conversion between a certain class C1 and class C2 is obtained. In CycleGAN, a network in which "Generator" and "Discriminator" are prepared for two classes is used. The “Generator” of the class C1 learns a distribution for converting an image of the class C2 into an image of the class C1. The class C1 “Discriminator” distinguishes between the real class C1 image and the converted image. The class C2 “Generator” and “Discriminator” do the opposite.

CNN is also the mainstream in learning CycleGAN. Learning of “Discriminator” is the same as that of a normal GAN, and “Generator” learns an appropriate conversion by using images for input and output.

FIG. 7 is a functional block diagram illustrating main functions of the image generation unit 60. The image generator 60 mainly includes a first generator 62, a first classifier 64, a second generator 66, and a second classifier 68.

The first generator 62 corresponds to “Generator” of class C1, and the first discriminator 64 corresponds to “Discriminator” of class C1. The second generator 66 corresponds to “Generator” of class C2, and the second discriminator 68 corresponds to “Discriminator” of class C2.

The first generator 62 receives the image x02, which is the special light image 16A, and generates an image x11 that is a normal light conversion image obtained by converting the image x02 into a normal light image. The first classifier 64 receives the generated image x11 and the normal light image 14A prepared as the learning image x01, and classifies whether the input image is generated data or learning data.

Further, the second generator 66 receives an image x01 (an example of a second endoscope image captured with the second wavelength band light), which is the normal light image 14A, and converts the image x01 into a special light image. An image x12 (an example of a first endoscope image of the first wavelength band light) that is the converted special light image is generated. The second classifier 68 receives the generated image x12 and the special light image 16A prepared as the learning image x02, and classifies whether the input image is generated data or learning data.

Furthermore, the first generator 62 receives the image x12 which is the special light conversion image generated by the second generator 66 as an input, and generates an image x21 which is a normal light conversion image of the image x12. The first classifier 64 receives the generated image x21 and the learning image x01 as inputs, and classifies whether the input image is generated data or learning data.

Similarly, the second generator 66 receives the image x11 that is the normal light conversion image generated by the first generator 62 as an input, and generates an image x22 that is a special light conversion image of the image x11. The second classifier 68 receives the generated image x22 and the learning image x02 as inputs, and classifies whether the input image is generated data or learning data.

第 The second generator 66 obtained by this learning is the generation model. As described above, according to the second generator 66 obtained by learning using the normal light image 14A and the special light image 16A, the special light image that is not identified by the second classifier 68 is converted from the white light image. It can be said. The endoscope image learning method is the same as in the first embodiment.

According to the generation model of the second embodiment, it is possible to convert only the tint without changing the composition of the normal light image 14A before the conversion. Since a statistically optimal conversion method is learned from a large number of images used for generating the generation model, more natural image generation can be expected.

技術 In the technique described in Patent Document 1, a 3 × 3 matrix operation is performed on pixel values of RGB (Red Green Blue), and the expressive power is relatively small. In the present embodiment, since GAN is used as the generation model, the expressiveness of the conversion is relatively large due to the non-linear operation using large parameters.

GAN performs a convolution operation, so that it is possible to perform a conversion using not only linear conversion of pixel values but also information of surrounding pixels.

Further, since the position of the image information of the lesion does not change between the normal light image 14A and the special light converted image, the correct data indicating the correct image recognition of the special light converted image can use the correct data 14B in common. . Therefore, it is possible to develop a learning model more efficiently than capturing and collecting a large amount of special light images in which a lesion is captured, and performing a process of coloring and learning the correct answer data. Note that the correct answer data indicating the correct answer of the image recognition of the special light converted image may be correct answer data generated using the correct answer data of the normal light image 14A.

Furthermore, even when an endoscope system that shoots with a new special light is released, learning data can be created by reusing the data with correct answers in the past as long as a generated model is created.

<Third embodiment>
An endoscope system to which the endoscope image learning device is applied will be described.

[Configuration of endoscope system]
FIG. 8 is an external view illustrating an endoscope system 100 according to the third embodiment. As shown in FIG. 8, the endoscope system 100 includes an endoscope 112, a light source device 114, a processor device 116, a display unit 118, and an input unit 120.

The endoscope 112 is a lower endoscope inserted from the anus of the subject and used for observing the rectum, large intestine and the like. The endoscope 112 is optically connected to the light source device 114. Further, the endoscope 112 is electrically connected to the processor device 116.

The endoscope 112 includes an insertion portion 112A inserted into a body cavity of a subject, an operation portion 112B provided at a base end portion of the insertion portion 112A, a bending portion 112C provided at a distal end side of the insertion portion 112A, And a distal end portion 112D.

The operation unit 112B is provided with an angle knob 112E and a mode changeover switch 113.

By operating the angle knob 112E, the bending portion 112C bends. By this bending operation, the distal end portion 112D is directed in a desired direction.

The mode switch 113 is used for switching the observation mode. The endoscope system 100 has a plurality of observation modes with different wavelength patterns of irradiation light. The doctor can set a desired observation mode by operating the mode changeover switch 113. The endoscope system 100 generates an image according to the set observation mode based on a combination of the wavelength pattern and the image processing, and displays the image on the display unit 118.

(4) The operation unit 112B is provided with an acquisition instruction input unit (not shown). The acquisition instruction input unit is an interface for a doctor to input a still image acquisition instruction. The acquisition instruction input unit accepts a still image acquisition instruction. The instruction to acquire a still image received by the acquisition instruction input unit is input to the processor device 116.

The processor device 116 is electrically connected to the display unit 118 and the input unit 120. The display unit 118 is a display device that outputs and displays an image of the observation target, information related to the image of the observation target, and the like. The input unit 120 functions as a user interface that receives input operations such as setting of functions of the endoscope system 100 and various instructions.

FIG. 9 is a block diagram showing functions of the endoscope system 100. As shown in FIG. 9, the light source device 114 includes a first laser light source 122A, a second laser light source 122B, and a light source control unit 124.

The first laser light source 122A is a blue laser light source having a center wavelength of 445 nm. The second laser light source 122B is a violet laser light source having a center wavelength of 405 nm. Laser diodes can be used as the first laser light source 122A and the second laser light source 122B. Light emission of the first laser light source 122A and the second laser light source 122B is individually controlled by the light source control unit 124. The emission intensity ratio between the first laser light source 122A and the second laser light source 122B is changeable.

As shown in FIG. 9, the endoscope 112 includes an optical fiber 128A, an optical fiber 128B, a phosphor 130, a diffusing member 132, an imaging lens 134, an imaging element 136, and an analog-to-digital converter 138. And

The first laser light source 122A, the second laser light source 122B, the optical fiber 128A, the optical fiber 128B, the phosphor 130, and the diffusion member 132 constitute an irradiation unit.

(4) The laser beam emitted from the first laser light source 122A is applied to the phosphor 130 disposed on the distal end 112D of the endoscope 112 by the optical fiber 128A. The phosphor 130 is configured to include a plurality of types of phosphors that absorb part of the blue laser light from the first laser light source 122A and excite and emit green to yellow light. As a result, the light emitted from the phosphor 130 emits green to yellow excitation light L11 using blue laser light from the first laser light source 122A as excitation light, and blue laser light transmitted without being absorbed by the phosphor 130. L12 is combined with the light L1 to produce white (pseudo white) light L1.

白色 Incidentally, the white light mentioned here is not limited to a light that strictly includes all wavelength components of visible light. For example, any light that includes light in a specific wavelength band, such as R, G, and B, may be used, and light that includes a wavelength component from green to red or light that includes a wavelength component from blue to green is also broadly included. Shall be considered.

On the other hand, the laser beam emitted from the second laser light source 122B is irradiated by the optical fiber 128B onto the diffusion member 132 arranged at the distal end 112D of the endoscope 112. For the diffusion member 132, a light-transmitting resin material or the like can be used. The light emitted from the diffusion member 132 is light L2 having a narrow band wavelength, in which the light amount is uniform in the irradiation area.

FIG. 10 is a graph showing the intensity distribution of light L1 and light L2. The light source controller 124 changes the light amount ratio between the first laser light source 122A and the second laser light source 122B. Thus, the light amount ratio between the light L1 and the light L2 is changed, and the wavelength pattern of the irradiation light L0 that is a combined light of the light L1 and the light L2 is changed. Therefore, it is possible to irradiate the irradiation light L0 having a different wavelength pattern according to the observation mode.

に Returning to the description of FIG. 9, the imaging lens 134, the imaging element 136, and the analog-to-digital conversion unit 138 constitute an imaging unit (camera). The imaging unit is disposed at the distal end 112D of the endoscope 112.

The imaging lens 134 forms an image of the incident light on the imaging element 136. The image sensor 136 generates an analog signal according to the received light. As the imaging element 136, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor is used. An analog signal output from the image sensor 136 is converted into a digital signal by an analog-to-digital converter 138 and input to the processor device 116.

{Circle around (4)} The processor device 116 includes an imaging control unit 140, an image processing unit 142, an image acquisition unit 144, and an image recognition unit 146.

The imaging control unit 140 controls the light source control unit 124 of the light source device 114, the image sensor 136 and the analog-to-digital conversion unit 138 of the endoscope 112, and the image processing unit 142 of the processor device 116, thereby controlling the endoscope. The system 100 comprehensively controls shooting of moving images and still images.

The image processing unit 142 performs image processing on the digital signal input from the analog-to-digital conversion unit 138 of the endoscope 112, and generates image data indicating an image (hereinafter, referred to as an image). The image processing unit 142 performs image processing according to the wavelength pattern of irradiation light at the time of shooting.

The image acquisition unit 144 acquires the image generated by the image processing unit 142. That is, the image acquisition unit 144 sequentially acquires a plurality of images obtained by photographing the inside of the body cavity of the subject (an example in the living body) in a time-series manner at a fixed frame rate. Note that the image acquisition unit 144 may acquire an image input from the input unit 120 or an image stored in the storage unit 162. Further, an image may be obtained from an external device such as a server connected to a network (not shown). The images in these cases are also preferably a plurality of images taken in time series.

The image recognition unit 146 (an example of an endoscope image recognition device) performs image recognition of an image acquired by the image acquisition unit 144 using a learning model learned by the endoscope image learning device 10. In the present embodiment, a lesion is detected from the image acquired by the image acquisition unit 144. The lesion here is not limited to the one caused by the disease, but includes a region in a state different from a normal state in appearance. Examples of lesions include treatment scars such as polyps, cancer, colonic diverticulum, inflammation, EMR (Endoscopic Mucosal Resection) scars or ESD (Endoscopic Submucosal Dissection) scars, clipping points, bleeding points, perforations, and vascular atypia. Can be

The display control unit 158 causes the display unit 118 to display the image generated by the image processing unit 142. Further, the lesion detected by the image recognition unit 146 may be superimposed on the image so that the lesion can be recognized.

The storage control unit 160 causes the storage unit 162 to store the image generated by the image processing unit 142. For example, the storage unit 162 stores an image captured according to a still image acquisition instruction and information on a wavelength pattern of the irradiation light L0 when the image is captured.

The storage unit 162 is a storage device such as a hard disk, for example. Note that the storage unit 162 is not limited to the one built in the processor device 116. For example, an external storage device (not shown) connected to the processor device 116 may be used. The external storage device may be connected via a network (not shown).

The endoscope system 100 configured as described above normally shoots a moving image at a fixed frame rate, and displays the shot image on the display unit 118. Further, a lesion is detected from the captured moving image, and the detected lesion is superimposed on the moving image and displayed on the display unit 118 so as to be recognizable.

According to the endoscope system 100, since the image recognition of the endoscope image is performed by the image recognition unit 146 using the learning model learned by the endoscope image learning device 10, the image recognition of the special light image is appropriately performed. It can be carried out.

<Appendix>
In addition to the above-described aspects and examples, the following configurations are also included in the scope of the present invention.

(Appendix 1)
The medical image analysis processing unit detects a region of interest, which is a region of interest, based on the feature amounts of pixels of the medical image (endoscope image),
The medical image analysis result acquisition unit is a medical image processing device that acquires an analysis result of the medical image analysis processing unit.

(Appendix 2)
The medical image analysis processing unit detects the presence or absence of a target to be noted based on the feature amount of the pixel of the medical image,
The medical image analysis result acquisition unit is a medical image processing device that acquires an analysis result of the medical image analysis processing unit.

(Appendix 3)
The medical image analysis result acquisition unit,
Acquired from a recording device that records the analysis results of medical images,
The analysis result is a medical image processing apparatus in which either or both of a region of interest, which is a region of interest included in the medical image, and a target of interest, which are notable, are included.

(Appendix 4)
A medical image processing apparatus, wherein a medical image is a normal light image obtained by irradiating white band light or light of a plurality of wavelength bands as white band light.

(Appendix 5)
A medical image is an image obtained by irradiating light in a specific wavelength band,
The medical image processing apparatus has a specific wavelength band narrower than the white wavelength band.

(Appendix 6)
The specific wavelength band is a medical image processing device in a visible blue or green band.

(Appendix 7)
The specific wavelength band includes a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and the light of the specific wavelength band has a peak wavelength in the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. Image processing device.

(Appendix 8)
The medical image processing device has a specific wavelength band in a visible red band.

(Appendix 9)
The specific wavelength band includes a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light of the specific wavelength band has a peak wavelength in the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm. Image processing device.

(Appendix 10)
The specific wavelength band includes a wavelength band having a different extinction coefficient between oxyhemoglobin and reduced hemoglobin, and light of a specific wavelength band has a peak wavelength in a wavelength band having a different extinction coefficient between oxyhemoglobin and reduced hemoglobin. Medical image processing device.

(Appendix 11)
The specific wavelength band includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and light of the specific wavelength band is 400 ± 10 nm, 440 ± 10 nm, 470 ± A medical image processing apparatus having a peak wavelength in a wavelength band of 10 nm or 600 nm to 750 nm.

(Appendix 12)
A medical image is an in-vivo image of a living body,
The in-vivo image is a medical image processing apparatus having information on fluorescence emitted by a fluorescent substance in a living body.

(Appendix 13)
A medical image processing apparatus that obtains fluorescence by irradiating the living body with excitation light having a peak of 390 nm to 470 nm.

(Appendix 14)
A medical image is an in-vivo image of a living body,
The specific wavelength band is a wavelength band of infrared light in the medical image processing apparatus.

(Appendix 15)
The specific wavelength band includes a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and the light of the specific wavelength band has a peak wavelength in a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm. Processing equipment.

(Appendix 16)
The medical image acquisition unit is configured to acquire a special light image having information of a specific wavelength band based on a normal light image obtained by irradiating light in a plurality of wavelength bands as light in a white band or light in a white band. An optical image acquisition unit,
The medical image is a medical image processing device that is a special light image.

(Appendix 17)
A medical image processing apparatus in which a signal in a specific wavelength band is obtained by a calculation based on RGB (Red Green Blue) or CMY (Cyan, Magenta, Yellow) color information included in a normal light image.

(Appendix 18)
Light in the white band, or a normal light image obtained by irradiating light in a plurality of wavelength bands as light in the white band, and a calculation based on at least one of the special light image obtained by irradiating light in a specific wavelength band, A feature image generating unit that generates a feature image;
A medical image processing device in which the medical image is a feature image.

(Appendix 19)
The medical image processing apparatus according to any one of supplementary notes 1 to 18,
Light in a white wavelength band, or an endoscope that acquires an image by irradiating at least one of light in a specific wavelength band,
An endoscope device comprising:

(Appendix 20)
A diagnosis support device comprising the medical image processing device according to any one of supplementary notes 1 to 18.

(Appendix 21)
A medical service support device comprising the medical image processing device according to any one of supplementary notes 1 to 18.

<Others>
The endoscope image learning method described above is configured as a program for causing a computer to realize each step, and a non-temporary recording medium such as a CD-ROM (Compact Disk-Read Only Memory) storing the program is configured. It is also possible.

In the embodiment described so far, for example, a hardware structure of a processing unit (processing @ unit) that executes various processes of the endoscope image learning device 10 and the endoscope image learning device 50 is as follows. Various processors. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a GPU (Graphics Processing Unit), which is a processor specialized in image processing, Dedicated to execute specific processing such as Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit). A dedicated electric circuit which is a processor having a designed circuit configuration is included.

One processing unit may be configured by one of these various processors, or may be configured by two or more processors of the same type or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a CPU). (Combination of GPUs). Further, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, as represented by a computer such as a server and a client, one processor is configured by a combination of one or more CPUs and software. There is a form in which a processor functions as a plurality of processing units. Second, as represented by a system-on-chip (SoC), a form using a processor that realizes the functions of the entire system including a plurality of processing units with one integrated circuit (IC) chip is used. is there. As described above, the various processing units are configured using one or more various processors as a hardware structure.

Further, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

技術 The technical scope of the present invention is not limited to the scope described in the above embodiment. The configuration and the like in each embodiment can be appropriately combined between the embodiments without departing from the spirit of the present invention.

Reference Signs List 10 Endoscope image learning device 12 Communication unit 14 First database 14A Normal light image 14B Correct data 16 Second database 16A Special light image 16B Correct data 18 Operation unit 26 Display unit 30 Machine learning unit 32 ... CNN
32A ... input layer 32B ... intermediate layer 32C ... output layer 34 ... error calculating unit 36 ... parameter updating unit 40 ... image generating unit 42 ... generator 44 ... discriminator 50 ... endoscope image learning device 60 ... image generating unit 62 ... 1st generator 64 ... 1st discriminator 66 ... 2nd generator 68 ... 2nd discriminator 100 ... endoscope system 112 ... endoscope 112A ... insertion part 112B ... operation part 112C ... bending part 112D ... tip part 112E ... Angle knob 113 ... Mode switch 114 ... Light source device 116 ... Processor device 118 ... Display unit 120 ... Input unit 122A ... First laser light source 122B ... Second laser light source 124 ... Light source control unit 128A ... Optical fiber 128B ... Optical fiber 130 ... Phosphor 132 132 Diffusion member 134 ... Imaging lens 136 ... Imaging element 138 ... Analog-to-digital converter 140 ... Shooting controller 142 ... Image Processing unit 144 ... image acquisition unit 146 ... image recognition unit 158 ... display control unit 160 ... storage control unit 162 ... storage unit

Claims (15)

An image generation unit that generates an endoscope image of the first wavelength band light using a generation model;
A machine learning unit that learns a learning model for image recognition using the endoscope image generated by the image generation unit,
An endoscope image learning device comprising: The image generation unit receives a second endoscope image captured with a second wavelength band light different from the first wavelength band light as an input, and converts the second endoscope image into the first wavelength band light. The endoscope image learning device according to claim 1, wherein the device converts the wavelength band light into a first endoscope image. The endoscope image learning device according to claim 2, wherein the first wavelength band light is special light, and the second wavelength band light is white light. 4. The endoscope image learning device according to claim 2, wherein the first endoscope image and the second endoscope image have the same correct answer data indicating the correct answer of image recognition. 5. The correct answer data indicating the correct answer of the image recognition of the first endoscope image is correct answer data generated using the correct answer data indicating the correct answer of the second endoscope image. Endoscope image learning device. 6. The endoscope image learning device according to claim 1, wherein the learning model detects a lesion area from the input endoscope image. 7. 7. The endoscope image learning device according to claim 6, wherein the learning model classifies the lesion area as benign or malignant. The generation model according to any one of claims 2 to 5, wherein the generation model is a generation model obtained by learning using an endoscope image captured with the first wavelength band light and the second endoscope image. 2. The endoscope image learning device according to claim 1. The endoscope image learning device according to any one of claims 1 to 8, wherein the generation model is learned using a convolutional neural network. The endoscope image learning device according to claim 9, wherein the generation model is GAN (Generative Adversarial Networks). The endoscope image learning device according to any one of claims 1 to 10, wherein the machine learning unit learns the learning model using a convolutional neural network. An image acquisition unit that acquires an endoscope image captured with the first wavelength band light,
An image recognition unit that performs image recognition of the obtained endoscope image using a learning model learned by the endoscopic image learning device according to any one of claims 1 to 11,
Endoscope image recognition device provided with. An image generation step of generating an endoscope image of the first wavelength band light by a generation model;
Machine learning step of learning a learning model for image recognition using the endoscope image generated in the image generation step,
Endoscope image learning method comprising: A program for causing a computer to execute the endoscopic image learning method according to claim 13. 14. A non-transitory and computer-readable recording medium that causes a computer to execute the program according to claim 13 when a command stored in the recording medium is read by the computer.

Images