JP7005767B2 - Endoscopic image recognition device, endoscopic image learning device, endoscopic image learning method and program - Google Patents

Description

The present invention relates to an endoscopic image recognition device, an endoscopic image learning device, an endoscopic image learning method and a program, and particularly relates to a technique for learning a learning model for image recognition.

In endoscopy, it is expected that not only normal light but also special light emitted using a special wavelength balance will be used to improve the accuracy of lesion discrimination and detection and reduce the oversight of lesions. There is.

On the other hand, a technique of recognizing a lesion on an endoscopic image by a computer is known. In particular, by using an image recognizer created by machine learning, highly accurate diagnosis becomes possible. Even in the image recognition technology using such an image recognizer, it is expected that the recognition accuracy of the endoscopic image using special light will be improved.

The accuracy of the image recognizer created by machine learning largely depends on the image data used for learning. Therefore, it is necessary to collect a large amount of endoscopic images taken by various light sources. However, in the actual field, inspection is usually performed with white light, and special light is selectively used according to the purpose of inspection, etc., so it is difficult to collect images taken with special light. rice field. Therefore, in order to improve the recognition accuracy of the special light image, it is necessary to generate a special light image suitable for machine learning and use it as learning data.

In Patent Document 1, in generating the spectroscopic estimation images of the endoscope first observation image and the endoscope second observation image, respectively, the endoscopic first observation image is described according to a predetermined wavelength set. One spectroscopic estimation image is generated, and for the second observation image of the endoscope, an approximate wavelength set that approximately generates the spectrum of the irradiation light from the first light source is set, and according to the set approximate wavelength set. A technique for generating a second spectroscopic estimation image is disclosed.

Further, Patent Document 2 describes a technique for creating a deformed image by performing affine deformation on a part image.

Further, Patent Document 3 describes a technique of learning a plurality of combinations of a low-resolution image and a high-resolution image, performing inference based on the combination, and performing resolution conversion.

Japanese Unexamined Patent Publication No. 2011-194882 Japanese Unexamined Patent Publication No. 2012-043151 Japanese Unexamined Patent Publication No. 2018-005841

However, the techniques described in Patent Documents 1 to 3 aim to improve the visibility of the image presented to the user. Further, the image conversion method described in Patent Documents 1 and 2 is a so-called manually designed conversion method, and faithfully reproduces a complicated conversion such as conversion of wavelength band light such as a white light image to a special light image. Is not easy. Therefore, it was not possible to generate an image suitable for machine learning, and it was not possible to properly train a learning model for image recognition.

The present invention has been made in view of such circumstances, and is an endoscopic image recognition device, an endoscopic image learning device, and an endoscopy that appropriately learns a learning model for image recognition that recognizes an endoscopic image. It is an object of the present invention to provide a mirror image learning method and a program.

In order to achieve the above object, one aspect of the endoscopic image learning device is an image generation unit that generates an endoscopic image of first wavelength band light by a generation model, and an internal vision generated by the image generation unit. It is an endoscopic image learning device including a machine learning unit that learns a learning model for image recognition using a mirror image.

According to this aspect, an endoscopic image of the first wavelength band light is generated by a generation model, and the generated endoscopic image is used to train a learning model for image recognition. It is possible to appropriately train a learning model for image recognition that recognizes an endoscopic image.

The image generation unit receives a second endoscope image taken with a second wavelength band light different from the first wavelength band light as an input, and uses the second endoscope image as the first wavelength band light. It is preferable to convert it into a first endoscopic image. Thereby, the first endoscopic image of the first wavelength band light can be appropriately generated.

It is preferable that the first wavelength band light is special light and the second wavelength band light is white light. This makes it possible to generate a special light image with a small number of data from a white light image with a large number of data.

It is preferable that the first endoscope image and the second endoscope image have the same correct answer data indicating the correct answer of the image recognition. As a result, it can be used without creating new correct answer data.

The correct answer data indicating the correct answer of the image recognition of the first endoscope image may be the correct answer data generated by using the correct answer data indicating the correct answer of the second endoscope image. Even if such correct answer data is used, the learning model can be appropriately trained.

The learning model preferably detects the lesion area from the input endoscopic image. This makes it possible to appropriately detect the lesion area from the input endoscopic image.

The learning model preferably classifies the lesion area as either benign or malignant. This makes it possible to appropriately classify whether the lesion area is benign or malignant.

The generative model is preferably a generative model obtained by learning using the endoscopic image taken with the first wavelength band light and the second endoscopic image. This makes it possible to appropriately generate an endoscopic image of the first wavelength band light.

The generative model is preferably trained using a convolutional neural network. This makes it possible to appropriately generate an endoscopic image of the first wavelength band light.

The generative model is preferably GAN (Generative Adversarial Networks). This makes it possible to appropriately generate an endoscopic image of the first wavelength band light.

It is preferable that the machine learning unit learns the learning model using a convolutional neural network. This makes it possible to appropriately learn the learning model for image recognition.

In order to achieve the above object, one aspect of the endoscopic image recognition device is an image acquisition unit that acquires an endoscopic image captured by a first wavelength band light, and the endoscopic image learning device described above. It is an endoscope image recognition device including an image recognition unit for image recognition of an endoscope image acquired by using a learned learning model.

According to this aspect, since the endoscopic image taken by the first wavelength band light is acquired and the image recognition of the endoscopic image acquired by using the learning model is performed, the endoscopic image can be recognized. Image recognition can be performed appropriately.

In order to achieve the above object, one aspect of the endoscopic image learning method is an image generation step of generating an endoscopic image of a first wavelength band light by a generation model and an endoscopy generated in the image generation step. It is an endoscopic image learning method including a machine learning step of learning a learning model for image recognition using a mirror image.

According to this aspect, an endoscopic image of the first wavelength band light is generated by a generation model, and the generated endoscopic image is used to train a learning model for image recognition. It is possible to appropriately train a learning model for image recognition that recognizes an endoscopic image.

One aspect of the program for causing the computer to execute in order to achieve the above object is the program for causing the computer to execute the above-mentioned endoscopic image learning method.

According to this aspect, an endoscopic image of the first wavelength band light is generated by a generation model, and the generated endoscopic image is used to train a learning model for image recognition. It is possible to appropriately train a learning model for image recognition that recognizes an endoscopic image.

According to the present invention, it is possible to appropriately learn a learning model for image recognition that recognizes an endoscopic image.

FIG. 1 is a block diagram showing an example of a hardware configuration of an endoscopic image learning device. FIG. 2 is a functional block diagram showing the main functions of the endoscopic image learning device. FIG. 3 is a functional block diagram showing the main functions of the machine learning unit. FIG. 4 is a functional block diagram showing the main functions of the image generation unit. FIG. 5 is a flowchart showing an example of the endoscopic image learning method. FIG. 6 is a functional block diagram showing the main functions of the endoscopic image learning device. FIG. 7 is a functional block diagram showing the main functions of the image generation unit. FIG. 8 is an external view showing an endoscope system. FIG. 9 is a block diagram showing the function of the endoscope system. FIG. 10 is a graph showing the intensity distributions of light L1 and light L2.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<Hardware configuration of medical image learning device>
FIG. 1 is a block diagram showing an example of a hardware configuration of the endoscopic image learning device according to the present invention.

The endoscopic image learning device 10 is composed of a personal computer or a workstation. The endoscope image learning device 10 mainly includes a communication unit 12, a first database 14, a second database 16, an operation unit 18, a CPU (Central Processing Unit) 20, a RAM (Random Access Memory) 22, and the RAM (Random Access Memory) 22. It is composed of a ROM (Read Only Memory) 24 and a display unit 26.

The communication unit 12 is an interface that performs communication processing with an external device by wire or wirelessly and exchanges information with the external device.

The first database 14 is a first data for learning including a normal light image group consisting of a plurality of normal light images 14A captured by normal light and correct answer data 14B showing correct image recognition results of each normal light image 14A. It is a large-capacity storage device that stores sets. The second database 16 is a second data for learning including a special light image group consisting of a plurality of special light images 16A captured by special light and correct answer data 16B showing correct image recognition results of each special light image 16A. It is a large-capacity storage device that stores sets.

Here, the normal light image and the special light image are color images taken under different light sources by an endoscope device (not shown).

Normal light is light (white light) in which light in all wavelength bands of visible light is mixed almost evenly, and normal light images are used for normal observation. Therefore, a relatively large number of normal optical image groups can be collected.

On the other hand, special light is light in various wavelength bands according to the purpose of observation, which is a combination of light in one specific wavelength band or light in a plurality of specific wavelength bands, and is narrower than the white wavelength band. (NBI (registered trademark), FICE (Flexible spectral imaging color enhancement, registered trademark)).

The first example of a specific wavelength band is, for example, a blue band or a green band in the visible region. The wavelength band of the first example includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and the light of the first example has a peak wavelength in the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. ..

A second example of a particular wavelength band is, for example, the red band in the visible range. The wavelength band of the second example includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light of the second example has a peak wavelength in the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. ..

The third example of a specific wavelength band includes a wavelength band having different extinction coefficients between oxidized hemoglobin and reduced hemoglobin, and the light of the third example has a peak wavelength in a wavelength band having different extinction coefficients between oxidized hemoglobin and reduced hemoglobin. Has. The wavelength band of the third example includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm or less, and the light of the third example is 400 ± 10 nm, 440 ± 10 nm, 470. It has a peak wavelength in the wavelength band of ± 10 nm or 600 nm or more and 750 nm or less.

The fourth example of the specific wavelength band is the wavelength band (390 nm to 470 nm) of the excitation light used for observing the fluorescence emitted by the fluorescent substance in the living body (fluorescence observation) and exciting the fluorescent substance.

A fifth example of a specific wavelength band is the wavelength band of infrared light. The wavelength band of the fifth example includes a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less, and the light of the fifth example has a peak wavelength in the wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less.

Since lesions are difficult to see in special light images taken under special light having such a specific wavelength band, they are used only for observation purposes such as observing the surface structure, and the number of data is not large.

In this example, the first data set of the normal optical image group stored in the first database 14 is prepared more than the second data set of the special optical image group stored in the second database 16. And.

Further, in the first database 14 and the second database 16, the correct answer data 14B and 16B stored in association with each normal light image 14A and each special light image 16A are reflected in, for example, the normal light image and the special light image. The type of lesion, data showing the location of the lesion, case-specific identification information, etc. can be considered. The classification of lesions includes two classifications, neoplastic and non-neoplastic, and NICE classification. The data indicating the position of the lesion may be rectangular information surrounding the lesion, mask data that covers the lesion, or the like.

The first database 14 and the second database 16 may be the same storage device. Further, the first database 14 and the second database 16 may be provided outside the endoscope image learning device 10 instead of being provided inside the endoscope image learning device 10. In this case, the data set for learning can be acquired from the external database via the communication unit 12.

The operation unit 18 is an input interface that receives various operation inputs to the endoscope image learning device 10. As the operation unit 18, a keyboard, a mouse, or the like that is wiredly or wirelessly connected to the computer is used.

The CPU 20 reads various programs stored in the ROM 24 or a hard disk device (not shown) and executes various processes. The RAM 22 is used as a work area of the CPU 20. Further, the RAM 22 is used as a storage unit for temporarily storing the read program and various data.

The display unit 26 is an output interface on which necessary information of the endoscope image learning device 10 is displayed. As the display unit 26, various monitors such as a liquid crystal monitor that can be connected to a computer are used.

In the endoscope image learning device 10, the CPU 20 reads out the endoscope image learning program stored in the ROM 24 or the hard disk device or the like in response to an instruction input from the operation unit 18, and executes the endoscope image learning program. As a result, as will be described later, the endoscopic image is generated by the generated model, and the learning model using the endoscopic image is learned.

<First Embodiment>
[Endoscopic image learning device]
FIG. 2 is a functional block diagram showing the main functions of the endoscope image learning device 10 according to the first embodiment. The endoscope image learning device 10 includes a machine learning unit 30 and an image generation unit 40.

The machine learning unit 30 is learned using a large number of learning images. The machine learning unit 30 uses the data set of the normal optical image 14A and the correct answer data 14B stored in the first database 14 and the data set of the special optical image 16A and the correct answer data 16B stored in the second database 16. By training, a learning model for image recognition is generated. The machine learning unit 30 constructs a convolutional neural network (CNN), which is one of the learning models.

FIG. 3 is a functional block diagram showing the main functions of the machine learning unit 30. The machine learning unit 30 is mainly composed of a CNN 32, an error calculation unit 34, and a parameter update unit 36.

The CNN 32 is a recognizer that recognizes the type of lesion shown in the endoscopic image. The CNN 32 has a plurality of layer structures and holds a plurality of weight parameters. The CNN 32 can change from an unlearned model to a trained model by updating the weight parameter from the initial value to the optimum value.

The CNN 32 includes an input layer 32A, an intermediate layer 32B, and an output layer 32C. The input layer 32A, the intermediate layer 32B, and the output layer 32C each have a structure in which a plurality of "nodes" are connected by "edges".

The normal light image 14A and the special light image 16A to be learned are input to the input layer 32A.

The intermediate layer 32B is a layer for extracting features from an image input from an input layer. The intermediate layer 32B has a plurality of sets including a convolution layer and a pooling layer as one set, and a fully connected layer. The convolution layer performs a convolution operation using a filter on a node near the node in the previous layer, and acquires a feature map. The pooling layer reduces the feature map output from the convolution layer to a new feature map. The fully connected layer joins all the nodes of the immediately preceding layer (here, the pooling layer). The convolution layer plays a role of feature extraction such as edge extraction from an image, and the pooling layer plays a role of imparting robustness so that the extracted features are not affected by translation or the like. The intermediate layer 32B is not limited to the case where the convolution layer and the pooling layer are set as one set, but also includes the case where the convolution layers are continuous and the normalization layer.

The output layer 32C is a layer that outputs a recognition result for detecting a lesion (an example of a lesion region) reflected in an endoscopic image based on the features extracted by the intermediate layer 32B. Further, the recognition result for classifying the detected lesion as benign or malignant may be output, or the recognition result for classifying the type of lesion may be output.

When classifying the types of lesions, the learned CNN32 classifies endoscopic images into three categories, for example, "neoplastic", "non-neoplastic", and "other", and the recognition result is "neoplastic". It is output as three scores corresponding to "non-neoplastic" and "other". The total of the three scores is 100%.

Arbitrary initial values are set for the coefficient of the filter applied to each convolution layer of the CNN 32 before training, the offset value, and the weight of the connection with the next layer in the fully connected layer.

The error calculation unit 34 acquires the recognition result output from the output layer 32C of the CNN 32 and the correct answer data for the input image, and calculates the error between the two. As a method of calculating the error, for example, soft max cross entropy, least squared error (MSE: Mean Squared Error), or the like can be considered.

The parameter update unit 36 adjusts the weight parameter of the CNN 32 by the error back propagation method based on the error calculated by the error calculation unit 34.

This parameter adjustment process is repeated, and repeated learning is performed until the difference between the output of CNN32 and the correct answer data becomes small.

The machine learning unit 30 uses all the data sets of the normal optical image group stored in the first database 14 and the special optical image group stored in the second database 16 to optimize each parameter of the CNN 32. That is, the machine learning unit 30 learns the learning model using the convolutional neural network.

For learning of the machine learning unit 30, a mini-batch method may be used in which a certain number of data sets are extracted from all the data sets, batch processing of learning is performed by the extracted data sets, and this is repeated.

The machine learning unit 30 does not have to use correct answer data depending on the recognition process to be realized. Further, the machine learning unit 30 may extract features by an algorithm designed in advance such as edge extraction, and may use the information to learn by a support vector machine or the like.

The image generation unit 40 generates a special optical image by a generation model. In the generative model, the conditional distribution p (x | C) of the data x for a certain class C is modeled. Here, the class corresponds to the type of light source, and the data corresponds to the image. Using this generative model, it is possible to generate pseudo data belonging to a certain class.

As an example of the generative model, a variational self-encoder (VAE: Variational Auto Encorder), a hostile generation network (GAN: Generative Adversarial Networks), and the like are known.

The image generation unit 40 of this example constructs a GAN as a generation model. GAN is a generative model formed by alternately learning a "Generator" that creates data and a "Discriminator" that identifies data.

FIG. 4 is a functional block diagram showing the main functions of the image generation unit 40. The image generation unit 40 is mainly composed of a generator 42 and a discriminator 44.

The generator 42 corresponds to the "Generator" and the discriminator 44 corresponds to the "Discriminator". The generator 42 takes noise z as an input and generates an image x1 which is a special light generation image. The classifier 44 inputs the generated image x1 and the special optical image 16A prepared as the learning image x0, and classifies whether the input image is the generated data or the learning data.

The generator 42 and the classifier 44 are trained using a convolutional neural network.

The learning of the classifier 44 is performed in the same flow as that of the machine learning unit 30. Specifically, an image is input, and whether the input image is an image generated by the generator 42 or a realistic endoscopic image is classified.

On the other hand, in the learning of the generator 42, the noise z is used as an input, and the data is converted into the shape of an image by tracing the general CNN in the reverse direction. In the learning of the generator 42, an upsampling method such as "fractionally-strided convolution" that increases the size of the image is used.

As the learning progresses, the generator 42 and the classifier 44 improve their accuracy, and the generator 42 is not discriminated by the classifier 44, and is a special light generation image closer to a real special light image (of the first wavelength band light). It will be possible to generate an example of an endoscopic image). The generator 42 acquired by this learning is a generative model.

[Endoscopic image learning method]
FIG. 5 is a flowchart showing an example of an endoscope image learning method by the endoscope image learning device 10. The endoscopic image learning method includes an image generation step S1 and a machine learning step S2.

In the image generation step S1, a special light generation image equivalent to the special light image is generated by the generation model of the image generation unit 40. For example, 100 special light images are generated.

In the machine learning step S2, the learning model of the machine learning unit 30 is learned using the special optical image generated in the image generation step S1. Here, learning is performed using 100 special optical images.

As described above, by generating the special light image by the generation model, the special light image with a small number of data can be inflated and collected. Further, by learning a learning model for endoscopic image recognition using the generated special light image, it is possible to appropriately learn a special light image having a small number of data. Therefore, it is possible to improve the performance of image recognition by a special optical image.

<Second embodiment>
[Endoscopic image learning device]
The hardware configuration of the endoscopic image learning device according to the second embodiment is the same as the block diagram shown in FIG.

FIG. 6 is a functional block diagram showing the main functions of the endoscope image learning device 50 according to the second embodiment. The same reference numerals are given to the parts common to the block diagram shown in FIG. 2, and detailed description thereof will be omitted.

The endoscope image learning device 50 includes an image generation unit 60. The image generation unit 60 is connected to the first database 14 and the second database 16.

In the second embodiment, a model obtained by applying the generated model to image conversion is used as an image generation unit. The image generation unit 60 constructs a CycleGAN as a generation model. In CycleGAN, the conversion between a certain class C1 and class C2 is obtained. CycleGAN uses a network that prepares "Generator" and "Discriminator" for two classes. The "Generator" of class C1 learns a distribution that converts an image of class C2 into an image of class C1. The class C1 "Discriminator" distinguishes between a real class C1 image and a converted image. Class C2 "Generator" and "Discriminator" do the opposite.

Also in the learning of CycleGAN, it is mainstream to use CNN. The learning of "Discriminator" is the same as that of normal GAN, and "Generator" uses images for input and output to learn appropriate conversion.

FIG. 7 is a functional block diagram showing the main functions of the image generation unit 60. The image generation unit 60 is mainly composed of a first generator 62, a first classifier 64, a second generator 66, and a second classifier 68.

The first generator 62 corresponds to a class C1 “Generator” and the first classifier 64 corresponds to a class C1 “Discriminator”. Further, the second generator 66 corresponds to the "Generator" of the class C2, and the second classifier 68 corresponds to the "Discriminator" of the class C2.

The first generator 62 takes an image x02, which is a special light image 16A, as an input, and generates an image x11, which is a normal light conversion image obtained by converting the image x02 into a normal light image. The first classifier 64 inputs the generated image x11 and the normal optical image 14A prepared as the learning image x01, and classifies whether the input image is the generated data or the learning data.

Further, the second generator 66 takes an image x01 (an example of a second endoscope image taken with the second wavelength band light) which is a normal optical image 14A as an input, and converts the image x01 into a special optical image. An image x12 (an example of a first endoscope image of a first wavelength band light) which is a special light conversion image is generated. The second classifier 68 inputs the generated image x12 and the special optical image 16A prepared as the learning image x02, and classifies whether the input image is the generated data or the learning data.

Further, the first generator 62 takes an image x12, which is a special light conversion image generated by the second generator 66, as an input, and generates an image x21, which is a normal light conversion image of the image x12. The first classifier 64 inputs the generated image x21 and the learning image x01, and classifies whether the input image is the generated data or the learning data.

Similarly, the second generator 66 takes an image x11, which is a normal light conversion image generated by the first generator 62, as an input, and generates an image x22, which is a special light conversion image of the image x11. The second classifier 68 takes the generated image x22 and the learning image x02 as inputs, and classifies whether the input image is the generated data or the learning data.

The second generator 66 acquired by this learning is a generative model. As described above, according to the second generator 66 obtained by learning using the normal light image 14A and the special light image 16A, the white light image is converted into the special light image not identified by the second classifier 68. It can be said. The endoscopic image learning method is the same as that of the first embodiment.

According to the generative model of the second embodiment, it is possible to convert only the color tone with the composition of the normal light image 14A before conversion. Since the statistically optimal conversion method is learned from the large number of images used to create the generative model, more natural image generation can be expected.

In the technique described in Patent Document 1, a 3 × 3 matrix operation is performed on RGB (Red Green Blue) pixel values, and the expressive power is relatively small. In this embodiment, since GAN is used as the generative model, the expressive power of the conversion is relatively large due to the non-linear operation with large parameters.

Since GAN performs a convolution operation, it is possible to convert not only the pixel value by linear conversion but also by using the information of surrounding pixels.

Further, since the position of the image information of the lesion does not change between the normal light image 14A and the special light conversion image, the correct answer data 14B can be commonly used as the correct answer data indicating the correct answer of the image recognition of the special light conversion image. .. Therefore, it is possible to efficiently develop a learning model rather than taking and collecting a large number of special optical images showing lesions and going through the steps of coloring and learning the correct answer data. The correct answer data indicating the correct answer of the image recognition of the special optical conversion image may be the correct answer data generated by using the correct answer data of the normal optical image 14A.

Furthermore, even when an endoscope system that shoots with new special light is released, it is possible to create learning data by reusing past data with correct answers as long as a generative model is created.

<Third embodiment>
An endoscope system to which an endoscope image learning device is applied will be described.

[Configuration of endoscope system]
FIG. 8 is an external view showing the endoscope system 100 according to the third embodiment. As shown in FIG. 8, the endoscope system 100 includes an endoscope 112, a light source device 114, a processor device 116, a display unit 118, and an input unit 120.

The endoscope 112 is a lower endoscope inserted through the anus of a subject and used for observing the rectum, the large intestine, and the like. The endoscope 112 is optically connected to the light source device 114. Further, the endoscope 112 is electrically connected to the processor device 116.

The endoscope 112 includes an insertion portion 112A inserted into the body cavity of the subject, an operation portion 112B provided at the base end portion of the insertion portion 112A, and a bending portion 112C provided on the distal end side of the insertion portion 112A. It has a tip portion 112D.

The operation unit 112B is provided with an angle knob 112E and a mode changeover switch 113.

By operating the angle knob 112E, the curved portion 112C bends. This bending motion directs the tip 112D in a desired direction.

The mode changeover switch 113 is used for the observation mode changeover operation. The endoscope system 100 has a plurality of observation modes in which the wavelength patterns of the irradiation light are different from each other. The doctor can set the desired observation mode by operating the mode changeover switch 113. The endoscope system 100 generates an image according to the set observation mode by the combination of the wavelength pattern and the image processing, and displays it on the display unit 118.

Further, the operation unit 112B is provided with an acquisition instruction input unit (not shown). The acquisition instruction input unit is an interface for a doctor to input an acquisition instruction for a still image. The acquisition instruction input unit receives an acquisition instruction for a still image. The acquisition instruction of the still image received by the acquisition instruction input unit is input to the processor device 116.

The processor device 116 is electrically connected to the display unit 118 and the input unit 120. The display unit 118 is a display device that outputs and displays an image to be observed and information related to the image to be observed. The input unit 120 functions as a user interface that accepts input operations such as function settings of the endoscope system 100 and various instructions.

FIG. 9 is a block diagram showing the functions of the endoscope system 100. As shown in FIG. 9, the light source device 114 includes a first laser light source 122A, a second laser light source 122B, and a light source control unit 124.

The first laser light source 122A is a blue laser light source having a center wavelength of 445 nm. The second laser light source 122B is a purple laser light source having a center wavelength of 405 nm. A laser diode can be used as the first laser light source 122A and the second laser light source 122B. The light emission of the first laser light source 122A and the second laser light source 122B is individually controlled by the light source control unit 124. The emission intensity ratio between the first laser light source 122A and the second laser light source 122B can be freely changed.

Further, as shown in FIG. 9, the endoscope 112 includes an optical fiber 128A, an optical fiber 128B, a phosphor 130, a diffusion member 132, an image pickup lens 134, an image pickup element 136, and an analog-to-digital conversion unit 138. And have.

The irradiation unit is composed of the first laser light source 122A, the second laser light source 122B, the optical fiber 128A, the optical fiber 128B, the phosphor 130, and the diffusion member 132.

The laser light emitted from the first laser light source 122A is irradiated to the phosphor 130 arranged at the tip portion 112D of the endoscope 112 by the optical fiber 128A. The phosphor 130 includes a plurality of types of phosphors that absorb a part of the blue laser light from the first laser light source 122A and excite and emit light from green to yellow. As a result, the light emitted from the phosphor 130 is the green to yellow excitation light L11 using the blue laser light from the first laser light source 122A as the excitation light, and the blue laser light transmitted without being absorbed by the phosphor 130. Combined with L12, it becomes white (pseudo-white) light L1.

It should be noted that the white light referred to here is not limited to that strictly includes all wavelength components of visible light. For example, it may be any light containing a specific wavelength band such as R, G, B, etc., and broadly includes light containing a wavelength component from green to red, light containing a wavelength component from blue to green, and the like. It shall be muted.

On the other hand, the laser light emitted from the second laser light source 122B is irradiated to the diffusion member 132 arranged at the tip portion 112D of the endoscope 112 by the optical fiber 128B. As the diffusion member 132, a translucent resin material or the like can be used. The light emitted from the diffusing member 132 becomes light L2 having a narrow band wavelength in which the amount of light is uniform in the irradiation region.

FIG. 10 is a graph showing the intensity distributions of light L1 and light L2. The light source control unit 124 changes the light amount ratio between the first laser light source 122A and the second laser light source 122B. As a result, the light amount ratio between the light L1 and the light L2 is changed, and the wavelength pattern of the irradiation light L0, which is the combined light of the light L1 and the light L2, is changed. Therefore, it is possible to irradiate the irradiation light L0 having a different wavelength pattern depending on the observation mode.

Returning to the description of FIG. 9, the imaging unit (camera) is configured by the image pickup lens 134, the image pickup element 136, and the analog-to-digital conversion unit 138. The photographing unit is arranged at the tip portion 112D of the endoscope 112.

The image pickup lens 134 forms an image of the incident light on the image pickup element 136. The image sensor 136 generates an analog signal according to the received light. As the image pickup device 136, a CCD (Charge Coupled Device) type image sensor or a CMOS (Complementary Metal Oxide Semiconductor) type image sensor is used. The analog signal output from the image pickup element 136 is converted into a digital signal by the analog-to-digital conversion unit 138 and input to the processor device 116.

Further, the processor device 116 includes a shooting control unit 140, an image processing unit 142, an image acquisition unit 144, and an image recognition unit 146.

The photographing control unit 140 controls the light source control unit 124 of the light source device 114, the image pickup element 136 and the analog-digital conversion unit 138 of the endoscope 112, and the image processing unit 142 of the processor device 116 to control the endoscope. It controls the shooting of moving images and still images by the system 100.

The image processing unit 142 performs image processing on the digital signal input from the analog-digital conversion unit 138 of the endoscope 112 to generate image data (hereinafter referred to as an image) indicating an image. The image processing unit 142 performs image processing according to the wavelength pattern of the irradiation light at the time of shooting.

The image acquisition unit 144 acquires the image generated by the image processing unit 142. That is, the image acquisition unit 144 sequentially acquires a plurality of images taken in time series of the inside of the body cavity of the subject (an example in the living body) at a constant frame rate. The image acquisition unit 144 may acquire an image input from the input unit 120 or an image stored in the storage unit 162. Further, the image may be acquired from an external device such as a server connected to a network (not shown). The images in these cases are also preferably a plurality of images taken in time series.

The image recognition unit 146 (an example of the endoscope image recognition device) performs image recognition of the image acquired by the image acquisition unit 144 by using the learning model learned by the endoscope image learning device 10. In the present embodiment, the lesion is detected from the image acquired by the image acquisition unit 144. The lesion here is not limited to the one caused by the disease, and includes a region of a state different from the normal state in appearance. Examples of lesions include treatment scars such as polyps, cancer, colon diverticulum, inflammation, EMR (Endoscopic Mucosal Resection) scars or ESD (Endoscopic Submucosal Dissection) scars, clip sites, bleeding points, perforations, and vascular atypia. Will be.

The display control unit 158 causes the display unit 118 to display the image generated by the image processing unit 142. Further, the lesion detected by the image recognition unit 146 may be superposed on the image so as to be recognizable.

The storage control unit 160 stores the image generated by the image processing unit 142 in the storage unit 162. For example, the storage unit 162 stores the image taken according to the acquisition instruction of the still image and the information of the wavelength pattern of the irradiation light L0 when the image is taken.

The storage unit 162 is a storage device such as a hard disk. The storage unit 162 is not limited to the one built in the processor device 116. For example, it may be an external storage device (not shown) connected to the processor device 116. The external storage device may be connected via a network (not shown).

The endoscope system 100 configured in this way normally shoots a moving image at a constant frame rate, and displays the shot image on the display unit 118. In addition, a lesion is detected from the captured moving image, and the detected lesion is recognizable and superimposed on the moving image and displayed on the display unit 118.

According to the endoscope system 100, the image recognition of the endoscope image is performed by the image recognition unit 146 using the learning model learned by the endoscope image learning device 10, so that the image recognition of the special optical image can be appropriately performed. It can be carried out.

<Additional Notes>
In addition to the embodiments and examples described above, the configurations described below are also included within the scope of the invention.

(Appendix 1)
The medical image analysis processing unit detects a region of interest, which is a region of interest, based on the feature amount of pixels of a medical image (endoscopic image).
The medical image analysis result acquisition unit is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

(Appendix 2)
The medical image analysis processing unit detects the presence or absence of a noteworthy object based on the feature amount of the pixel of the medical image.
The medical image analysis result acquisition unit is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

(Appendix 3)
The medical image analysis result acquisition department
Obtained from a recording device that records the analysis results of medical images,
The analysis result is a medical image processing apparatus that is either or both of a notable area included in a medical image and a notable object.

(Appendix 4)
A medical image is a medical image processing device that is a normal optical image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in the white band.

(Appendix 5)
A medical image is an image obtained by irradiating light in a specific wavelength band.
A medical image processing device in which a specific wavelength band is narrower than the white wavelength band.

(Appendix 6)
A medical image processing device in which a specific wavelength band is a blue or green band in the visible range.

(Appendix 7)
The specific wavelength band includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and the light of the specific wavelength band has a peak wavelength in the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. Image processing device.

(Appendix 8)
A specific wavelength band is a medical image processing device that is a red band in the visible range.

(Appendix 9)
The specific wavelength band includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light of the specific wavelength band has a peak wavelength in the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. Image processing device.

(Appendix 10)
The specific wavelength band includes a wavelength band in which the absorption coefficient differs between the oxidized hemoglobin and the reduced hemoglobin, and the light in the specific wavelength band has a peak wavelength in the wavelength band in which the absorption coefficient differs between the oxidized hemoglobin and the reduced hemoglobin. Medical image processing equipment.

(Appendix 11)
The specific wavelength band includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm or less, and light in the specific wavelength band is 400 ± 10 nm, 440 ± 10 nm, 470 ±. A medical image processing apparatus having a peak wavelength in a wavelength band of 10 nm or 600 nm or more and 750 nm or less.

(Appendix 12)
A medical image is an in-vivo image that shows the inside of a living body.
An in-vivo image is a medical image processing device that has information on fluorescence emitted by a fluorescent substance in the living body.

(Appendix 13)
Fluorescence is a medical image processing device obtained by irradiating a living body with excitation light having a peak of 390 nm or more and 470 nm or less.

(Appendix 14)
A medical image is an in-vivo image that shows the inside of a living body.
A specific wavelength band is a medical image processing device that is a wavelength band of infrared light.

(Appendix 15)
The specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less, and the light of the specific wavelength band has a peak wavelength in the wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less. Processing equipment.

(Appendix 16)
The medical image acquisition unit acquires a special optical image having information in a specific wavelength band based on a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in the white band. Equipped with an optical image acquisition unit
Medical images are medical image processing devices that are special optical images.

(Appendix 17)
A medical image processing device that obtains a signal in a specific wavelength band by calculation based on RGB (Red Green Blue) or CMY (Cyan, Magenta, Yellow) color information normally included in an optical image.

(Appendix 18)
By calculation based on at least one of a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band and a special light image obtained by irradiating light in a specific wavelength band. Equipped with a feature amount image generation unit that generates feature amount images,
A medical image is a medical image processing device that is a feature image.

(Appendix 19)
The medical image processing apparatus according to any one of Supplementary note 1 to 18.
An endoscope that irradiates at least one of light in a white wavelength band or light in a specific wavelength band to acquire an image, and
Endoscope device equipped with.

(Appendix 20)
A diagnostic support device including the medical image processing device according to any one of Supplementary note 1 to 18.

(Appendix 21)
A medical business support device including the medical image processing device according to any one of Supplementary note 1 to 18.

<Others>
The above-mentioned endoscopic image learning method is configured as a program for realizing each process on a computer, and a non-temporary recording medium such as a CD-ROM (Compact Disk-Read Only Memory) in which this program is stored is configured. It is also possible to do.

In the embodiments described so far, for example, the hardware structure of the processing unit that executes various processes of the endoscope image learning device 10 and the endoscope image learning device 50 is as shown below. Various processors. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, and a GPU (Graphics Processing Unit), which is a processor specialized in image processing. Dedicated to execute specific processing such as programmable logic device (PLD), ASIC (Application Specific Integrated Circuit), which is a processor whose circuit configuration can be changed after manufacturing FPGA (Field Programmable Gate Array), etc. A dedicated electric circuit or the like, which is a processor having a designed circuit configuration, is included.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types (for example, a plurality of FPGAs, or a combination of a CPU and an FPGA, or a CPU and a processing unit. It may be composed of a combination of GPUs). Further, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as a server and a client. There is a form in which the processor functions as a plurality of processing units. Secondly, as typified by System On Chip (SoC), there is a form of using a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip. be. As described above, the various processing units are configured by using one or more various processors as a hardware-like structure.

Further, the hardware-like structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations and the like in each embodiment can be appropriately combined between the embodiments without departing from the spirit of the present invention.

10 ... Endoscopic image learning device 12 ... Communication unit 14 ... First database 14A ... Normal optical image 14B ... Correct data 16 ... Second database 16A ... Special optical image 16B ... Correct data 18 ... Operation unit 26 ... Display unit 30 ... Machine learning department 32 ... CNN
32A ... Input layer 32B ... Intermediate layer 32C ... Output layer 34 ... Error calculation unit 36 ... Parameter update unit 40 ... Image generation unit 42 ... Generator 44 ... Discriminator 50 ... Endoscopic image learning device 60 ... Image generation unit 62 ... 1st generator 64 ... 1st classifier 66 ... 2nd generator 68 ... 2nd classifier 100 ... Endoscope system 112 ... Endoscope 112A ... Insertion part 112B ... Operation part 112C ... Curved part 112D ... Tip part 112E ... Angle knob 113 ... Mode changeover switch 114 ... Light source device 116 ... Processor device 118 ... Display unit 120 ... Input unit 122A ... First laser light source 122B ... Second laser light source 124 ... Light source control unit 128A ... Optical fiber 128B ... Optical fiber 130 ... Phosphor 132 ... Diffusing member 134 ... Image pickup lens 136 ... Image pickup element 138 ... Analog digital conversion unit 140 ... Imaging control unit 142 ... Image processing unit 144 ... Image acquisition unit 146 ... Image recognition unit 158 ... Display control unit 160 ... Storage control Part 162 ... Memory part

Claims (13)

An image generation unit that generates an endoscopic image of first wavelength band light by a generation model,
A machine learning unit that learns a learning model for image recognition using the endoscopic image generated by the image generation unit.
Equipped with
The image generation unit receives a second endoscope image taken with a second wavelength band light different from the first wavelength band light as an input, and uses the second endoscope image as the first. Converted to the first endoscopic image of wavelength band light,
The endoscopic image learning device in which the first wavelength band light is special light and the second wavelength band light is white light . The endoscope image learning device according to claim 1 , wherein the first endoscope image and the second endoscope image have common correct answer data indicating the correct answer of image recognition. The endoscope according to claim 1 , wherein the correct answer data indicating the correct answer of the image recognition of the first endoscope image is the correct answer data generated by using the correct answer data indicating the correct answer of the second endoscope image. Mirror image learning device. The endoscopic image learning device according to any one of claims 1 to 3 , wherein the learning model detects a lesion region from an input endoscopic image. The endoscopic image learning device according to claim 4 , wherein the learning model classifies whether the lesion region is benign or malignant. The generated model is any one of claims 1 to 5 , which is a generated model obtained by learning using the endoscopic image taken with the first wavelength band light and the second endoscopic image. The endoscopic image learning device according to item 1. The endoscopic image learning apparatus according to any one of claims 1 to 6 , wherein the generative model is trained using a convolutional neural network. The endoscopic image learning device according to claim 7 , wherein the generative model is GAN (Generative Adversarial Networks). The endoscopic image learning device according to any one of claims 1 to 8 , wherein the machine learning unit learns the learning model using a convolutional neural network. An image acquisition unit that acquires an endoscopic image taken with the first wavelength band light, and an image acquisition unit.
An image recognition unit that performs image recognition of the acquired endoscopic image using the learning model learned by the endoscopic image learning device according to any one of claims 1 to 9 .
Endoscopic image recognition device equipped with. An image generation step of generating an endoscopic image of first wavelength band light by a generation model,
A machine learning process for learning a learning model for image recognition using an endoscopic image generated in the image generation process, and a machine learning process.
Equipped with
In the image generation step, a second endoscope image taken with a second wavelength band light different from the first wavelength band light is input, and the second endoscope image is used as the first. Converted to the first endoscopic image of wavelength band light,
The endoscopic image learning method in which the first wavelength band light is special light and the second wavelength band light is white light . A program for causing a computer to execute the endoscopic image learning method according to claim 11 . A recording medium that is non-temporary and computer-readable and causes a computer to execute the program according to claim 12 when a command stored in the recording medium is read by the computer.

Images