WO2018180386A1 - Ultrasound imaging diagnosis assistance method and system - Google Patents

Abstract

An ultrasonic examination system wherein detection precision involving the automatic detection of lesions is increased on the basis of moving images comprising a plurality of temporally continuous frame sequences outputted from ultrasonic examination equipment by moving an ultrasonic probe.　In a learning phase, a model is created by inputting previously cut out images in which a tumor appears together with images other than the aforementioned images, and then, on the basis of the aforementioned images (patch images), classifying images into tumor and non-tumor images using a deep learning method. In an examination phase, regions serving as tumor candidates are detected (S21) by comparing the image in each frame of the moving images with the model obtained in the learning phase (S20). Subsequently, mammary gland tissue is automatically extracted, and the tumor candidate regions in the non-mammary-gland regions are removed (S22). Furthermore, tumor candidate regions sporadically generated using frame continuity are removed (S23), and the ultimately remaining tumor candidate region is output as the detection result.

Description

Ultrasound image diagnosis support method and system

The present invention relates to an ultrasonic diagnostic imaging support method, system, and apparatus.

As a lesion detection for breast ultrasound images, there is B-CAD developed by Canadian company Medipattern (registered trademark).
In B-CAD, a lesion (tumor) that appears as a dark block is targeted, and the examiner (user) specifies the approximate location of the lesion. The contour of the lesion is automatically extracted based on the position information, and the malignancy is calculated based on the shape and size.

Also, focusing on the fact that the lesion appears dark, a method for automatically detecting the lesion using the gradient of the luminance value has been proposed.

In the ultrasonic inspection apparatus, the measurement result is displayed as a moving image.
For example, in a mammary gland ultrasonography, a tumor is depicted as a dark blocky shadow, and can be detected even if each frame of a moving image is treated as an independent still image.

Therefore, the tumor can be detected even by the abnormality detection technique from the still image (pathological image) disclosed in Patent Document 1. However, the shape of the non-mass lesion is not clear, and the texture change indicated by the mammary gland tissue must be observed. Therefore, the approach of the method described in Patent Document 1 cannot be used, and the correlation between the previous and next frames is measured. The moving image pattern recognition is essential.

In Patent Document 2, a position sensor is attached to an ultrasonic probe, and image information and position information are combined to construct three-dimensional data of the internal structure. Based on the surface area and volume ratio of the tumor, the image is a tumor. A technique for determining whether or not is is disclosed.

Patent Document 3 discloses a technique for estimating a position by analyzing an acquired image instead of attaching a position sensor to an ultrasonic probe.

WO2012 / 011579 JP 2000-126182 A JP 2010-166773 A

Lafferty J, McCallum A, Pereira F C: Conditional random fields: Probabilistic models for segmenting and labeling sequence data. Proc of International Conference on Machine Learning: 282-289, 2001 Bell A J, Sejnowski T J: Edges are the "independent components" of natural scenes. NIPS: 831-837, 1996 Kingma D, Ba J: ADAM: A Method for Stochastic Optimization. Proc of International Conference for Learning Representations (ICLR), 2015

However, Medipattern (registered trademark) B-CAD and the method using the gradient of the brightness value assume that the lesion is reflected in the input image. Always overdetect the normal area. Therefore, it cannot be applied when an image showing no lesion is targeted, and is not practical in breast ultrasonography.

In the technique disclosed in Patent Document 2, only a tumor having a clear shape is targeted, and a non-mass lesion having an unclear shape is not targeted.

Further, in the technique described in Patent Document 3, the estimated position information is displayed on the screen as a body mark, and is used only to make it easier for a doctor to grasp the examination site, and is not used for automatic detection of a lesion.

An object of the present invention is to increase the detection accuracy of an ultrasonic inspection system that automatically detects a lesion based on a moving image composed of a plurality of temporally continuous frames output from an ultrasonic inspection apparatus by moving an ultrasonic probe. There is to increase.

(Overview of the proposed system)
The ultrasonic image diagnosis support system or method shown in FIG. 1 includes a learning phase (S10) and an examination phase (S20 to S24).

Once the learning phase is generated and the learning model DB is generated, only the inspection phase can be repeated many times thereafter.
The diagnosis unit is a diagnosis tissue (observation site) and its periphery, and the output includes a display.
In the following, the diagnosis tissue (observation site) is described as a mammary gland tissue and the lesion is a tumor as an example, but the combination is not limited thereto.

In the learning phase (S10), an image showing a previously cut out mass and other images are used as input, and a model that classifies the mass and others using the deep learning method based on those images (patch images) is created. To do.
In the examination phase, a region obtained as a tumor candidate is detected by comparing the model obtained in the learning phase with the image (S20) of each frame of the moving image (S21).
Thereafter, the breast tissue is automatically extracted, and the tumor candidate region in the region other than the mammary gland is removed (S22).
Further, a tumor candidate region that occurs once using the continuity of frames is removed (S23), and the finally remaining tumor candidate region is output as a detection result (S24).

(1)
An ultrasonic diagnostic imaging support method comprising a learning phase (S10) and an examination phase (S20 to S24),
In the learning phase (S10)
The image showing the previously cut out lesion and the other images are input as patch images.
Create a model that classifies the lesion and others using Deep Learning method based on the patch image,
In the inspection phase
A moving image consisting of a plurality of frames of a diagnostic unit including a diagnostic tissue is acquired by operating an ultrasonic probe of an ultrasonic inspection apparatus (S20),
Comparing the model obtained in the learning phase with the image of the frame of the moving image to detect a lesion candidate area of the image of the frame in the diagnosis unit (S21),
Performing automatic extraction of the diagnostic tissue region in the diagnostic unit from the frame image of the moving image, removing the region other than the diagnostic tissue region and the lesion candidate region included therein (S22),
Using the continuity of the sequence of frames to remove the lesion candidate region that occurs only once in the diagnostic tissue (S23),
Only the region of the diagnostic tissue in which the lesion candidate region finally remaining in the image of the moving image frame is marked is output as a detection result (S24),
An ultrasonic diagnostic imaging support method characterized by the above.
(2)
The diagnostic imaging support method according to (1), wherein the diagnostic tissue is a mammary gland tissue, and the lesion is a mass thereof.

(3)
The detection of the lesion candidate area (S21)
A multi-resolution image composed of a plurality of resolution images is created from the moving image frame (S210),
Perform detection processing of the lesion candidate area for each layer image of the multi-resolution image (S211),
Converting the coordinates of the abnormal region in the image of each hierarchy into the coordinates of the original resolution, and integrating the images at each of the plurality of resolutions (S212),
(2) The ultrasonic image diagnosis support method according to (2).
(4)
The ultrasonic image diagnosis support method according to (3), wherein the lesion candidate area detection process (S211) is performed by a sliding window (S211a).
(5)
The ultrasonic image diagnosis support method according to (3), wherein the lesion candidate region detection process (S211) is performed by a super pixel (S211b).

(6)
The ultrasonic image diagnosis support according to any one of (4) and (5), wherein the lesion candidate area is removed (S22) by performing Otsu binarization and automatic extraction of the mammary gland tissue by graph cut Method.
(7)
The ultrasonic image diagnosis support method according to any one of (4) and (5), wherein the lesion candidate region is removed (S22) by automatic extraction of the mammary gland tissue by a CRF (Conditional Random Field) method .

(8)
An ultrasonic diagnostic imaging support system that uses an ultrasonic moving image (hereinafter simply referred to as a moving image) obtained by the operation of an ultrasonic probe of an ultrasonic inspection apparatus,
It consists of a learning phase (S10) and an inspection phase (S20-S24)
In the learning phase (S10)
The image showing the previously cut out lesion and the other images are input as patch images.
Create a model that classifies the lesion and others using Deep Learning method based on the patch image,
In the inspection phase (S20-S24)
A moving image consisting of a plurality of frames of a diagnostic unit including a diagnostic tissue is acquired by operating an ultrasonic probe of an ultrasonic inspection apparatus (S20),
Comparing the model obtained in the learning phase with the image of the frame of the moving image to detect a lesion candidate area of the lesion in the diagnostic unit of the image of the frame (S21);
Performing automatic extraction of the diagnostic tissue region in the diagnostic unit from the frame image of the moving image, removing the region other than the diagnostic tissue region and the lesion candidate region included therein (S22),
Using the continuity of the sequence of frames to remove the lesion candidate region that occurs only once in the diagnostic tissue (S23),
Only the region of the diagnostic tissue in which the lesion candidate region finally remaining in the image of the moving image frame is marked is output as a detection result (S24),
An ultrasonic diagnostic imaging support system characterized by the above.
(9)
The ultrasonic diagnostic imaging support system according to claim 8, wherein the diagnostic tissue is a mammary gland tissue, and the lesion is a mass thereof.

(10)
The detection of the lesion candidate area (S21)
A multi-resolution image composed of a plurality of resolution images is created from the moving image frame (S210),
Perform detection processing of the lesion candidate area for each layer image of the multi-resolution image (S211),
Converting the coordinates of the abnormal region in the image of each hierarchy into the coordinates of the original resolution, and integrating the images at each of the plurality of resolutions (S212),
The ultrasonic image diagnosis support system according to (9), which is performed by:
(11)
The ultrasonic image diagnosis support system according to (10), wherein the lesion candidate region detection process (S211) is performed by a sliding window (S211a).
(12)
The ultrasonic image diagnosis support system according to (10), wherein the lesion candidate area detection process (S211) is performed by a super pixel (S211b).

(13)
The ultrasonic image diagnosis support according to any one of (11) and (12), wherein the lesion candidate region is removed (S22) by performing automatic extraction of the mammary gland tissue by binarization of Otsu and graph cut system.
(14)
The ultrasonic image diagnosis support system according to any one of (11) and (12), wherein the lesion candidate region is removed (S22) by automatic extraction of the mammary gland tissue by CRF.

The accuracy of the ultrasonic image diagnosis support method / system is improved by the combination of the automatic extraction processing of the diagnostic tissue and the improved detection of excessive detection of lesions according to the present invention.
As a result, more ultrasonic image diagnosis support can be accurately performed in a short period of time.

It is a figure showing the flowchart of a proposal system. It is a figure showing a patch image. It is a figure showing the network structure of the Deep-Learning method. It is a figure showing a multi-resolution image. It is a figure showing the method of the raster scan with respect to an image pyramid. It is the figure which compared the coordinate of the tumor candidate area | region in a reduction image and an original image. It is the figure showing the threshold value obtained by the brightness | luminance histogram of a breast ultrasound image, and the binarization of Otsu. It is a figure showing the frame to refer. It is a figure showing the center coordinate of a tumor candidate area | region. It is the figure which compared the overdetection number at the time of applying priority learning, and a detection rate. It is the figure which compared the overdetection number before and after the automatic extraction application of a mammary gland tissue. It is the figure compared before and after the application of the over detection suppression process using the continuity of a frame. It is a figure showing the flowchart of a tumor detection process. It is a figure showing the flowchart of the over detection suppression process other than a mammary gland tissue.

The processing in the learning phase and the inspection phase is described in detail below.

The processing in the learning phase (S10) will be described in detail below.
(Generation of learning patch images)
In many machine learning methods including the Deep Learning method, it is necessary to perform learning using both an image of a detection target (a tumor in the case of the present invention) and other images.
As shown in FIG. 2, in the proposed system, an image cut out into a rectangle (patch image) is used as an image for learning.

In order to appropriately learn a tumor, it is necessary to input an image in which the majority of the tumor is depicted.
Therefore, a patch image of a tumor is created as an abnormal image for learning by perturbing the center of gravity of the tumor within a range where the tumor does not protrude from the patch image.

In addition, normal patch images are created as a normal image of any size by specifying the position with a random number in a frame in which a tumor image of a breast ultrasound image taken as a moving image is not drawn. To do.

(Model learning)
In order to calculate a model for classifying normal and abnormal (tumor), the proposed system uses a machine learning method.
As a concrete method of the machine learning method, a deep learning method such as Deep Belief Network (DBN) or Neural Network by Stacked Denoising Auto Encoder (SDAE) is used.

As other machine learning methods, sequential learning by stochastic gradient descent can be applied, ConvolutionalvolutionNeural Network (CNN) and Support や Vector Machine (SVM), logistic regression analysis, linear discriminant analysis, random forest method, Boosting method (AdaBoost, LogitBoost etc.) may be used.

The Neural Network shown in Fig. 3 consists of links that connect units, and automatically adjusts (updates) the link weights (parameters) based on the patch images for learning, so that normality and abnormalities are determined. Calculate the model to be classified.

(Learning method for unbalanced number of samples)
In deep learning method learning, weights are updated sequentially using optimization methods such as stochastic gradient descent, momentum method, Adam, AdaGrad, and AdaDelta.

Generally, when the number of normal patch images and abnormal patch images is extremely imbalanced, the learning effect is reduced.
For example, when the number of abnormal samples is very small compared to normal samples, there is a possibility that only normal samples can be learned to detect abnormalities.

In order to avoid the influence of this sample number imbalance, the following processing is performed for each parameter update in the learning phase in the proposed system.

(1) Randomly select the same number of normal samples and abnormal samples from the learning data to automatically adjust the network weight, and (2) randomly replace normal samples and abnormal samples at the next parameter update.

(Priority learning)
A normal region (region other than a tumor) of a breast ultrasound image may be similar to a tumor, and this normal region may be erroneously determined as a tumor (overdetection).

Therefore, instead of randomly selecting the normal sample number selection method described above, a normal sample that is easily overdetected is preferentially selected, and the model is updated using the image.

(Pre-training)
In the Deep Learning method, pre-training and fine tuning learning are performed in two stages.

In pre-training, weights are set by unsupervised learning. Thereafter, the weight obtained by the pre-training is used as an initial value, and the weight is updated by a normal learning method such as an error back propagation method (fine tuning).

∙ There are two types of pre-training: Deep Belief Network (DBN) and Stacked Denoising Auto 、 Encode (SDAE). SDAE is effective when priority learning is adopted, and DBN is effective when priority learning is not adopted.

The processing in the inspection phase (S20 to S25) will be described in detail below.

(Frame image acquisition process)
First, an ultrasonic moving image of the subject's observation site and its surroundings for a predetermined time is acquired (S20).
A breast ultrasound moving image of the cut surface in the depth direction obtained when the ultrasound probe is scanned in one direction along the surface of the examination part generally called a breast is composed of multiple frame images arranged in the order of time occurrence. Is done. This image is called a breast ultrasound image.

(Mass detection processing)
FIG. 13 shows a flowchart of the tumor detection process (S21).
In the tumor detection process (S21), the deep learning method is applied to the local region of each frame of the input moving image, and the region determined to be abnormal (tumor) is set as a tumor candidate region.

The detailed procedure of the tumor detection process is shown below.

A multi-resolution image composed of a plurality of resolution images is created from the input image (moving image frame) (S210).

The tumor candidate region detection process is performed on the images of each layer of the multi-resolution image (S211).

The coordinates of the abnormal area in the image of each hierarchy calculated above are converted into the coordinates of the original resolution, and the detection results at each resolution are integrated (S212).
Each process is described in detail below.

(Create multi-resolution image) (S210)
By reducing (or enlarging) the input image, a multi-resolution image having a plurality of resolutions is created.

Specifically, by converting the input image into arbitrary magnifications a ₁ ,..., A _k , a multi-resolution image having k layers is created.

FIG. 4 shows an example of a multi-resolution image when the magnification is set to 1 ×, 0.75 ×, and 0.5 ×.
Note that the Bicubic method was used as the image enlargement / reduction algorithm. The Nearest Neighbour method or Bilinear method can be selected as an image enlargement / reduction algorithm.

(Detection process of tumor candidate area from each layer) (S211)
There are two methods for detecting a tumor candidate region.
First, prepare a rectangular area (search window) that has been set in advance, and perform a raster scan on the image of each layer of the multi-resolution image to determine whether each area is normal or abnormal. This is a determination method (S211a).
The second is a method of dividing an image of each layer into a plurality of regions by the super pixel method and determining whether each region is normal or abnormal (S211b).

(Detection of tumor candidate area by sliding window) (S211a)
In the detection of a tumor candidate area using a sliding window, normality and abnormality are estimated by the following procedure for the images of each layer of the multi-resolution image created above.
(Acquisition of feature vector) The area inside the search window of vertical h (pixel) and horizontal w (pixel) is cut out as a patch image.
The pixel values of the patch image are rearranged in one line and used as a feature vector (hxw dimension). In addition to pixel values, HOG (Histograms of Oriented Gradients) features, LBP (Local Binary Pattern) features, GLAC (Gradient Local AutoCorrelation) features, NLAC (Normal Local AutoCorrelations) features, HLAC (Higher-order Local AutoCorrelation) Feature quantities, feature quantities based on GLCM (Gray Level Correlation Matrix), and Gabor feature quantities can be used.

(Normal / abnormal judgment)
Using the model (weight) calculated in the above “model learning” and the feature vector obtained above, the label (normal or abnormal) of the feature vector obtained above is estimated.

While shifting the search window, the process of “acquisition of feature vector” and “judgment of normal / abnormal” is repeated to scan the entire area in the input image.
The search window is moved by dx (pixel) in the horizontal direction and dy (pixel) in the vertical direction, and the above-mentioned “feature vector acquisition” and “normal / abnormal determination” are repeated.

The coordinates of each search window and the labels at each search window position are accumulated.
Here, the moving widths dx and dy of the search window are made smaller than the size hxw of the search window (for example, dx is half of w and dy is half of h) so as to include a certain amount of overlap with the area once determined. Move the search window.
As a result, a single tumor can be determined using a plurality of search windows whose positions are shifted, so that it can be expected that the detection accuracy of the tumor is improved.

(Determine tumor candidate area)
The region determined to be abnormal above is set as a tumor candidate region, and the upper left coordinate (x0, y0) and the lower right coordinate (x1, y1) of the region are acquired.

(Detection of tumor candidate area by superpixel) (S211b)
In the detection of a tumor candidate area by superpixels, normality and abnormality are estimated by the following procedure for the images of each layer of the multi-resolution image created above.

(Get feature vectors)
A superpixel method is applied to the image to divide the image into a plurality of non-overlapping regions (superpixels).
A feature vector (hxw dimension) is obtained by rearranging the pixel values of the rectangular area of the vertical h and horizontal w centered on the center of gravity of the superpixel into one line. In addition to pixel values, HOG (Histograms of Oriented Gradients) features, LBP (Local Binary Pattern) features, GLAC (Gradient Local AutoCorrelation) features, NLAC (Normal Local Auto-Correlations) features, HLAC (Higher-order Local) AutoCorrelation) feature quantities, feature quantities based on GLCM (Gray Level Correlation Matrix), and Gabor feature quantities can be used.

(Normal / abnormal judgment)
Using the model (weight) calculated in “model learning” and the feature vector obtained above, the label (normal or abnormal) of the feature vector obtained above is estimated.

For all the superpixels, the “feature vector acquisition” and “normal / abnormal determination” processes are applied.
The coordinates of each search window and the labels at each search window position are accumulated.

(Determine tumor candidate area)
The region determined to be abnormal above is set as a tumor candidate region, and the upper left coordinates (x0, y0) and lower right coordinates (x1, y1) of the region are acquired.

(Integration of tumor candidate areas)
As shown in FIG. 6, the coordinates (x0, y0) and (x1, y1) of the tumor candidate region in the image reduced / enlarged by a times are represented by the coordinates (X0, Y0) in the original image using the following equations. And (X1, Y1).

(Excessive detection suppression for non-breast tissue) (S22)
FIG. 14 shows a flowchart of the overdetection suppression process (S22) other than the mammary gland tissue.
Since the tumor occurs in the mammary gland tissue, the region detected in the tissue other than the mammary gland is overdetected in the region detected in the tumor detection process (S21).

In the present invention, in order to remove obvious over-detection other than in the mammary gland, mammary gland tissue is automatically extracted from the breast ultrasound image, and tumor candidate areas other than the mammary gland are removed (S22).
There are two methods of automatic extraction of breast tissue: “Otsu's binarization and automatic extraction of mammary tissue by graph cut (S221)” and “ZCA whitening and automatic extraction of mammary tissue by CRF (S222)”. Any of the methods can be selected.

In the “automatic extraction method of mammary gland tissue by binarizing Otsu and graph cut (S221)”, it is not necessary to learn mammary gland tissue in advance.

On the other hand, “Automatic extraction method of mammary gland tissue by ZCA whitening and CRF (S222)” needs to learn mammary gland tissue in advance, and can automatically extract mammary gland tissue with higher accuracy than “Otsu”.

(Automatic extraction method of mammary gland tissue by binarization of Otsu and graph cut) (S221)
In the automatic extraction method of mammary gland tissue by binarization and graph cut of Otsu, first, binarization processing of Otsu is performed in order to obtain a rough luminance value of the mammary gland tissue.
Next, a graph cut method is performed in order to determine that adjacent regions are the same tissue (mammary gland or non-mammary gland) as much as possible.

(Otsu's binarization)
The breast ultrasound image is divided into strip-shaped regions of width w, and Otsu's binarization is applied to each strip-shaped region.
The average u and the variance σ of the luminance values of the original image in an area (area determined to be white) that is higher than the threshold that is automatically set when Otsu's binarization is applied are acquired.

(Graph cut method)
The breast ultrasound image is divided into M local regions of length h and width w.
When the average luminance value in M local regions is X = [x ₁ , ..., x _M ], the graph cut method uses the label group Y of M local regions that minimizes the following energy function E (Y) = [y ₁ , ..., y _M ] is calculated.

Here, V represents a set of M local regions, and Ni represents an adjacent region in the local region i (in this system, there are 8 adjacent regions).
Φ _u (y _i ) is called a data term, and φ _p (y _i , y _j ) is called a smoothing term, which are defined as follows in this system.

Here, || x || ² represents the L2 norm of x. Λ and k are parameters specified by the user. When λ is increased and k is decreased, adjacent regions are easily determined as the same label. In the present invention, λ = 1 and k = 0.5. Pr (x _i | y _i ) represents the probability of x _i when y _i , and is defined by Equation (4) in the present invention.

(Automatic extraction method of mammary gland tissue by ZCA whitening and CRF) (S222)
In automatic extraction of mammary gland tissue by ZCA whitening and CRF, first, preprocessing by ZCA whitening and interval linear functions is performed (S222a).
Next, the image is divided into local regions, and a luminance histogram is acquired from each local region (S222b).

Finally, a mammary gland tissue is automatically extracted by a conditional random field (CRF) (S222c).
Hereinafter, three processes will be described.

(Image pre-processing) (S222a)
The following two image processes are performed on the breast ultrasound image.

The first is ZCA whitening, which reduces the correlation between depth and luminance values in order to reduce the effect of depth on luminance values. Here, the depth is a distance from the upper end of the image to the pixel. Since the ultrasonic wave is attenuated as it goes deeper into the tissue from the skin and the reflected wave is weakened, the phenomenon that the echo level (brightness on the ultrasonic image) is reduced is reduced.

Secondly, in order to reduce the speckle noise contained in the ultrasound image and emphasize the mammary gland tissue, the brightness value in an arbitrary range is enhanced using a piecewise linear function.

ZCA whitening is a process of making the correlation between variables close to zero in order to eliminate the bias in a plurality of variables having strong correlations.
The luminance value at each of the L pixels in the image is v _i and the depth is d _i (i = 1,..., L).
In this system, the y-coordinate of each pixel when the image upper left as an origin and a depth d _i.

A vector with the brightness value and depth as elements

Apply ZCA whitening to.
T represents transposition.
In ZCA whitening, first, principal component analysis is applied to _L vector groups [t ₁ , ..., t _L ] so that a matrix U having eigenvectors as columns and a pair of eigenvalues λ as diagonal elements. An angle matrix Λ is calculated. Here, Λ = diag (λ ₁ , λ ₂ ) (where λ ₁ > λ ₂ ).
Next, calculate the transformation matrix of ZCA whitening, then the vector after ZCA whitening

Is calculated.
here,

And v _i (with superscript bar) is used as the luminance value after applying ZCA whitening.
In the luminance conversion by the piecewise linear function, the luminance value z _i after the linear conversion is calculated by converting the luminance value by the following equation.

Here, G represents the number of gradations in the converted image.
From preliminary experiments, z _l = -1, z _u = 3, and G = 8.

(Luminance histogram acquisition) (S222b)
In order to capture the brightness in the mammary tissue, a histogram of luminance values is used as a feature vector.
A breast ultrasound image is divided into M rectangular regions (patch images), and a histogram of luminance values is calculated from each patch image.
Since the image has been converted to G gradation, the G-dimensional feature vector is obtained from the patch image of each region divided into M pieces.

Is calculated.

(Conditional random field) (S222c)
The likelihood (likelihood) that each of the M patch images is mammary tissue is calculated.
The mammary gland tissue is continuous, and if any region is mammary gland tissue, the adjacent region is also likely to be a mammary gland (spatial continuity).

Therefore, by considering the spatial continuity by capturing the relationship with adjacent regions, each conditional random field (CRF) that can estimate the tissue (label) depicted in each patch image is used. The mammary gland likelihood is calculated from the region.
Feature vector group X extracted from M patch images and the corresponding label set Y,

(Where y _i = 1 represents mammary tissue and y _i = −1 represents tissue other than mammary gland), CRF is defined by the following probability model.

, Where Z is called the partition function and in order to guarantee 0 = <Pr (Y | X, w) = <1,

It is.
E (X, Y, w) is called an energy function, V represents a set of patch images, Ni represents n neighbors to the patch image i (note that the number of neighbors n can be set arbitrarily) Yes, usually 8 neighbors are used). In the energy function, φ _u is called a data term, and φ _p is called a pair-wise term (smoothing term). In the present invention, φ _u is defined as follows.

Here, δ (y _i ≠ y _j ) is a function that takes 1 when y _i = y _j and takes 0 when y _i ≠ y _j . Also, σ is a parameter set by the user, and σ = 3 in the present invention.
w = [w _u , w _p ] is a learning parameter of CRF. Actually, given N sets of learning images Xn = {x _N } and their label sets Yn = {y _N }, w (with a hat) is calculated by solving the following equation: Use. The following equation can be solved by an optimization method such as the stochastic gradient descent method, the momentum method, Adam, AdaGrad, or AdaDelta.

∙ Feature vector group extracted from M patch images in CRF learning parameter w (with hat) and breast ultrasound image for examination

From each rectangular area (patch image) using

Is calculated.
A larger value means that the degree of mammary glandness is higher. In the present invention, an area of 0.5 or more is determined as a mammary gland tissue.

(Overdetection suppression processing using frame continuity) (S23)
In a breast ultrasound image in which a cross section of the breast is recorded as a moving image, a mass having a volume as a three-dimensional structure is highly likely to be drawn continuously at the same position in a plurality of frames.
Therefore, when a tumor appears, the tumor candidate areas calculated in step S21 are continuously detected at the same position in a plurality of consecutive frames with a very high probability.

On the other hand, since the shadow generated by the influence of speckle noise has no volume, it is not drawn at the same position in multiple consecutive frames, but is only detected as a single tumor candidate region. Should be regarded as overdetection.
In the present invention, in order to suppress over-detection that occurs only once, only the tumor candidate region that occurs at the same position in consecutive frames is treated as the final tumor region.
In the present invention, except for the tumor candidate region detected at the same position in the continuous frame, the masses in a plurality of continuous frames (reference frames) photographed before the frame being observed (target frame) in order to detect the tumor are removed. The position information of the candidate area is used.

Specifically, in the candidate tumor regions in a plurality of consecutive frames, the candidate tumor regions that are isolated in the spatial and temporal directions are removed, and only the candidate tumor regions that are detected in multiple locations in the spatial and temporal directions are finally obtained. As a typical tumor area.

Here, the number of reference frames used for overdetection suppression can be set arbitrarily.
As the number of reference frames used increases, the effect of overdetection suppression increases.
For example, if the number of reference frames used for overdetection suppression is only two frames including the frame of interest, the ultrasound images drawn in those frames are similar, and there is a high possibility that overdetection will occur at the same position. Detection may not be removed.

On the other hand, when the number of reference frames used for overdetection suppression is more than 2 frames (for example, 5 frames including the frame of interest), the shape (pattern) of the drawn tissue varies, so all frames are the same The possibility of occurrence of overdetection at the position is reduced, and overdetection can be appropriately removed.

Increasing the number of reference frames to be used increases the effect of overdetection removal, but there is a risk that the tumor candidate area in which the tumor is detected may be erroneously removed.
For example, when the number of reference frames used for overdetection suppression is 5 frames, if only 2 frames are drawn, only 2 frames of the tumor candidate area are detected. There is a risk of removal.

In this system, in order not to mistakenly remove a tumor candidate region that was detected correctly, the following two measures were taken in advance so that a tumor that could only be drawn with a small number of reference frames could be detected correctly.
The first contrivance is (S211a) "Detection of tumor candidate area by sliding window", the search windows are moved so as to overlap each other, and a plurality of tumor candidate areas are detected at the same position as much as possible.
The second contrivance introduced (S210) “Create multi-resolution image” and used multiple resolution images to detect multiple tumor candidate regions in the same region.

Hereinafter, over-detection suppression using frame continuity will be described in detail.
First, the center coordinates of all the tumor candidate regions between the current frame and the previous frame by the specified value are acquired.

・ Mean shift clustering is performed based on the obtained center coordinates and frame number of the tumor candidate area, and the tumor candidate areas are grouped.

For the calculated group, the number of elements of each group is calculated, and all tumor candidate regions belonging to groups whose number of elements is equal to or less than a preset threshold value are removed.
The remaining tumor candidate area is finally determined as a tumor area.

(Acquisition of coordinates of tumor candidate area)
Let T be the currently displayed frame number and s be the number of frames of interest.
At this time, the center coordinates (Xc, Yc) and the frame number (Tc) of the tumor candidate region c in the T-th s frames (FIG. 8) from the T-s + 1-th frame in the moving image are acquired.

(Grouping by clustering)
The center coordinates (Xc, Yc, Tc) of all the tumor candidate regions acquired above are calculated.

The center coordinates are calculated as shown in FIG. 9 from the upper left coordinates (x0, y0) and lower right coordinates (x1, y1) of the tumor candidate region and the frame number T.

The following mean shift clustering is executed using the center coordinates and frame numbers of all the tumor candidate regions as input vectors, and K group numbers {g ₁ ,..., G _K } are assigned to all the input vectors.

Note that the number K of groups is automatically determined by the mean shift clustering algorithm. In addition to mean shift clustering, a clustering method that can automatically adjust the number of clusters such as the x-means method and Infinite Gaussian Mixture Model (IGMM) can be applied.

(Removing the number of elements in the cluster result)
The number of elements in each group is acquired for {g ₁ , ..., g _K } for the K groups calculated above.
If the number of elements is equal to or less than a preset threshold value Th _n , all abnormal areas belonging to the group are deleted. In the present invention, the value 5 is set as the threshold value Th _n .

(Verification of priority learning)
The effectiveness of the priority learning described in the learning phase (S10) was verified.
In the experiment, breast ultrasound images of 7 patients to be learned and 15 patients for examination were used.

In addition, the network structure based on the Deep Learning method has five layers (the number of units in each layer is 2500, 900, 625, 225, 2 in order from the input layer), and the search window size is 50 pixels vertically, 50 pixels horizontally, and the search window moves The width was 25 pixels in the y direction and 25 pixels in the x direction.

The multi-resolution image has three layers (1x, 0.75x, 0.5x), and in order to verify the effect of priority learning, the over-detection suppression process in steps S22 and S23 is not introduced in the examination phase, and the tumor in step S21 Only candidate regions were used.

FIG. 10 shows the average overdetection number and detection rate per frame.
It can be seen that by introducing priority learning, the number of overdetections is reduced and the detection rate is further increased.
From these results, it is considered that priority learning is effective as a technique for improving detection accuracy.

(Over-detection suppression by automatic extraction of breast tissue)
In order to verify the effectiveness of the over-detection suppression process for non-mammary gland tissue in step S22 in the examination phase, the number of over-detections when the automatic extraction of mammary tissue was not applied and when it was applied were compared.
In the experiment, breast ultrasound images of 7 patients for learning and 5 patients for examination were used.

In addition, the network structure based on the Deep Learning method has four layers (the number of units in each layer is 625, 500, 500, 2 in order from the input layer), the search window size is 50 pixels long, 50 pixels wide, and the movement width of the search window is 25 pixels in the y direction and 25 pixels in the x direction.

The 50x50 image in the search window was reduced to 25x25 by the Bicubic method when it entered the Deep Learning network. The multi-resolution image has 3 layers (1x, 0.75x, 0.5x).

In order to verify the effect of automatic extraction of breast tissue, priority learning in the learning phase and overdetection suppression processing in step S23 are not introduced, detection of a tumor candidate region in step S21 and overdetection suppression processing for other than breast tissue in step S22 Only used.

In addition, in the automatic extraction method of the mammary gland tissue, the automatic extraction method of the mammary gland tissue by binarization of Otsu in step S221 and graph cut was adopted.

Fig. 11 shows a comparison of the number of over-detected experimental results. It can be seen that the automatic extraction technique of the mammary gland tissue is effective in reducing overdetection in tissues other than the mammary gland (non-mammary gland) and suppressing overdetection.

(Verification of overdetection suppression processing using frame continuity)
In order to verify the effectiveness of the overdetection suppression process using the continuity of the frame in step S23 in the inspection phase, compare the overdetection when the overdetection suppression process using the continuity of the frame is not applied and when it is applied. went.

In the experiment, breast ultrasound images of 7 patients for learning and 5 patients for examination were used.

In addition, the network structure based on the Deep Learning method has four layers (the number of units in each layer is 625, 500, 500, 2 in order from the input layer), the search window size is 50 pixels long, 50 pixels wide, and the movement width of the search window is 25 pixels in the y direction and 25 pixels in the x direction.

The 50x50 image in the search window was reduced to 25x25 by the Bicubic method when it entered the Deep Learning network.

The multi-resolution image has 3 layers (1x, 0.75x, 0.5x). In order to verify the effect of the overdetection suppression process using the continuity of the frame, the priority detection in the learning phase and the overdetection suppression process other than the mammary gland tissue in step S22 are not introduced, and the detection of the tumor candidate region in step S21 and the step Only the overdetection suppression process using the frame continuity of S23 was used.

FIG. 12 shows a comparison of the average overdetection number per frame.
It can be seen that the overdetection suppression process using the continuity of frames is applied to reduce overdetection.

1 Abnormality judgment area (lesion candidate area)
2 Abnormality judgment area not included in the area of the observation site
3 Region of observation site
4 Area that is not the area of the observation site
5 Anomaly judgment area that is in the region of the observation site and is not drawn continuously in the still image frame
6 Patch image 7 Learning model DB
8 Observation site and its surroundings (diagnostic part)

Claims (14)

An ultrasonic diagnostic imaging support method comprising a learning phase (S10) and an examination phase (S20 to S24),
In the learning phase (S10)
The image showing the previously cut out lesion and the other images are input as patch images.
Create a model that classifies the lesion and others using Deep Learning method based on the patch image,
In the inspection phase
A moving image consisting of a plurality of frames of a diagnostic unit including a diagnostic tissue is acquired by operating an ultrasonic probe of an ultrasonic inspection apparatus (S20),
Comparing the model obtained in the learning phase with the image of the frame of the moving image to detect a lesion candidate area of the image of the frame in the diagnosis unit (S21),
Performing automatic extraction of the diagnostic tissue region in the diagnostic unit from the frame image of the moving image, removing the region other than the diagnostic tissue region and the lesion candidate region included therein (S22),
Using the continuity of the sequence of frames to remove the lesion candidate region that occurs only once in the diagnostic tissue (S23),
Only the region of the diagnostic tissue in which the lesion candidate region finally remaining in the image of the moving image frame is marked is output as a detection result (S24),
An ultrasonic diagnostic imaging support method characterized by the above. The ultrasonic diagnostic imaging support method according to claim 1, wherein the diagnostic tissue is a mammary gland tissue, and the lesion is a mass thereof. The detection of the lesion candidate area (S21)
A multi-resolution image composed of a plurality of resolution images is created from the moving image frame (S210),
Perform detection processing of the lesion candidate area for each layer image of the multi-resolution image (S211),
Converting the coordinates of the abnormal region in the image of each hierarchy into the coordinates of the original resolution, and integrating the images at each of the plurality of resolutions (S212),
The ultrasonic image diagnosis support method according to claim 2, wherein the ultrasonic image diagnosis support method is performed. The ultrasonic image diagnosis support method according to claim 3, wherein the lesion candidate region detection process (S211) is performed by a sliding window (S211a). The ultrasonic image diagnosis support method according to claim 3, wherein the lesion candidate region detection process (S211) is performed by a super pixel (S211b). The ultrasonic image according to any one of claims 4 and 5, wherein the lesion candidate region is removed (S22) by automatic extraction of the mammary gland tissue by binarization of Otsu and graph cut. Diagnosis support method. 6. The ultrasonic image diagnosis support method according to claim 4, wherein the lesion candidate region is removed (S22) by automatic extraction of the mammary gland tissue by CRF. An ultrasonic diagnostic imaging support system that uses an ultrasonic moving image (hereinafter simply referred to as a moving image) obtained by the operation of an ultrasonic probe of an ultrasonic inspection apparatus,
It consists of a learning phase (S10) and an inspection phase (S20-S24)
In the learning phase (S10)
The image showing the previously cut out lesion and the other images are input as patch images.
Create a model that classifies the lesion and others using Deep Learning method based on the patch image,
In the inspection phase (S20-S24)
A moving image consisting of a plurality of frames of a diagnostic unit including a diagnostic tissue is acquired by operating an ultrasonic probe of an ultrasonic inspection apparatus (S20),
Comparing the model obtained in the learning phase with the image of the frame of the moving image to detect a lesion candidate area of the lesion in the diagnostic unit of the image of the frame (S21);
Performing automatic extraction of the diagnostic tissue region in the diagnostic unit from the frame image of the moving image, removing the region other than the diagnostic tissue region and the lesion candidate region included therein (S22),
Using the continuity of the sequence of frames to remove the lesion candidate region that occurs only once in the diagnostic tissue (S23),
Only the region of the diagnostic tissue in which the lesion candidate region finally remaining in the image of the moving image frame is marked is output as a detection result (S24),
An ultrasonic diagnostic imaging support system characterized by the above. The ultrasonic diagnostic imaging support system according to claim 8, wherein the diagnostic tissue is a mammary gland tissue, and the lesion is a mass thereof. The detection of the lesion candidate area (S21)
A multi-resolution image composed of a plurality of resolution images is created from the moving image frame (S210),
Perform detection processing of the lesion candidate area for each layer image of the multi-resolution image (S211),
Converting the coordinates of the abnormal region in the image of each hierarchy into the coordinates of the original resolution, and integrating the images at each of the plurality of resolutions (S212),
The ultrasonic image diagnosis support system according to claim 9, wherein the ultrasonic image diagnosis support system is performed. The ultrasonic image diagnosis support system according to claim 10, wherein the lesion candidate region detection process (S211) is performed by a sliding window (S211a). The ultrasonic image diagnosis support system according to claim 10, wherein the lesion candidate region detection process (S211) is performed by a super pixel (S211b). The ultrasonic image according to any one of claims 11 and 12, wherein the lesion candidate region is removed (S22) by automatic extraction of the mammary gland tissue by binarization of Otsu and graph cut. Diagnosis support system. The ultrasonic image diagnosis support system according to claim 11 or 12, wherein the lesion candidate region is removed (S22) by automatic extraction of the mammary gland tissue by CRF.

Images