JP6692485B1 - Method, computer and program for generating a difference image based on two images - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for improving the diagnostic accuracy and diagnostic efficiency of a specialist doctor for the diagnosis of changes over time due to rheumatism of bone and changes similar to the changes. SOLUTION: A step of extracting one or more bones from each of a first image and a second image, a first partial image including the first bone extracted from the first image, and an extraction from the second image A computer-implemented method comprising: a difference deriving step 140 for deriving a difference from a corresponding second partial image containing a second bone; and a step 150 for generating a difference image based on the difference. The difference image exists in the first bone but does not exist in the second bone, and a pixel corresponding to a portion that does not exist in the second bone and exists in the second bone but exists in the first bone. The method is characterized in that it is generated so that it can be distinguished from pixels corresponding to the non-existing portion. [Selection diagram] Figure 1

Description

  The present invention relates to medical image processing technology.

  Conventionally, medical radiation images (hereinafter, also referred to as “X-ray images”) have been used, and in many cases, various diagnoses have been made by a specialist doctor. In particular, conventionally, a diagnosis of a bone change over time using an X-ray image has been made only by a specialist doctor.

  More specifically, in order to diagnose changes in bone over time, time-series comparative reading of X-ray images has been performed using an observation device such as Schaukasten. Since the changed portion often overlaps with the complicated normal structure of the human body, it is difficult for even a specialist doctor to recognize the changed portion over time, and the changed portion may be overlooked.

  Further, in the above-described comparative interpretation of X-ray images, even when a time-varying portion is detected, in order to recognize the accurate position, range, degree of change, etc., a plurality of X-ray image There is a problem that the efficiency of diagnosis is poor because it is necessary to select a corresponding region based on observation and knowledge and judge the regions by comparing them with each other.

  In this regard, in Patent Document 1, two images obtained by photographing a subject such as a spine at different times are acquired, and after these images are aligned, the difference between the two images is calculated. A medical image diagnosis support system for calculating a difference image is described.

JP, 2017-63936, A

  Although Patent Document 1 describes that the medical image diagnosis support system described in the document describes that the calculated difference image is an image in which a lesion such as bone metastasis is emphasized, the system does not It is not specialized in assisting in diagnosing changes in rheumatoid diseases over time.

  The present invention has been made in view of the above, and its problem is to improve the diagnostic accuracy and the diagnostic efficiency of a specialist doctor with respect to the diagnosis of changes over time due to rheumatism of bone and changes similar to the changes. It is to provide a method.

  According to an embodiment of the present invention, a step of extracting one or more bones from each of the first image and the second image; a first partial image including the first bone extracted from the first image; A computer-implemented method including a difference deriving step of deriving a difference from a second partial image including the corresponding second bone extracted from the second image, and a step of generating a difference image based on the difference. Then, the difference image exists in the first bone but does not exist in the second bone, and a pixel corresponding to a portion that does not exist in the second bone and the first bone A pixel corresponding to a portion that does not exist in the pixel is generated so that it can be distinguished.

  In one embodiment, the pixel corresponding to a portion that is present in the first bone but not in the second bone, and the pixel that is present in the second bone but is in the first bone. Can have a different color than the pixel corresponding to the non-existing portion.

  According to such a configuration, it is possible to distinguish and identify a part where bones are disappearing and a part where bones are appearing, so that both bone disappearance (exacerbation of symptoms) and appearance (remission of symptoms) occur. It is possible to generate a difference image more suitable for the diagnosis of the obtained disease, for example, rheumatism.

  The first bone and the second bone may be composed of two end portions and an intermediate portion between the two end portions. In one embodiment, the difference deriving step includes A first end image that includes one of the two ends of the first bone and does not include the other and the middle portion; and a corresponding one of the two ends of the second bone, the other and The method may include deriving a difference based on the second end image that does not include the middle part.

According to such a configuration, it is possible to generate a difference image suitable for diagnosis of a disease in which a symptom may occur mainly in the end of the bone, for example, rheumatism, while reducing the calculation amount.
A method according to an embodiment is configured such that a first coordinate set, which is a set of coordinates forming the first bone, is extracted from the first image, and coordinates forming the second bone are extracted from the second image. And extracting a second coordinate set which is a set of the first coordinate set and the second coordinate set based on the first coordinate set and the second coordinate set. A step of performing at least one operation of scaling, wherein when the first partial image and the second partial image for which the at least one operation has been performed on at least one are overlapped, Performing the at least one operation so that the first bone and the second bone on the image overlap, and the first partial image in which the at least one operation has been performed for at least one From the second partial image may further include the step of extracting said first end image and the second end image, respectively.

According to such a configuration, it is possible to obtain two edge images from which a difference should be extracted, including the corresponding portions of the two corresponding bones.
A method according to an embodiment is configured to extract a first contour coordinate set, which is a set of coordinates forming the contour of the first bone, from the first coordinate set, and to extract the second contour set from the second coordinate set. The method may further include the step of extracting a second contour coordinate set, which is a set of coordinates forming the contour of the bone, and the step of performing the at least one operation may include the first contour coordinate set and the second contour coordinate. It can also be set-based.

According to such a configuration, the processing based on the contour coordinate set having a smaller amount of information can be performed, so that the amount of calculation can be reduced.
The method according to an embodiment is configured such that at least one of the first edge image and the second edge image is adjusted so that a correlation between the first edge image and the second edge image is maximized. The method may further include performing at least one operation of translation, rotation, and scaling with respect to the first edge image and the second edge image, the difference based on at least one of the at least one. The difference may be the difference between the first edge image and the second edge image on which one operation has been performed.

With such a configuration, it is possible to exactly align the two end images from which the difference should be extracted.
A method according to an embodiment is the step of generating the difference image for each bone of the one or more bones, and displaying the one or more difference images in an overlapping manner on one of the first image and the second image. The method may further include the step of:

  According to such a configuration, it is possible to provide an image showing the difference between one or more bones in the two images, and thus it is possible to improve the diagnostic accuracy and the diagnostic efficiency of the specialist doctor who is involved in the difference.

Further, according to an embodiment of the present invention, there is provided a computer that executes the above method.
Further, according to an embodiment of the present invention, there is provided a computer that causes a computer to execute the above method.

  ADVANTAGE OF THE INVENTION According to this invention, the diagnostic accuracy and diagnostic efficiency of a specialist doctor can be improved about the diagnosis regarding the time-dependent change by bone rheumatism etc. and the change similar to the said change.

3 is a schematic flow chart of an exemplary method for displaying differences for bones between two images. 6 is a schematic flowchart of an exemplary process for extracting information about bones. 9 is a schematic flowchart of an exemplary process for deriving a relationship between two bones and the like. 7 is a schematic flowchart of an exemplary process for deriving a size ratio of a shooting target on an image. 7 is a schematic flowchart of an exemplary process for deriving a bone inclination. 7 is a schematic flowchart of an exemplary process for deriving a correction angle between two bones. 7 is a schematic flowchart of an exemplary process for deriving a difference between two images. 7 is a schematic flowchart of an exemplary process for performing global alignment. 9 is a schematic flowchart of an exemplary process for performing local alignment. 7 is a schematic flowchart of an exemplary process for deriving a difference. 9 is a schematic flowchart of an exemplary process for displaying an image based on a difference. 3 illustrates the structure of a neural network that is an example of a bone extraction model. FIG. 4 is an exemplary diagram represented by a set of bone contour coordinates. FIG. 4 is an exemplary diagram represented by a set of bone contour coordinates. It is an example figure represented by the union of the outline coordinate set of a bone. The example of 1st edge part _image'1iU is represented. The example of 1st edge part _image'2iU is represented. An example of the positive difference image is shown. An example of a negative difference image is shown. An example of the difference image _iU is shown. Is an example image _j It is an image _j superimposed with the illustrated difference image. It shows an example of the configuration of a computer.

1 One Embodiment of the Present Invention One embodiment of the present invention is a computer-implemented method for displaying the differences between two images obtained by imaging a bone of interest for that bone. .. It is assumed that the bone in the following description is an elongated shape having two ends and is capable of deriving or grasping the central axis, direction, etc., for example, a bone of the hand, but is not limited to this. Do not mean.

1-1 Method Overview FIG. 1 is a schematic flow chart of an exemplary method 100 for displaying differences in bone between two images.

110 shows the step of preparing _two images, image ₁ and image ₂ . The image ₁ and the image ₂ may be derived from X-ray images of hands and the like. Alternatively, the image ₁ and the image ₂ may be derived from MRI images of hands and the like. In other words, image _j may be a grayscale image with brightness information. Note that the preparation of the images ₁ and _2, by cutting a predetermined portion of such X-ray images and MRI images derived and, by such as changing to a predetermined size, to generate an image ₁ and image ₂ May be included. Further, the image ₁ and the image ₂ may be derived from images of the same object taken at different times, and in this case, the difference in the bone between the two images represents a change with time of the bone. However, the image ₁ and the image ₂ are not limited to this. FIG. 16A represents exemplary image ₁ or image ₂ .

_{Reference numeral} 120 denotes a step of extracting information about one or more captured bones _1i and _2i from the images ₁ and ₂ , respectively. Each of the one or more bones _1i and _{2i corresponds} to each bone (first distal phalanx, first proximal phalanx, seed bone ...) When the hand is imaged, for example. Further, 1 ≦ i ≦ n (n is the total number of bones included in the image ₁ or the image ₂ that form the imaging target), and the bones _1i and _2i having the same _i refer to the same bone in the imaging target. It is a thing. Note that the plurality of bones may be regarded as one bone.

_{Reference numeral} 130 denotes a step of deriving the relationship between the bone _1i and the bone _2i in the images ₁ and ₂ . The relationship and the like mentioned here are the relative position and inclination of the bone _1i and the bone _2i , the scale, and the like in the images ₁ and ₂ .

Reference numeral 140 denotes a step of deriving the difference between the bone _1i and the bone _2i in the image ₁ and the image ₂ by using the relationship derived in step 130.
Reference numeral 150 denotes a step of displaying an image based on the difference derived in step 140.

Note that steps 120 to 140 may be executed for all the bones _1i and _2i with 1 ≦ i ≦ n.
1-2 About Extracting Information About Bone FIG. 2 is a schematic flowchart of an exemplary process 200 that can be included in step 120, which is a process for extracting information about bone. Note that j used in the following description is 1 or 2. That is, the example process 200 indicates that the same process is performed on each of the image ₁ and the image ₂ . Also, the example process 200 may be performed for all bones _1i and _2i with 1 ≦ i ≦ n.

210 shows a step of reading a bone extraction model for extracting information about the bone _ji based on the image _j .
Reference numeral 220 denotes a step of obtaining the data _j having a data structure that becomes the input of the bone extraction model read in step 210 from the image _j , for example, the horizontal direction (x-axis direction) of the image to which w and h correspond. And the number of pixels in the vertical direction (y-axis direction), and the value of i _{x, y} is such that the following array I is obtained when the luminance value of the pixel (x, y) of the corresponding image is obtained. Good.

Here, when the luminance value of the pixel (x, y) when arbitrary predetermined image processing is performed on I is i ⁽¹⁾ _{x, y} , i _{x, y} is

May be replaced with a vector such as. Therefore, if i ⁽¹⁾ _{x, y} , ..., I ^(m) _{x, y} is obtained by processing m images, i _{x, y} is

May be replaced with a vector such as. The replaced array I of i _{x, y} becomes the data _j prepared in step 220.
A step 240 extracts a coordinate set B _ji of pixels belonging to the bone _ji in the image _j (hereinafter, referred to as “coordinate set of bone _ji ”) from the data _j using the bone extraction model read in step 210. Shows. However, in another exemplary process, the coordinate set of the bone _ji may be extracted from the image _j instead of the data _j .

260, from the coordinate set _{B ji} bone _ji, shows the step of extracting the outline coordinate set of pixels corresponding to the contour of the bone _ji (hereinafter, referred to as. "Outline coordinates set of bone _ji") a _{C ji.}
According to an exemplary process 200, as an example of information on bone _ji, so that the coordinate set _{B ji} bone _ji, and the outline coordinates set _{C ji} bone _ji are extracted.

1-2-1 About Bone Extraction Model At 240, any bone extraction model may be used to extract a set of bone coordinates. However, in one embodiment of the present invention, information about bones _ji may be extracted by neural net 1200 having the structure depicted in FIG. In such an embodiment, the bone extraction model read in step 210 may include data indicating the weight of each edge connecting the nodes forming the neural network 1200.

The neural network 1200 is roughly divided into two stages (first stage and second stage), and each stage corresponds to a known neural network having a U-net structure.
Reference numeral 1210 indicates a layer that receives an input of the neural network 1200 (hereinafter referred to as “input layer”), and 1220 indicates a layer from which an output is obtained (hereinafter referred to as “output layer”). The number of nodes in the input layer 1210 of the neural network 1200 is equal to the number of nodes in the output layer 1220.

The input of the neural net 1200 may be a vector based on the data _j generated in step 220. For example, the input may be the vector I ′ below.

In other words, the value of each element of the vector I ′ may be input to each node of the input layer 1210. Therefore, the number of nodes in the input layer 1210 (and the output layer 1220) may be h × w × (m + 1).

  On the other hand, the output of the neural network 1200, that is, the value of each node of the output layer 1220 may be represented by the following vector O.

At this time, a predetermined h · w element of the elements of the vector O, for example, an element from o _0,0 to oh _{1, w−1} represents the classification value of the corresponding pixel. The net 1200 can be configured. More specifically, provided a predetermined classification value for each type of bone, for example, the image _j coordinates (x, y) when it constitutes a part of the bone has pixels, o _x, the _y The neural network 1200 can be configured such that the value becomes a predetermined classification value corresponding to the certain bone or a value having a difference from the predetermined classification value that is equal to or less than a predetermined error. The neural net 1200 thus configured can be obtained by training the neural net 1200 using a known method.

1-2-2 Extraction of Bone Contour Coordinate Set At 260, the contour coordinate set C _ji of the bone _ji may be extracted by an arbitrary method, but the following procedure can be taken as an example.

From the elements of the coordinate set B _ji , the end points of the bone _{ji in} the x-axis direction are extracted, and the end points are added to the elements of the contour coordinate set C _ji ,
From the elements of the coordinate set B _ji , the endpoints of the bone _{ji in} the y-axis direction are extracted, and the endpoints are added to the elements of the contour coordinate set C _ji ,
Remove the duplicate elements of the contour coordinate set C _ji .

1-3 Derivation of Relationship between Two Bones, etc. FIG. 3 is a flowchart of an exemplary process 300 that can be included in step 130 that is a process of deriving a relationship between two bones.

Reference numeral 310 denotes a step of deriving the ratio of the sizes of the imaging target between the image ₁ and the image ₂ . Step 310 is the image ₁ and the image ₂ are obtained by photographing the same subject, the photographing conditions, and aims to deal with the case where the size of the images ₁ and ₂ of the target is different It is a thing.

_{Reference numeral} 320 denotes a step of deriving the inclinations S _1i and S _2i of the bone _1i and the bone _2i in the images ₁ and ₂ based on a predetermined definition, respectively.
_{Reference numeral} 330 denotes a step of deriving a correction angle θ between the bones _1i and _2i in the images ₁ and ₂ based on a predetermined definition.

1-3-1 Derivation of Size Ratio of Imaging Target on Image FIG. 4 is a flowchart of an exemplary process 400 that can be included in step 310 that is a process of deriving a size ratio of the imaging target on the image. Shows.

_{Reference numeral} 410 represents a step of deriving the center axis and the center of gravity of the bone _1i based on a predetermined definition, and 420 represents the step of deriving the center axis and the center of gravity of the bone _2i based on the predetermined definition. The central axis of the bone _1i and bone _2i in steps 410 and 420 may derive respectively based on the outline coordinates set _{C 1i} and _{C 2i} bone _1i and bone _2i. Moreover, the x-coordinate of the center of gravity of the bone _1i and bone _2i in steps 410 and 420, respectively, by the average value of x-coordinate of each element included in the outline coordinates set _{C 2i} contour coordinate set _{C 1i} and bone _2i bone _1i be in a, y coordinates of the center of gravity of the bone _1i and bone _2i may be the average of the values of y coordinates of each element included in the outline coordinates set _{C 2i} contour coordinate set _{C 1i} and bone _2i bone _1i.

Reference numeral 430 represents a step of deriving the contour coordinate set C ′ _1i by coordinate-transforming the contour coordinate set C _1i of the bone _1i such that the central axis derived in step 410 becomes vertical and the center of gravity has predetermined coordinates. and, step 440, the derived center axis becomes vertical at step 420, and, as the center of gravity has a predetermined coordinate, derive a contour coordinate set C _'2i outline coordinate set C _2i bone _2i and coordinate transformation Shows the steps to be taken. Note that “vertical” in steps 430 and 440 means parallel to the y-axis.

For reference, exemplary diagrams represented by the contour coordinate set C _ji and the contour coordinate set C ′ _{ji of} the bone _ji are shown in FIGS. 13A and 13B, respectively. 1310 indicates the central axis of the bone _ji , 1320 indicates the center of gravity of the bone _ji , and 1330 indicates the predetermined coordinates.

Reference numeral 450 denotes a step of deriving all the union sets C ′ ₁ from the contour coordinate sets C ′ ₁₁ to C ′ _1n , and 460 shows all the union sets from the contour coordinate sets C ′ ₂₁ to C ′ _2n. shows the step of deriving the C _'2. That is, the union C ′ ₁ = C ′ ₁₁ ∪C ′ ₁₁ ∪ ... ∪C ′ _1n , and the union C ′ ₂ = C ′ ₂₁ ∪C ′ ₂₁ ∪ ... ∪C ′ _2n .

Reference numeral 470 denotes a step of deriving a difference D ₁ between a value having a lower p ₁ % appearance frequency and a value having a higher p ₂ % appearance frequency with respect to y coordinate values of elements included in the union C ′ _1. 480 indicates a step of deriving a difference D ₂ between a value having a lower p ₁ % appearance frequency and a value having a higher p ₂ % appearance frequency with respect to y coordinate values of elements included in the union C ′ _2. There is. It should be noted that p ₁ and p ₂ may each be any percentage greater than 0% and less than 50%. For reference, an exemplary diagram represented by the union C ′ _j is shown in FIG. 1410 shows the value of the y-coordinate whose appearance frequency is the lower p ₁ %, 1420 shows the value of the y-coordinate whose the appearance frequency is the upper p ₂ %, and 1430 shows the difference D _j .

Reference numeral 490 denotes a step of deriving the ratio of D ₁ and D ₂ as the ratio M used in step 310. Step 490 may use D ₂ / D ₁ as M if D ₁ ≦ D ₂ and use D ₁ / D ₂ as M if D ₂ ≦ D ₁ .

According to the example process 400, the size ratio of the imaging target on the image can be calculated in consideration of the sizes of all the bones forming the imaging target on the image.
1-3-1-1 Derivation of central axis of bone An example process flow for deriving the central axis of bone _ji , which can be included in steps 410 and 420, is as follows.

Derivation of each value Y from the minimum value y _min to the maximum value y _max that the y coordinate can take in the element (x, y) of the contour coordinate set C _ji of the bone _ji ,
For each value Y, the average value x _G of the x coordinates of the elements of C _ji where y = Y is obtained, and the set G of (x _G , Y) is obtained,
A regression line obtained by end regression analysis when the y coordinate value of the element of the set G is the explanatory variable and the x coordinate value is the objective variable is the central axis of the bone _ji .

Note that the above processing reduces the calculation amount by deriving the central axis based on the outline coordinate set C _ji having a smaller amount of information, instead of the coordinate set B _ji of the bone _ji .
1-3-2 Derivation of Bone Inclination FIG. 5 is a flowchart of an exemplary process 500 that can be included in step 320, which is a process of deriving a bone inclination.

Reference numeral 510 denotes a step of performing coordinate transformation of the contour coordinate set C _2i of the bone _2i based on the ratio M derived in step 10 to derive a transformed contour coordinate set C ″ _2i . The coordinate transformation is based on the ratio M, as figure represented by the outline coordinates set C _2i bone _2i is enlarged or reduced by moving the coordinates indicating element of the set is to complement the element or elements May be removed. In the step 510, the bone _2i rather, may derive the transformation outline coordinates set C _{'' 1i} outline coordinate set _{C 1i} bone _1i by coordinate transformation.

_{Reference numeral} 520 denotes a step of deriving the central axes of the bones _1i and _2i based on the contour coordinate set C _1i and the transformed contour coordinate set C ″ _2i . Note that when deriving the central axis of the bone _2i by the above-described exemplary process 600, C _ji in the process may be replaced with C ″ _2i . In step 520, the central axes of the bones _1i and _2i are respectively calculated based on the transformed contour coordinate set C ″ _1i and the contour coordinate set C _2i , not the contour coordinate set C _1i and the transformed contour coordinate set C ″ _2i. You may derive it.

530, as the slope _{S 1i} and _{S 2i,} shows the step of deriving respective inclination of the central axis of the bone _1i and bone _2i. It should be noted that this inclination may be represented by an angle formed by the x axis or the y axis and the central axis.

In the example process 500, the central axis of the bone _2i is derived based on the transformed contour coordinate set C ″ _2i is the central axis of the bone _{2i and} the central axis of the bone _2i based on the pre-transformed contour coordinate set C _2i. This is in consideration of the case where and do not exactly match.

1-3-3 Derivation of Correction Angle between Two Bones FIG. 6 is a flowchart of an exemplary process 600 that can be included in step 330, which is a process for deriving a correction angle between two bones.
Step 610 shows a step of deriving a transformed coordinate set B ′ _1i by performing coordinate transformation on the coordinate set B _1i of the bone _1i based on the inclination S _{1i so} that the center of gravity has predetermined coordinates. The part of the coordinate conversion based on the inclination S _1i may be a coordinate conversion in which the central axis of the bone _1i has a predetermined orientation. Further, the x-coordinate of the center of gravity of the bone _1i value may be an average of the values of x coordinates of each element included in the set of coordinates B _1i or contour coordinate set C _1i bone _1i, y coordinate of the center of gravity of the bone _1i The value may be an average of the y-coordinate values of each element included in the coordinate set B _1i of the bone _1i or the contour coordinate set C _1i .

Reference numeral 620 represents a step of deriving a transformed coordinate set B ′ _2i by performing coordinate transformation on the coordinate set B _2i of the bone _2i based on the ratio M and the slope S _{2i so} that the center of gravity has predetermined coordinates. In the coordinate conversion, the portion based on the ratio M moves the coordinates indicated by the elements of the set so that the figure represented by the coordinate set B _2i of the bone _2i is enlarged or reduced based on the ratio M. It may complement elements or remove elements. Further, the portion based on the inclination S _2i may be coordinate conversion such that the central axis of the bone _2i has a predetermined orientation. Furthermore, the value of x-coordinate of the center of gravity of the bone _2i may be an average of the values of x coordinates of each element included in the converted coordinate set B _'2i or converted outline coordinate set C'_'2i, the center of gravity of the bone _2i y The coordinate value may be the average of the y coordinate values of the respective elements included in the transformed coordinate set B ′ _2i or the transformed contour coordinate set C ″ _2i .

Reference numeral 630 denotes a step of deriving a correction angle between the bones _1i and _2i based on the transformed coordinate set B ′ _1i and the transformed coordinate set B ′ _2i . In step 630, specifically, when the transformed coordinate set B ″ _2i is derived by _slightly rotating the transformed coordinate set B ′ _2i by an angle Δφ in the range of −φ to + φ with the center of gravity as the center, Between B ′ _1i and B ″ _2i , there may be included a step of _{obtaining a} rotation angle φ _max at which the degree of similarity quantified by any predetermined definition is maximum, and the obtained rotation angle φ _max is It is the derived correction angle θ. An example of the degree of similarity quantified by an arbitrary predetermined definition is a dice coefficient.

  The exemplary process 600 uses the inclinations of the two previously derived bones to match the approximate orientations of the two bones, and then derives a correction angle that maximizes the overlap of the two bones. The processing speed is improved while maintaining the accuracy of the derived correction angle.

1-4 Derivation of Difference between Two Images FIG. 7 is a flowchart of an exemplary process 700 that can be included in step 140, which is a process of deriving the difference between two images.

_{Reference numeral} 710 represents a step of extracting the partial image _1i including the bone _1i from the image ₁ , and 720 represents a step of extracting the partial image _2i including the bone _2i from the image ₂ . Steps 710 and 720 may be steps of extracting an image in a predetermined range based on the center of gravity of the bone _ji based on the coordinate set B _ji of the bone _ji or the contour coordinate set C _ji . Further, steps 710 and 720 may include the step of deriving a translation parameter (including direction, amount, etc.) for superimposing the partial image _ji on the corresponding partial image _i .

Reference numeral 730 denotes a step of performing global alignment between the bones _1i and _2i based on the partial images 1 _i and _2i .
Reference numeral 740 indicates a step of performing local alignment between the bones _1i and _2i based on the partial images 1 _i and _2i .

Reference numeral 750 denotes a step of deriving a difference between the bone _1i and the bone _2i based on the partial images 1 _i and _2i .
1-4-1 About Global Alignment FIG. 8 is a flowchart of an exemplary process 800 that may include step 730, a process for performing global alignment.

_{Reference numeral} 810 represents a step of deriving the partial image ′ _1i by converting the partial image _1i so that the center of gravity has predetermined coordinates based on the inclination S _1i . In this conversion, the portion based on the inclination S ₁ and the correction angle θ is rotated about the center of gravity about the center of gravity of the partial image _1i by the angle corresponding to the inclination S ₁ , so that the center axis of the bone _{1i in} the partial image ′ _1i is changed. The orientation may be vertical. Further, in this conversion, the one in which the center of gravity is the predetermined coordinate is to translate the partial image _1i in parallel.

820, based on the ratio M, the slope S _1i and correction angle theta, and such that the center of gravity is equal to a predetermined coordinate, represents the step of deriving the partial image _'2i converts the partial images _2i. The portion of the conversion based on the ratio M may be the one that enlarges or reduces the partial image _2i based on the ratio M. Further, in the portion based on the inclination S ₁ and the correction angle θ in this conversion, the partial image ′ _2i is rotated about the center of gravity by an angle corresponding to the inclination S ₁ and the correction angle θ, and the bone _2i in the partial image ′ _2i . The central axis may be oriented vertically. Further, of these conversions, the one in which the center of gravity is the predetermined coordinate is to translate the partial image _2i . Partial image 'conversion to derive _2i includes a bone _2i on the image, partial image' may be those overlapping with the bone _1i on _1i is to be maximized.

1-4-2 About Local Alignment FIG. 9 is a flowchart of an exemplary process 900 that may include step 740, which is a process for performing local alignment.

910, part _{'from 1i} first end _image' image _'shows the step of extracting the _1IL, 920, the partial _{image' 1 IU} and a second end image first end image from _{2i '2 IU} and second It shows a step of extracting the edge image ' _2iL . The bone _ji is assumed to be an elongated shape with two ends and is composed of two ends and an intermediate portion between the two ends, the first end image ' _1iU and the The one edge image ' _2iU includes only one of the two edge portions and does not include the other and the intermediate portion, and the second edge image' _1iL and the second edge image ' _2iL are It is preferable that only the other of the two end portions is included and one and the middle portion are not included. Specifically, steps 910 and 920 are performed by dividing the partial image ' _ji into upper and lower parts with the center of gravity of the bone _{ji as} the center, and further removing the image near the center of gravity to obtain the first end image' _jiU and the second end. It may be a step of extracting the _partial image ' _jiL . The image to be removed may include the bone _ji in the central axis direction at a predetermined ratio, and it is particularly preferable that the image include about 10%.

930 shows the step of generating a first edge image '' 2 _{iU aligned} with the first edge image '1 _iU using the rotation invariant phase only correlation method (RIPOC). The first edge image ' _1iU and the aligned first edge image'' _2iU are, in principle, translated, rotated, and scaled so that the correlation derived by RIPOC is maximized. 2 _iU of the first end image ' _{2iU on} which one or more operations of No. 2 are performed. Step 930 includes the step of deriving parameters (direction, amount, etc.) indicating one or more operations performed. It should be _noted that the first edge image ' _2iU and the first edge image'' _2iU may be the same, in other words, translation, rotation, and scaling may not be performed. Please note.

940, using RIPOC, shows the step of generating a _'2iL' second end image is aligned with _1IL 'second end image. The second end image '' _1iL and the aligned second end image '' _2iL are basically translated, rotated, and scaled so that the correlation derived by RIPOC is maximized. It is the second end image ' _{2iL on} which one or more operations have been performed. Step 940 includes deriving a parameter that is indicative of the amount of one or more manipulations that have been performed. It should be _noted that the second edge image ′ _2iL and the first edge image ″ _2iL may be the same, in other words, translation, rotation, and scaling may not be performed. Please note.

The _purpose of extracting the first end image “ _jiU” and the second end image “ _jiL ” in the example process 900 is to focus on both ends of the bone _ji . Such a method is particularly suitable for diagnosing a disease in which symptoms occur at both ends of bone such as rheumatism. Also, as described above, the example process 900 is performed after global alignment. This preliminary global alignment is for the two images to which the RIPOC is applied to include only substantially the same portion in the central axis direction of the bone _1i and the bone _2i . Further, the derivation of the correlation between images is not limited to that by RIPOC, and the correlation between images can be derived by using an arbitrary method.

1-4-3 Regarding Derivation of Difference FIG. 10 is a flowchart of an exemplary process 1000 that can be included in step 750, which is a process of deriving a difference.

1010 shows a step of deriving a difference D _iU between the first end image “ _1iU and the first end image” _2iU, and 1020 indicates the second end image “ _1iL and the second end image”. 'shows the step of deriving the difference _{D iL} of the _2iL. The derived difference may be the difference between the brightness values of the corresponding pixels in the two images.

1-5 Display of Image Based on Difference FIG. 11 is a flowchart of an exemplary process 1100 that can be included in step 150 that is a process for displaying an image based on a difference.

1110 indicates a step of generating a difference image _iU based on the difference D _iU , and 1120 indicates a step of generating a difference image _iL based on the difference D _iL . When the difference is represented by the following array D, the color (including transparency) of the pixel at the coordinates (x, y) of the generated difference image may correspond to the value of d _{x, y} .

More specifically, the color of the pixel at the coordinates (x, y) of the generated difference image differs depending on whether the value obtained by subtracting a predetermined value from the value of d _{x, y} is positive or negative. Good. For example, when the value is positive, the pixel color may be red, and when the value is negative, the pixel color may be blue. According to such a configuration, when the difference image is generated based on two images derived from images of the same target taken at different times, the color of the difference image exacerbates, for example, the symptoms of rheumatism and the bones of the past. It is possible to distinguish between a part which is present in the bone but is not present in the present bone and a part which is present in the present bone but is not present in the past bone, for example, in which the symptoms of rheumatism are ameliorated.

Further, the transparency of the color of the pixel at the coordinates (x, y) of the generated difference image may be a value corresponding to the absolute value of a value obtained by subtracting a predetermined value from the value of d _{x, y} . For example, the greater the absolute value of the value, the smaller the transparency of the color of the pixel. It should be noted that a positive difference image in which the color of a pixel in which the value obtained by subtracting a predetermined value from the value of d _{x, y} is not positive is transparent and a negative difference image in which the color of a pixel whose value is not negative is transparent are separately provided. May be generated. Further, the predetermined value may be zero.

15A to 15E respectively show examples of a first end image “ _1iU” , a first end image “ _2iU , a positive difference image, a negative difference image, and a difference image _iU . _{Reference numeral} 1510 denotes a bone portion which is present in the first end image ' _1iU but not present in the first end image' _2iU , and 1520 is a first end image present in the first end image ' _2iU. 'It shows a bone part that does not exist in 1 _iU . In practice, it is preferable that the portions 1510 and 1520 have different colors.

Reference numeral 1130 denotes a step of generating an image in which the difference image _iU and the difference image _iL are superimposed on _one of the image ₁ and the image ₂ . This step corresponds the difference image _iU and the difference image _iL to _one of the images ₁ and ₂ based on the ratio M, the slope S _ji , the correction angle θ, the parameters derived in steps 710, 720, 930 and 940, and the like. Including the step of overlaying the portions. Figure 16B represents an example of an image superimposed, 1610 is present in the image ₁ is shown a portion of the bone which is not present in the image _2, 1620 present in the image ₁ is present in the image ₂ Not showing the bone part. In practice, it is preferable that the portion 1610 and the portion 1620 are displayed in different colors.

1140 shows the step of displaying the superimposed images.
Note that another exemplary method may include directly displaying a difference image including one or both of the positive difference image and the negative difference image described above, instead of steps 1130 and 1140.

2 Another Embodiment of the Present Invention Another embodiment of the present invention is a computer that executes a method for displaying the difference between two images obtained by imaging a bone of a hand or the like, Alternatively, it is a program for causing a computer to execute the method.

3 Computer Hereinafter, an example of a hardware configuration of a computer that can be used for implementing one embodiment of the present invention will be described. It should be noted that any computer can be used to implement the embodiment of the present invention, and examples thereof include a personal computer, a tablet computer, a smartphone, and a computer on a cloud.

  FIG. 17 shows an example of the hardware configuration of a computer. As shown in the figure, the computer 1700 mainly includes a processor 1710, a main storage device 1720, an auxiliary storage device 1730, an input / output interface 1740, and a communication interface 1750 as hardware resources. These are connected to each other via a bus line 1760 including an address bus, a data bus, a control bus and the like. An interface circuit (not shown) may be appropriately interposed between the bus line 1760 and each hardware resource.

  The processor 1710 controls the entire computer. Note that one computer may include a plurality of processors 1710. In such a case, the “processor” in the above description may be a general term for the plurality of processors 1710.

  The main storage device 1720 provides a work area to the processor 1710, and is a volatile memory such as SRAM (Static Random Access Memory) and DRAM (Dynamic Random Access Memory).

  The auxiliary storage device 1730 is a non-volatile memory such as an HDD, an SSD, or a flash memory that stores software programs and data. The program, data, etc. are loaded from the auxiliary storage device 1730 to the main storage device 1720 via the bus line 1760 at any time. Auxiliary storage 1730 may be referred to as a non-transitory computer-readable storage medium. The program includes an instruction that causes the processor to execute a desired process.

  The input / output interface 1740 is for presenting information and / or receiving information input, and is a digital camera, keyboard, mouse, display, touch panel display, microphone, speaker, temperature sensor, or the like. is there.

  The communication interface 1750 is connected to the network 1770 and transmits / receives data via the network 1770. The communication interface 1750 and the network 1770 can be connected by wire or wirelessly. The communication interface 1750 may also acquire information related to the network, for example, information related to the Wi-Fi access point, information related to the base station of the communication carrier, and the like.

  It will be apparent to those skilled in the art that the computer 1700 can function as a desired means, perform a desired step, and realize a desired function by the cooperation of the hardware resource and the software exemplified above. Let's see

4 Conclusion Although some examples of the embodiments of the present invention have been described above, it should be understood that these are merely examples and do not limit the technical scope of the present invention. It should be understood that modifications, additions, improvements, and the like of the embodiments can be appropriately made without departing from the spirit and scope of the present disclosure. The technical scope of the present invention should not be limited by any of the above-described embodiments, but should be defined only by the claims and their equivalents.

1200 ... Neural net 1210 ... Input layer 1220 ... Output layer 1310 ... Center axis 1320 ... Center of gravity 1330 ... Predetermined coordinate 1410 ... Value of y coordinate where appearance frequency is lower p ₁ % 1420 ... Appearance frequency is higher p ₂ % _'Although the _1iU present first end _image' values 1430 ... difference 1510 ... first end image of y-coordinate in the portion 1520 ... first end image _{'2 IU} of bone that are not present in the _{2 IU} exists but the portion of the bone is not present in the image ₁ to the first end image _{'1 IU} present in the portion 1610 ... image ₁ of a bone that does not exist is present in the portion 1620 ... image ₂ of the bone which is not present in the image ₂

Claims (21)

Extracting one or more bones from each of the first image and the second image;
A difference deriving step of deriving a difference between a first partial image including the first bone extracted from the first image and a second partial image including the corresponding second bone extracted from the second image;
Generating a difference image based on the difference
A computer-implemented method, including:
The difference image exists in the first bone but does not exist in the second bone, and a pixel corresponding to a portion that does not exist in the second bone but does exist in the first bone. It is generated so that it can be distinguished from the pixel corresponding to the part that does not exist.
It features a, the first bone and the second bone, and two end portions, is composed of an intermediate portion between the two ends, the difference deriving step, the first bone A first end image that includes one of the two ends of the second bone and does not include the other and the middle part, and a corresponding one of the two ends of the second bone that includes the other and the middle part. Deriving a difference based on the second edge image that is not present. The method according to claim 1, wherein the pixel corresponding to a portion that is present in the first bone but not in the second bone and is present in the second bone, The method, wherein the pixel corresponding to a portion not present in the first bone has a different color. The method according to claim 1 or 2 , wherein
A first coordinate set, which is a set of coordinates forming the first bone, is extracted from the first image, and a second coordinate set is a set of coordinates forming the second bone, from the second image. To extract
A step of performing at least one operation of translation, rotation, and scaling on at least one of the first partial image and the second partial image based on the first coordinate set and the second coordinate set. And when the first partial image and the second partial image for which the at least one operation has been performed on at least one are overlapped, the first bone and the second bone on the image are overlapped. Performing the at least one operation such that
Extracting each of the first edge image and the second edge image from the first partial image and the second partial image on which at least one operation has been performed on at least one of them. .. The method of claim 3 , wherein
From the first coordinate set, a first contour coordinate set, which is a set of coordinates forming the outline of the first bone, is extracted, and from the second coordinate set, a set of coordinates forming the outline of the second bone is extracted. Further comprising the step of extracting a second contour coordinate set that is a set, wherein the step of performing the at least one operation is also based on the first contour coordinate set and the second contour coordinate set.
Method. The method according to claim 3 or 4 , wherein
Translation, rotation, and enlargement with respect to at least one of the first edge image and the second edge image so that the correlation between the first edge image and the second edge image is maximized. The method further includes performing at least one operation of reduction, wherein the difference based on the first edge image and the second edge image is the first edge on which the at least one operation is performed. The method, which is the difference between the partial image and the second end image. A method according to any one of claims 1 to 5 , wherein
Generating the difference image for each bone of the one or more bones;
Overlaying one or more of the difference images on one of the first image and the second image. Extracting one or more bones from each of the first image and the second image;
A difference deriving step of deriving a difference between a first partial image including the first bone extracted from the first image and a second partial image including the corresponding second bone extracted from the second image;
Generating a difference image based on the difference
A computer running
The difference image exists in the first bone but does not exist in the second bone, and a pixel corresponding to a portion that does not exist in the second bone but does exist in the first bone. It is generated so that it can be distinguished from the pixel corresponding to the part that does not exist.
It features a, the first bone and the second bone, and two end portions, is composed of an intermediate portion between the two ends, the difference deriving step, the first bone A first end image that includes one of the two ends of the second bone and does not include the other and the middle part, and a corresponding one of the two ends of the second bone that includes the other and the middle part. A computer comprising deriving a difference based on the second edge image that is not present. 8. The computer according to claim 7 , wherein the pixel corresponding to a portion that is present in the first bone but not in the second bone and is present in the second bone, A computer having a different color from the pixel corresponding to a portion that does not exist in the first bone. The computer according to claim 7 or 8 , wherein
A first coordinate set, which is a set of coordinates forming the first bone, is extracted from the first image, and a second coordinate set is a set of coordinates forming the second bone, from the second image. To extract
A step of performing at least one operation of translation, rotation, and scaling on at least one of the first partial image and the second partial image based on the first coordinate set and the second coordinate set. And when the first partial image and the second partial image for which the at least one operation has been performed on at least one are overlapped, the first bone and the second bone on the image are overlapped. Performing the at least one operation such that
And further extracting each of the first end image and the second end image from the first partial image and the second partial image on which at least one operation has been performed on at least one of them. Computer. The computer according to claim 9 ,
From the first coordinate set, a first contour coordinate set, which is a set of coordinates forming the outline of the first bone, is extracted, and from the second coordinate set, a set of coordinates forming the outline of the second bone is extracted. Further performing a step of extracting a second contour coordinate set, the step of performing the at least one operation is also based on the first contour coordinate set and the second contour coordinate set,
Computer. The computer according to claim 9 or 10 , wherein
Translation, rotation, and enlargement with respect to at least one of the first edge image and the second edge image so that the correlation between the first edge image and the second edge image is maximized. The step of performing at least one operation of reduction is further executed, and the difference based on the first edge image and the second edge image is the first at which the at least one operation has been performed on at least one. A computer, which is the difference between the edge image and the second edge image. The computer according to any one of claims 7 to 11 ,
Generating the difference image for each bone of the one or more bones;
And a step of superimposing one or more of the difference images on one of the first image and the second image. Extracting one or more bones from each of the first image and the second image;
A difference deriving step of deriving a difference between a first partial image including the first bone extracted from the first image and a second partial image including the corresponding second bone extracted from the second image;
Generating a difference image based on the difference
A program that causes a computer to execute
The difference image exists in the first bone but does not exist in the second bone, and a pixel corresponding to a portion that does not exist in the second bone but does exist in the first bone. It is generated so that it can be distinguished from the pixel corresponding to the part that does not exist.
It features a, the first bone and the second bone, and two end portions, is composed of an intermediate portion between the two ends, the difference deriving step, the first bone A first end image that includes one of the two ends of the second bone and does not include the other and the middle part, and a corresponding one of the two ends of the second bone that includes the other and the middle part. A program comprising deriving a difference based on the second edge image that is not present. The program according to claim 13 , wherein the pixel exists in the first bone but does not exist in the second bone, and the pixel exists in the second bone, A program having a different color from the pixel corresponding to a portion that does not exist in the first bone. The program according to claim 13 or 14 , wherein
A first coordinate set, which is a set of coordinates forming the first bone, is extracted from the first image, and a second coordinate set is a set of coordinates forming the second bone, from the second image. To extract
Performing at least one operation of translation, rotation, and scaling on at least one of the first partial image and the second partial image based on the first coordinate set and the second coordinate set. And when the first partial image and the second partial image for which the at least one operation has been performed on at least one of them are overlapped, the first bone and the second bone on the image are overlapped. Performing the at least one operation such that
A step of extracting the first end image and the second end image from the first partial image and the second partial image on which at least one operation has been performed on at least one, respectively; The program to run. The program according to claim 15 ,
From the first coordinate set, a first contour coordinate set, which is a set of coordinates forming the outline of the first bone, is extracted, and from the second coordinate set, a set of coordinates forming the outline of the second bone is extracted. Causing the computer to further execute a step of extracting a second contour coordinate set that is a set, and the step of performing the at least one operation is also based on the first contour coordinate set and the second contour coordinate set.
program. The program according to claim 15 or 16 , wherein
Translation, rotation, and enlargement with respect to at least one of the first edge image and the second edge image so that the correlation between the first edge image and the second edge image is maximized. The step of performing at least one operation of reduction is further executed by the computer, and the difference based on the first edge image and the second edge image is the at least one operation performed by the at least one operation. A program that is a difference between a first edge image and the second edge image. The program according to any one of claims 13 to 17 ,
Generating the difference image for each bone of the one or more bones;
And a step of superimposing one or more of the difference images on one of the first image and the second image and displaying the difference image. Extracting one or more bones from each of the first image and the second image;
A difference deriving step of deriving a difference between the first partial image including the first bone extracted from the first image and the second partial image including the corresponding second bone extracted from the second image;
A computer-implemented method, the first bone and the second bone comprising two ends and an intermediate portion between the two ends, wherein the difference deriving step Includes a first end image that includes one of the two ends of the first bone and does not include the other and the middle, and a corresponding one of the two ends of the second bone. , Deriving a difference based on the other and the second edge image that does not include the middle portion. Extracting one or more bones from each of the first image and the second image;
A difference deriving step of deriving a difference between the first partial image including the first bone extracted from the first image and the second partial image including the corresponding second bone extracted from the second image;
A computer for execution, the first bone and the second bone, and two end portions, is composed of an intermediate portion between the two ends, the difference deriving step, the second A first end image that includes one of the two ends of one bone and does not include the other and the middle, and a corresponding one of the two ends of the second bone that includes the other and the middle A computer comprising the step of deriving a difference based on a second edge image that does not include a copy. Extracting one or more bones from each of the first image and the second image;
A difference deriving step of deriving a difference between the first partial image including the first bone extracted from the first image and the second partial image including the corresponding second bone extracted from the second image;
A program for causing a computer to execute: the first bone and the second bone are composed of two end portions and an intermediate portion between the two end portions, and the difference deriving step includes: A first end image that includes one of the two ends of the first bone and does not include the other and an intermediate portion, and a corresponding one of the two ends of the second bone, and the other And a step of deriving a difference based on the second end image not including the middle part.