JP6996145B2 - Image display device, program and control method - Google Patents

Description

The present invention relates to a technique for displaying an image of an object such as a fertilized egg.

In the field of infertility treatment, the implantation rate of embryos is predicted from the viewpoint of the treatment rate and the QOL (Quality Of Life) of the patient, and the embryos to be transplanted are selected. Conventionally, as a method for non-invasively evaluating embryos, morphological observation with a microscope and determination of developmental status from the morphology at that time have been performed.

In recent years, with the spread of time-lapse devices, it has become possible to capture changes over time in addition to the above, and it is also possible to analyze multiple growth prediction indicators and predict growth using a large number of images. Has become. In order for embryo culture specialists to observe such a large number of images and evaluate cell quality, an easy-to-use image viewing system is required. Patent Document 1 discloses a technique for displaying a well having a high occupancy rate based on an estimation result of the installation direction of the culture vessel and the position of the well when the presence of the culture vessel provided with a plurality of wells is detected. Has been done.

Japanese Patent No. 60078182

In a time-lapse device, a fertilized egg of the same patient is usually stored in a plurality of wells in one container, and a plurality of containers can be stored and observed at the same time. Here, when an embryo culture person observes an image of a fertilized egg and evaluates the quality morphologically, he / she usually selects a specific patient from a plurality of containers, and further, a plurality of fertilizations in the container. It is necessary to display an image for each egg and evaluate the quality.

Here, when evaluating the quality of a fertilized egg, it is good to display a large image without loss of information on the screen, and it is common to display a large image of one fertilized egg on the screen. Is. At this time, in order to display the image as large as possible without occupying the screen displaying a wide variety of information excessively, it has been done to cut out and display the captured image, but the narrower the cutout range is, the larger the image is. While it is easy to display, there is a problem that the fertilized egg is easy to frame out from the cut out image. Further, when the display position of the fertilized egg or the like in the captured image in the time series shifts due to the deviation of the XY position during time lapse observation or the movement of the fertilized egg itself, there is also a problem that it is difficult to see when playing back the moving image.

An object of the present invention is to provide an image display device capable of suitably improving the visibility of a display object.

From one aspect of the present invention, the image display device has an acquisition unit that acquires a plurality of captured images of an object captured at a plurality of focal positions at a plurality of times, and a display unit that displays a part of the captured image. A display control unit is provided, and the display control unit adjusts the amount of deviation between the shooting range of the shot image corresponding to the focal position designated by the external input and the shooting range of the reference image set for each focus position. Based on this, the display range of the captured image is determined, and the determined image of the display range is displayed on the display unit in chronological order of capture .

In one aspect of the image display device, the display control unit determines the position of the display range defined in a predetermined size based on the deviation amount.

In another aspect of the image display device, the object is housed in a recess, and the acquisition unit acquires the captured image including the recess in the imaging range.

In another aspect of the image display device, the display control unit has the deviation amount so that the object or the recess in which the object is housed in the captured image is located near the center of the display range. The display range is determined based on. Here, "located near the center of the display range" means that the display range does not exist unevenly at the edges (for example, four corners).

In another aspect of the image display device, the display control unit displays information related to the captured image or information related to the object at a position within the display range that does not overlap with the display position of the object. , Displayed on the display unit.

In another aspect of the image display device, the acquisition unit acquires an image obtained by repeatedly photographing a plurality of objects by changing the position of the photographing unit for each object to be photographed as the captured image.

From another aspect of the present invention, the program includes an acquisition unit that acquires a plurality of captured images of an object captured at a plurality of focal positions at a plurality of times, and a display control unit that displays a part of the captured images on the display unit. The display control unit is based on the amount of deviation between the shooting range of the shot image corresponding to the focal position specified by the external input and the shooting range of the reference image set for each focus position. The display range of the captured image is determined, and the determined image of the display range is displayed on the display unit in chronological order of capture.

Another aspect of the present invention is a control method executed by the image display device, which is an acquisition step of acquiring a plurality of captured images obtained by capturing an object at a plurality of focal positions at a plurality of times, respectively, and the captured image. The display control step includes a display control step of displaying a part of the image on the display unit, and the display control step includes a shooting range of the shot image corresponding to a focal position designated by an external input and a reference set for each focus position. The display range of the captured image is determined based on the amount of deviation from the image capture range, and the determined image of the display range is displayed on the display unit in chronological order of capture.

According to the image display device according to the present invention, the object is deviated from the photographed image by determining the display range of the photographed image based on the deviation amount of the image-taking range displayed on the photographed image of the object. It is possible to prevent frame-out and improve observability. Further, when the captured image is displayed as a moving image on the display unit in chronological order, it can be smoothly displayed as a moving image without rattling, so that the observability is also excellent.

The configuration of the image display system of the fertilized egg according to the embodiment is shown. An example of a photographed image of a culture vessel and a fertilized egg is shown. It is a figure explaining the method of photographing a fertilized egg. This is a display example of the image viewing screen. The outline of the stabilization process using the reference image is shown. The outline of the first and second examples of the stabilization process considering the Z position is shown. The outline of the 3rd and 4th examples of the stabilization process considering the Z position is shown.

Hereinafter, preferred embodiments of the present invention will be described.

[System configuration]
FIG. 1 shows a configuration of an image display system for a fertilized egg according to an embodiment. As shown in the figure, the image display system 100 is roughly classified into one or a plurality of photographing devices 50, a storage device 55, and an image display device 60.

The photographing apparatus 50 has an incubator 9 and photographs and stores an image of a fertilized egg in a culture vessel 10 placed in a chamber (chamber) (not shown) provided in the incubator 9. A plurality of fertilized eggs are placed in each culture container 10, and the photographing apparatus 50 associates them with identification information for identifying the fertilized eggs, for example, identification information of a well containing the fertilized eggs. The captured image is transmitted to the storage device 55. Wells are an example of the "recess" of the present invention. Specifically, the captured image 50 records the XY coordinates corresponding to the positions of the wells in the culture vessel 10 in advance, and each well may be identified from the XY coordinate information, or at the start of imaging, etc. In addition, the XY coordinates associated with each well position may be finely adjusted and the coordinates may be individually registered as the well position.

The photographing device 50 includes a lighting device 51, a camera 52, a control unit 53, and a communication unit 54. The lighting device 51 illuminates the fertilized egg in the culture vessel 10 from above for photographing. The lighting device 51 may be a point light source, a surface light source, a plurality of point light sources, or a surface light source. The lighting by the lighting device 51 is controlled by the control unit 53.

The camera 52 is movably arranged below the culture vessel 10 and photographs the fertilized egg in a state of being illuminated by the lighting device 51. The camera 52 is moved by an actuator (not shown) or the like. The movement of the camera 52 by the actuator and the shooting by the camera 52 are controlled by the control unit 53. The camera 52 photographs a plurality of fertilized eggs in the same culture vessel 10 at different times. When a plurality of culture containers 10 are placed in the incubator 9, the camera 52 moves each fertilized egg in each culture container 10 to a position where it can be photographed based on the control of the actuator by the control unit 53. And shoot. The fertilized egg photographed by the camera 52 is an example of the "object" of the present invention. When photographing a plurality of fertilized eggs in the same culture vessel 10, the structure may be such that the culture vessel 10 moves instead of the camera 52.

The control unit 53 controls lighting by the lighting device 51, movement of the camera 52, and shooting by the camera 52. Further, when the control unit 53 acquires the captured image from the camera 52, the control unit 53 controls the communication unit 54 to transmit the captured image to the storage device 55.

The storage device 55 has a control unit, a storage unit, a communication unit, and the like (not shown), receives a photographed image transmitted from each photographing device 50, and stores the received photographed image. In this case, for example, the storage device 55 provides a folder for each incubator 9 provided in the photographing device 50, and stores the photographed image in association with the corresponding folder. The storage device 55 may further provide a subfolder for each chamber with respect to the above-mentioned folder, and classify and store the captured image for each chamber. Further, when the storage device 55 receives the acquisition request of the captured image from the image display device 60, the storage device 55 transmits the captured image specified by the acquisition request to the image display device 60.

The image display device 60 is configured by a PC (Personal Computer) or the like, and displays a photographed image of a fertilized egg designated by a user such as an embryo culture person on a predetermined display screen (also referred to as an “image viewing screen”). The image display device 60 mainly includes a communication unit 61, an input unit 62, a storage unit 63, a display unit 64, and a control unit 65.

The communication unit 61 receives the captured image from the storage device 55 under the control of the control unit 65. The input unit 62 is a keyboard, a mouse, a wheel device capable of rotation operation, a touch panel, a voice input device, and the like, and generates an input signal for designating a captured image to be displayed and supplies the input signal to the control unit 65. The storage unit 63 stores the program executed by the control unit 65 and the information necessary for executing the program. For example, the storage unit 63 stores a program required for display processing of the image viewing screen. The display unit 64 displays the image viewing screen based on the control of the control unit 65. The control unit 65 controls the entire image display device 60. The control unit 65 is an example of the “acquisition unit” and the “display control unit” of the present invention. Further, the control unit 65 is an example of a "computer" that executes the program of the present invention.

[Shooted image]
FIG. 2 shows an example of a photographed image of the culture vessel 10 and the fertilized egg. The culture container 10 is a circular container, and a storage portion 11 is formed in the central portion. A plurality of (5 × 5 = 25 in the example of FIG. 2) wells 12 are formed in the accommodating portion 11. The well 12 is a recess for accommodating a fertilized egg, and one fertilized egg is accommodated in one well 12. As shown in FIG. 2A, the accommodating portion 11 is filled with the culture solution 13.

FIG. 2B shows a photographed image 20 of a portion of one well 12. When the fertilized egg is housed in the well 12, the photographed image 20 includes the well 12 and the fertilized egg 30 in the well 12. The photographing apparatus 50 generates and stores a plurality of photographed images of the same fertilized egg (that is, the same well 12) at predetermined time intervals. The photographing device 50 shall capture an image for each well 12.

FIG. 3 is a diagram illustrating a method of photographing a fertilized egg. FIG. 3A is a view of the accommodating portion 11 in one culture vessel 10 viewed from above. In the accommodating portion 11, five wells 12 are formed in the horizontal direction (X direction) and five in the vertical direction (Y direction), for a total of 25 wells 12. Hereinafter, assuming that the position in the X direction is indicated by "1" to "5" and the position in the Y direction is indicated by "A" to "E", the well 12 on the upper left of FIG. 3 (A) is the position "A1". (Hereinafter, also referred to as "well A1"). In the example of FIG. 3A, the fertilized egg 30 is housed in 13 wells 12 from wells A1 to C3. The arrangement of the wells 12 formed in the culture vessel 10 is not limited to the arrangement shown in FIG. 3A, and different numbers of wells 12 may be formed in the culture vessel 10 by a predetermined arrangement.

At the time of photographing the fertilized egg, for example, the lighting device 51 provided for each chamber is arranged at a predetermined fixed position, and the camera 52 moves to photograph the fertilized egg in each well 12. For example, the camera 52 first photographs the well A1, then moves in the X direction to photograph the well A2, then moves in the X direction to photograph the well A3, and so on, and so on, each well 12 is photographed. After moving in the X direction and taking pictures of the wells A1 to A5, the camera 52 moves in the Y direction to take pictures of the wells B1 to B5. In this way, the camera 52 takes pictures of all the wells 12 containing the fertilized eggs while moving in the X direction and the Y direction. Then, the camera 52 repeatedly photographs the fertilized eggs in each well 12 at predetermined time intervals by repeating the above-mentioned movement.

Here, when the camera 52 shoots the same well 12 again after a predetermined time, the shooting position may be slightly deviated from the previous time, whereby the fertilized egg and the fertilized egg and the fertilized egg when the shot image is displayed as a moving image in chronological order. It may cause the display position of the well to be blurred. In the present embodiment, the image display device 60 takes into consideration the position reproducibility of the camera 52 and performs a stabilization process described later when displaying a moving image of the captured image.

The imaging device 50 can also move the camera 52 in the vertical direction (Z direction) of the culture vessel 10. By moving the camera 52 in the Z direction, the photographing device 50 adjusts the focal position of the camera 52 with respect to the fertilized egg, and can photograph the fertilized egg in one well 12 at different focal positions. FIG. 3B shows an example of the focal position of the camera 52 with respect to the fertilized egg. In FIG. 3B, the position “M” indicates the focal position of the camera 52 in the plane passing through the substantially center of the fertilized egg, and an image obtained by slicing the fertilized egg at the position M in the Z direction is obtained as a photographed image. The position "H" indicates the focal position of the camera 52 in the plane passing from the upper part of the fertilized egg, and as a photographed image, an image obtained by slicing the fertilized egg at the position H in the Z direction can be obtained. The position "L" indicates the focal position of the camera 52 in the plane passing from the lower part of the fertilized egg, and as a photographed image, an image obtained by slicing the fertilized egg at the position L in the Z direction can be obtained. In this example, the photographing apparatus 50 generates an image in which the fertilized egg in each well 12 is photographed at three focal positions (slice planes) of “H”, “M”, and “L”.

It should be noted that preferably, the bottom surface of each well 12 has a shape that is inclined so as to be recessed toward the center of the bottom surface as shown in FIG. 3 (C) instead of the planar shape shown in FIG. 3 (B). It is good to have. As a result, the movement of the fertilized egg can be suppressed, and the observability of the moving image on the image viewing screen described later and the frame-out phenomenon in which the fertilized egg is detached from the captured image displayed on the image viewing screen are suitably suppressed. be able to.

The control unit 53 of the photographing device 50 adds a file name based on the position information in the X direction, the Y direction, and the Z direction to the image captured by the camera 52 and stores the image in the storage unit 54. For example, when the fertilized egg of the well A1 is photographed at the focal position of M in the Z direction at time t1 for the culture vessel 10 having a certain container ID, the control unit 53 displays the image on the image.
"Container ID-well position-focus position-shooting time",
A file name such as "Container ID-A1-M-t1" is given and stored in the storage unit 54. As a result, a large number of captured images stored in the storage unit 54 can be identified based on the file name.

Similarly, the control unit 53 may further add identification information regarding the arrangement of the wells 12 on the culture vessel 10 to be imaged to the above-mentioned file name and store it in the storage unit 54. For example, the control unit 53 adds identification information (for example, “25”) to the arrangement of the wells 12 in the culture vessel 10 to the file name of the photographed image of the fertilized egg of the culture vessel 10 in which the 25 wells 12 are arranged. (Character) may be added in the same manner as the container ID and the like. The identification information of the arrangement of the wells 12 is suitably used in the display processing of the image viewing screen described later. Similarly, the control unit 53 may add information for identifying the chamber in which the culture container 10 to be imaged is placed to the file name of the photographed image and store it in the storage unit 54. Since the chamber position can be determined by the XY coordinate position of the camera 52, the control unit 53 may, for example, assign the chamber number specified from the XY coordinate position of the camera 52 to the file name of the captured image. ..

The image display device 60 performs data communication with the storage device 55, and by referring to a file name or the like determined based on the above-mentioned rule, acquires a captured image necessary for displaying an image viewing screen described later, and browses the image. Display the screen. Then, the embryo culture person who is the user of the image display device 60 performs morphological observation and quality evaluation based on the photographed image of the fertilized egg displayed on the image viewing screen. Then, the embryo culture person decides the treatment (transplantation, freezing, culture discontinuation, etc.) of the fertilized egg based on the result of the above-mentioned quality evaluation, and executes the determined treatment for the fertilized egg.

[Display image viewing screen]
FIG. 4 is a display example of the image viewing screen displayed by the image display device 60. As shown in FIG. 4, the image display device 60 has a culture container designation field 70, a patient information display field 71, a container information display field 72, a list image display field 73, and an individual image display on the image viewing screen. A column 74 and an operation explanation display column 75 are provided.

The culture container designation field 70 contains a plurality of incubator designation buttons 78 for designating an incubator containing the culture container 10 to be displayed as a photographed image, and the culture container 10 to be displayed as a photographed image. Includes a plurality of chamber designation buttons 79 for designating a designated chamber.

In this example, the image display device 60 has the No. 1 on the culture container designation field 70. 1 to No. The four incubator designation buttons 78 with the identification numbers up to 4 are displayed, and the nine chamber designation buttons 79 with the identification numbers 1 to 9 are displayed. Then, the image display device 60 has the No. 1 selected by the user input. The incubator designation button 78 of 1 is highlighted with a red frame f1, and the first chamber designation button 78 selected by user input is highlighted with a red frame f2. In this case, the image display device 60 displays No. 1 in the list image display field 73 and the individual image display field 74, which will be described later. The photographed image of the fertilized egg in the culture vessel 10 stored in the No. 1 chamber of the incubator corresponding to No. 1 is displayed.

Further, the image display device 60 displays each incubator designation button 78 in a color-coded manner according to the state of the corresponding incubator. Here, the image display device 60 is No. 1 corresponding to the incubator in which the culture container 10 is being photographed by the camera 52. The incubator designation button 78 of No. 1 is colored in green and displayed, and the No. 1 corresponding to the incubator in the state where the imaging is completed and the culture vessel 10 has already been taken out. The incubator designation button 78 of 2 is colored in blue and displayed. The image display device 60 may determine the color coding of these greens and blues by, for example, comparing the shooting date and time of the newest shot image in the folder corresponding to each incubator with the current date and time. The determination may be made based on the presence / absence of a flag indicating the presence / absence of termination. Further, the image display device 60 has a No. 1 corresponding to an incubator whose existence cannot be confirmed. 3 and No. The incubator designation button 78 of 4 is displayed in an inactive state which is the same as the background color and cannot be selected. For example, the image display device 60 determines that the existence of the incubator cannot be confirmed when the folder of the incubator corresponding to the incubator designation button 78 cannot be confirmed in the storage device 55.

Further, the image display device 60 causes each chamber designation button 79 to display information on the corresponding chamber. Here, the image display device 60 causes the chamber designation button 79 to display patient information (here, name and patient ID) corresponding to the culture vessel 10 housed in the chamber, in addition to the number of the corresponding chamber. ing. The patient information displayed on the chamber designation button 79 may include the patient's status such as the treatment start date in addition to the name and patient ID. In this case, the image display device 60 is, for example, the storage device 55 or the storage unit 63 based on the incubator number corresponding to the incubator designation button 78 with the red frame f1 and the chamber number corresponding to each chamber designation button 79. By referring to the patient information database stored in the above, the patient information necessary for displaying each chamber designation button 79 is acquired.

In the patient information display field 71, the image display device 60 displays the name and patient ID (PID) of the patient corresponding to the culture container 10 housed in the chamber designated in the culture container designation field 70. In the container information display field 72, the image display device 60 displays the container ID of the culture container 10 housed in the chamber designated in the culture container designation field 70. Generally, the same culture vessel 10 contains fertilized eggs of the same patient. The container ID can be given by a barcode sticker or an IC tag attached to the culture container 10.

In the list image display column 73, the image display device 60 displays images of the captured images of each well 12 of the culture vessel 10 housed in the chamber designated in the culture vessel designation column 70 side by side according to the arrangement of each well 12 ( "List image Image List (hereinafter, also referred to as" IL ")" is displayed. The list image IL is composed of captured images of fertilized eggs in each well 12 of the same culture vessel 10, and the arrangement of these captured images is equal to the arrangement of the corresponding wells 12 in the culture vessel 10. Here, the image display device 60 is, for example, the latest (that is, the most shooting time) stored in the storage device 55 among the time-series photographed image groups corresponding to each well 12 of the culture container 10 to be displayed. The list image IL is generated by acquiring the captured image (slow) from the storage device 55 for each well 12. In this case, the image display device 60 may update the list image IL at regular time intervals. In another example, the image display device 60 is the earliest initial image capture time stored in the storage device 55 among the time-series photographed image groups corresponding to each well 12 of the culture container 10 to be displayed. A list image IL may be generated by acquiring an image from the storage device 55 for each well 12.

Further, the image display device 60 clearly indicates the positions (1 to 5) in the X direction and the positions (A to E) in the Y direction of the wells 12 corresponding to each captured image in the list image IL. Thereby, the embryo culture person who is an observer can easily grasp the identification number of the well 12 corresponding to each photographed image in the list image IL.

Further, each captured image in the list image IL can be selected individually, and the image display device 60 individually displays the captured image of the list image IL selected by the user input in the individual image display field 74. The selection by the user input described above may be a selection based on an arbitrary operation such as a click operation by a mouse, a mouse over, and a touch operation when the touch panel is laminated on the display unit 64. In the example of FIG. 4, since the captured image in the list image IL corresponding to the well A1 is selected by the user input, the image display device 60 attaches a red frame f3 to the captured image and highlights the captured image. do. In this case, the image display device 60 displays the captured image of the well A1 highlighted by the red frame f3 on the individual image display field 74. In the present embodiment, the image display device 60 displays on the individual image display field 74 a moving image in which the photographed images of the fertilized eggs of the target well 12 are switched in chronological order according to the photographing time. Here, the display size of the photographed image displayed in the individual image display field 74 is sufficiently larger than the display size of the photographed image included in the list image IL so that the embryo culture person can perform the quality evaluation of the cells.

When the image display device 60 displays the captured image acquired from the storage device 55 in the individual image display field 74, the image display device 60 cuts out (that is, trims) the captured image acquired from the storage device 55. It is displayed in the individual image display field 74. As a result, the image display device 60 suitably enlarges and displays the fertilized egg, which is important for quality evaluation and the like. The cutting range by this cutting process differs depending on the on or off state of the stabilization processing designation button 87, which will be described later. Further, the image display device 60 may display the captured image to be displayed in the list image display field 73, which has been similarly cut out.

In the operation explanation display field 75, the image display device 60 displays an outline of the operation method using the keyboard and / and the mouse for the moving image to be played on the individual image display field 74.

Further, the image display device 60 includes a time control bar 80 indicating the shooting time of the captured image of the fertilized egg to be displayed on the individual image display column 74 in the area of the image viewing screen other than the above-mentioned columns 70 to 75, and fertilization. A slice control bar 81 that specifies a position for slicing an egg (a position in the Z direction), a display magnification control bar 82, an export button 83, a brightness control bar 84, a contrast control bar 85, and a playback control button 86. A stabilization process designation button 87 is provided. The cursor positions of the time control bar 80, the slice control bar 81, the display magnification control bar 82, the luminance control bar 84, and the contrast control bar 85 can be adjusted by user input such as mouse operation.

The image display device 60 moves the cursor position of the time control bar 80 to the right according to the transition of the shooting time to be displayed during the reproduction of the moving image in the individual image display field 74. Further, when the image display device 60 accepts the user input for changing the cursor position of the time control bar 80, the image display device 60 individually displays a moving image or a still image of the captured image based on the shooting time corresponding to the changed cursor position. Displayed in column 74. The slice control bar 81 can stepwise specify the Z position of the captured image of the fertilized egg to be displayed on the individual image display field 74 by moving the cursor. For example, in the case of targeting a photographed image of a fertilized egg photographed at each Z position of positions “H”, “M”, and “L” based on the example of FIG. 3 (B), the position on the slice control bar 81. It is possible to specify any of "H", "M", and "L". The image display device 60 displays the photographed image of the fertilized egg photographed at the Z position designated by the slice control bar 81 in the list image display column 73 and the individual image display column 74. The input using the slice control bar 81 by the user is an example of the "external input" in the present invention.

The display magnification control bar 82, the brightness control bar 84, and the contrast control bar 85 can each specify the magnification, brightness, and contrast of the captured image to be displayed on the individual image display field 74 by moving the cursor. The image display device 60 is displayed in the list image display column 73 and the individual image display column 74 so as to have the display magnification, brightness, and contrast corresponding to the cursor positions of the display magnification control bar 82, the brightness control bar 84, and the contrast control bar 85. Adjust the displayed captured image. Further, the export button 83 is a button for designating an output format (moving image format or still image format) when exporting the captured image to be displayed in the individual image display field 74. When any of the export buttons 83 is selected by user input, the image display device 60 outputs captured images in a predetermined folder of the storage unit 63, for example, in an output format corresponding to the selected export button 83. The playback control button 86 has a button for switching the captured image of the fertilized egg displayed on the individual image display field 74 to the captured image at the shooting start time, a button for starting playback of the moving image, and a fertilized egg displayed on the individual image display field 74. It consists of a button to switch the captured image of the image to the captured image at the shooting end time. When any of the reproduction control buttons 86 is selected by the user input, the image display device 60 displays the captured image to be displayed in the individual image display field 74 so as to execute the function corresponding to the selected reproduction control button 86. Display control is performed.

The stabilization processing designation button 87 is a button for designating on or off of stabilization processing for time-series captured images to be displayed as moving images in the individual image display field 74. When the stabilization processing designation button 87 is turned off, the image display device 60 has, for example, the center and cutout of each shot image before cutting out for each of the time-series shot images to be displayed in the individual image display field 74. An image cut out with equal widths at the top, bottom, left and right with respect to the center of the captured image is displayed in the individual image display field 74 so as to coincide with the center of the subsequent image. On the other hand, in the image display device 60, when the stabilization processing designation button 87 is turned on, for each of the time-series captured images to be displayed as a moving image in the individual image display field 74, the well or the fertilized egg in the captured image is displayed. Adjust the position of the cutout range (that is, the range cut out and displayed from the original shot image) so that the position does not shift between the shot images. As a result, the image display device 60 suppresses the frame-out of the fertilized egg and the positional deviation of the well 12 or the fertilized egg between the captured images displayed as moving images, and is used during quality evaluation by an embryo culture technician. Improve the observability of moving images in.

Further, in the example of FIG. 4, the fertilized egg is displayed substantially in the center by the stabilization process, and as a result, wells and spaces that do not overlap with the fertilized egg are formed at the four corners of the image displayed in the individual image display field 74. There is. Then, the image display device 60 displays the displayed photographed image and various information related to the fertilized egg displayed on the photographed image in the above-mentioned space formed by the stabilization process. Specifically, the image display device 60 accommodates the patient's name, PID, and the fertilized egg corresponding to the fertilized egg to be displayed in the space on the upper left of the captured image displayed in the individual image display field 74. The identification number of the well 12 is displayed. Further, the image display device 60 displays the display magnification of the photographed image in the space at the lower left of the photographed image displayed in the culture container designation field 70. Further, the image display device 60 displays information on the number of pronuclei (here, two) displayed in the photographed image in the space on the upper right of the photographed image displayed in the individual image display field 74.

Further, the image display device 60 displays the information on the shooting date and time and the Z position indicator 90 in the space at the lower right of the shot image displayed in the individual image display field 74. Here, as information on the shooting date and time, the image display device 60 includes the shooting date (September 8, 2017) and the date and time (5:30), as well as the elapsed time from the start of shooting (here, 0). Seconds) and the frame number (first frame), which is a serial number for the captured image being displayed, are displayed. Further, the image display device 60 arranges bars corresponding to each Z position (here, each Z position in 7 stages) as the Z position indicator 90, and corresponds to the Z position designated by the slice control bar 81. The bar is highlighted.

As described above, the image display device 60 superimposes the information on the photographed image on the lower left and lower right spaces of the photographed image displayed in the individual image display field 74, and relates the fertilized egg to be displayed in the upper left and upper right spaces. Information is superimposed and displayed. As a result, for example, after the captured image or its moving image is output to the storage unit 63 or the like with the export button 83 or the like, when the user browses the image, the content of the image processing performed on the image or the image is used. The fertilized egg to be displayed can be suitably identified, and image management becomes easy.

[Stabilize processing]
Next, the stabilization process to be executed when the stabilization process specification button 87 is on will be described.

FIG. 5A is a diagram showing an outline of the stabilization process. In FIG. 5A, the “reference image” indicates a reference image in the stabilization process, and the “alignment target image” is a time-series photograph of the fertilized egg to be displayed in the individual image display field 74. The photographed image is shown. Here, the reference image is an image of the fertilized egg and the well in which the frame-out of the fertilized egg has not occurred, and has the same number of pixels in the vertical and horizontal directions as the image to be aligned. The reference image may be one image selected from the alignment target image according to a predetermined rule (for example, the captured image of the first frame first captured for the target fertilized egg), and the fertilized egg or / and. The image may be processed or taken separately so that the center of gravity of the well overlaps with the center of the image. In the latter case, the image display device 60 stores the reference image in the storage unit 63 in advance. Further, the reference image may be different for each fertilized egg, or may be common to all fertilized eggs.

The image display device 60 determines the amount of deviation in the shooting range between the alignment target image and the reference image with respect to the fertilized egg to be displayed in the individual image display field 74 for each frame (that is, for each shot image at a different shooting time). To) Calculate. In the following, the two-dimensional coordinates of the pixel positions of the reference image and the alignment image are "(x, y)", and the brightness of the reference image and the arbitrary alignment target image are "P (x, y)" and "Q (", respectively. x, y) ”, the amount of deviation in the shooting range on the horizontal axis between the reference image and the image to be aligned is referred to as“ dx ”, and the amount of deviation in the shooting range on the vertical axis is referred to as“ dy ”. In this case, the image display device 60 calculates, for example, the deviation amounts dx and dy that minimize the total value of the luminance differences between the reference image and the alignment target image for each pixel for each alignment target image. That is, the image display device 60 calculates the deviation amounts dx and dy for each alignment target image based on the following equation (1).

minΣ (P (x, y) -Q (x-dx, y-dy)) Equation (1)
Then, the image display device 60 determines the position of the cutout range with respect to each alignment target image based on the deviation amounts dx and dy with respect to each alignment target image.

FIG. 5B is a diagram in which the cut-out range when the deviation amounts dx and dy are both 0 with respect to the reference image is clearly shown by the broken line frame 91. Here, the fertilized egg and the well are displayed near the center of the reference image, and the cutout range shown by the broken line frame 91 coincides with the center of the image before cutting out on the left, right, top and bottom. When the deviation amounts dx and dy are both 0, the cutting range is set to a size and position where the fertilized egg and the well of the reference image do not frame out. The size of the cutout range indicated by the broken line frame 91 is an example of the "predetermined size" in the present invention.

FIG. 5C is a diagram showing the cut-out range for the aligned image in which the deviation amounts dx and dy are other than 0, respectively, “dx1” and “dy1” by the alternate long and short dash line frame 92. Further, FIG. 5C also clearly shows the cutout range of the broken line frame 91 shown in FIG. 5B for convenience of explanation.

In this case, the image display device 60 sets the cutout range (see the alternate long and short dash line frame 92) of the target image to be aligned to the cutout range (see the broken line frame 91) when both the deviation amounts dx and dy are 0. The size (that is, the vertical and horizontal directions are the same width) is set, and the positions are set according to the calculated deviation amounts dx1 and dy1. Specifically, as shown in FIG. 5C, the image display device 60 sets the cutting range (see the broken line frame 91) when the deviation amounts dx and dy are both 0 in the horizontal direction with the deviation amount dx1 and the vertical direction. A cutout range (see the alternate long and short dash line frame 92) for the target aligned image is set at a position shifted by the amount of deviation dy1 in the direction. Here, the fertilized egg and the well are displayed in the one-dot chain line frame 92 indicating the cut-out range set for the alignment image of the target. Therefore, in this case, the image display device 60 appropriately sets the cutting range in which the fertilized egg and the well do not frame out even when the display positions of the fertilized egg and the well are displaced in the aligned image of the target. is doing.

In this way, the image display device 60 sets the cutout range by calculating the deviation amounts dx and dy based on the reference image for each alignment target image, and each alignment target is based on the set cutout range. The captured image obtained by cutting out the image is displayed on the individual image display field 74 in chronological order. Thereby, the photographed images of the fertilized eggs in the time series in which the blurring of the display positions of the fertilized eggs and the wells is suitably reduced can be displayed in the individual image display column 74.

Further, in the present embodiment, as described above, the image display device 60 displays the captured image corresponding to the arbitrary Z position specified by the user by the slice control bar 81 in the individual image display field 74, so that each Z is displayed. It is necessary to perform stabilization processing on the captured image corresponding to the position. Hereinafter, specific examples (first to fourth examples) of the stabilization process in consideration of the Z position will be described. In the following description, for convenience, as an example, it is assumed that the Z position has three stages of “H”, “M”, and “L” as in the example of FIG. 3 (B).

FIG. 6A is a diagram showing an outline of a first example of stabilization processing in consideration of the Z position. In the first example, the image display device 60 sets one reference image for the captured image corresponding to all the Z positions which are the alignment target images, and shifts the amount dx for each alignment target image. dy is calculated. In this case, the image display device 60 sets the cutout range of the corresponding alignment target image based on the calculated deviation amounts dx and dy, as in the description of FIGS. 5B and 5C. As a result, the image display device 60 can suitably execute the stabilization process on the captured image corresponding to all the Z positions.

FIG. 6B is a diagram showing an outline of a second example of the stabilization process in consideration of the Z position. In the second example, the image display device 60 considers that there is no change in the deviation amounts dx and dy due to the difference in the Z position, and calculates the deviation amounts dx and dy for the alignment target image corresponding to a certain Z position. , It is also applied to the image to be aligned corresponding to other Z positions.

Specifically, the image display device 60 sets one reference image for the alignment target image corresponding to all Z positions, and targets each alignment target image at a certain Z position (here, position M). The deviation amounts dx and dy from the reference image are calculated as, and the cutout range of each alignment target image at the position M is set. Further, the image display device 60 sets a cutting range for the alignment target image at the Z position other than the position M by using the deviation amounts dx and dy calculated with respect to the alignment target image at the position M. For example, when the image display device 60 calculates the deviation amounts dx and dy with respect to the Xth frame of the position M (X is an arbitrary positive integer), the Xth frame of the positions L and H which are other Z positions. The above-mentioned deviation amounts dx and dy are applied to the above-mentioned deviation amounts to set the cutting range of these frames. As a result, the processing load of the stabilization process can be suitably reduced as compared with the first example.

Further, when the deviation amounts dx and dy change depending on the Z position due to the mechanical structure of the drive mechanism of the camera 52, the image display device 60 may change the Z position in addition to the processing shown in FIG. 6 (B). The amount of deviation caused may be further calculated to correct the cutting range. For example, in the case of FIG. 6B, the image display device 60 determines the amount of deviation between the frames of the position M and the position H and the amount of the deviation between the frames of the position M and the position L for the Xth frame, respectively. calculate. Then, the image display device 60 corrects the position of the cutout range of each alignment target image at the position H by the amount of deviation between the calculated positions M and the frames. Similarly, the image display device 60 corrects the cutout range of each alignment target image at the position L by the amount of deviation between the calculated positions M and the frames. As a result, due to the mechanical structure of the drive mechanism of the camera 52, even if the deviation amounts dx and dy change depending on the Z position, the stabilization process of the captured image for all the Z positions can be executed with high accuracy. can.

FIG. 7A is a diagram showing an outline of a third example of the stabilization process in consideration of the Z position. In the third example, the image display device 60 sets a reference image for each Z position, calculates the deviation amounts dx and dy between the reference image corresponding to the same Z position and the alignment target image, and performs the alignment. Set the cutout range for the target image. As a result, even when a focus shift or the like occurs due to a difference in the Z position, the stabilization process of the captured image for each Z position can be executed with high accuracy.

FIG. 7B is a diagram showing an outline of a fourth example of the stabilization process in consideration of the Z position. The fourth example is common to the third example in that a reference image is set for each Z position, and is different from the third example in that the deviation amounts dx and dy common to each Z position are calculated. Here, the variable representing the Z position is "z", the brightness of the reference image corresponding to the position z is "P (x, y, z)", and the brightness of the alignment target image corresponding to the position z is "Q (". x, y, z) ”, the image display device 60 is, for example, based on the following equation (2), the total value of the luminance difference for each pixel between the reference image corresponding to each Z position and the image to be aligned. Is calculated as the minimum deviation amount dx and dy.

minΣ (P (x, y, z) -Q (x-dx, y-dy, z)) Equation (2)
In this way, the image display device 60 calculates the deviation amounts dx and dy common to the alignment target images at all Z positions for each frame, and the alignment target image of the target alignment target images based on the deviation amounts dx and dy. Set the cutting range. Also with this, the image display device 60 can suitably execute the stabilization process for the captured image corresponding to all the Z positions.

[Modification example]
Next, a modification of the above image display system will be described. The following modifications can be carried out in combination as appropriate.

(Modification 1)
In the above image display system, a plurality of wells 12 are formed in the culture vessel 10, and fertilized eggs are housed in each well 12, but it is essential in the above-mentioned stabilization process that the wells are formed in the culture vessel 10. is not it.

Even when a container in which a well is not formed is used, the image display device 60 uses, for example, an image in which the center of gravity of the fertilized egg is the center of the image or a captured image in the first frame as a reference image in the stabilization process, and is described above. Based on the above equation (1) and the like, the deviation amounts dx and dy based on the position of the fertilized egg between the reference image and the alignment target image are calculated based on the difference in brightness for each pixel. As a result, the image display device 60 can suitably set the cutting range of each captured image which is the image to be aligned, and can suitably display the moving image with less blurring of the fertilized egg in the individual image display field 74.

(Modification 2)
In the above embodiment, one fertilized egg is contained in one well, but instead, a plurality of (for example, two) fertilized eggs may be contained in one well. Further, even when the culture vessel 10 in which the well is not formed is used, a plurality of fertilized eggs may be contained in the vessel as one group. It is more preferable to house one fertilized egg in one well from the viewpoint of individual management.

(Modification 3)
The configuration of the image display system 100 shown in FIG. 1 is an example, and the configuration to which the present invention can be applied is not limited to the configuration shown in FIG.

For example, the image display device 60 may display an image viewing screen based on display information generated by the storage device 55 or a server device (not shown). In this case, the storage device 55 or the server device generates the display information of the image viewing screen by executing the display processing of the image viewing screen including the stabilization process performed by the image display device 60 in the embodiment, and the generated display. Information is transmitted to the image display device 60. Then, the image display device 60 displays the image viewing screen based on the received display information. Further, the image display device 60 transmits the input information corresponding to the user input on the image viewing screen to the above-mentioned storage device 55 or the server device, so that the image viewing screen reflects the input contents indicated by the input information. Receive the display information of. Also in this aspect, the image display device 60 can suitably display the image viewing screen. In this aspect, the storage device 55 or the server device described above functions as the "image display device" in the present invention, and the image display device 60 functions as the "display device" in the present invention. Further, the storage device 55 or the CPU of the server device functions as a "computer" for executing the program in the present invention.

In another example, the captured image stored in the storage device 55 may be stored in the storage unit of either the photographing device 50 or the image display device 60. Even in this case, the image display device 60 can suitably display the image viewing screen by acquiring the photographed image from the photographing device 50 or the storage unit 63.

(Modification example 4)
In the above embodiment, the photographing device 50 adds attribute information indicating a patient ID, a well position, a camera focus position, a shooting time, etc. to the captured image to a file name or the like and transmits the captured image to the storage device 55. However, at that time, a CRC code, checksum data, or the like may be added. This makes it possible to prevent the data of the captured image from being modified or the file name to be changed by mistake after shooting, resulting in a mistake in the captured image.

(Modification 5)
As a method of identifying the fertilized egg image group of the same patient generated from the same culture vessel 10, it may be determined from the time when the image was taken.

Generally, the movement of the camera 52 between the wells 12 is short, but the movement of the camera 52 between the culture vessels 10 takes a long time. In other words, since the moving distance when observing the well 12 in the culture vessel I is small, the imaging interval is short, but when moving from the culture vessel I to the culture vessel II, the moving distance becomes long and the imaging interval becomes long. Therefore, the image display device 60 may consider a group of captured images whose imaging time interval is less than a predetermined time as a fertilized egg image group of the same patient generated from the same culture vessel 10. Also in this aspect, the image display device 60 can suitably identify the fertilized egg image group of the same patient generated from the same culture vessel 10.

10 Culture container 12 well 20 Photographed image 30 Fertilized egg 50 Photographing device 51 Lighting 52 Camera 53 Control unit 54 Communication unit 55 Storage device 60 Image display device

Claims (9)

An acquisition unit that acquires multiple captured images of an object at multiple focal positions at multiple times, respectively .
A display control unit for displaying a part of the captured image on the display unit is provided.
The display control unit has a display range of the captured image based on the amount of deviation between the imaging range of the captured image corresponding to the focal position designated by the external input and the imaging range of the reference image set for each focal position. An image display device that determines and displays the determined image in the display range on the display unit in chronological order of shooting . The image display device according to claim 1 , wherein the display control unit determines a position of the display range defined in a predetermined size based on the deviation amount. The object is housed in a recess and is housed in a recess.
The image display device according to claim 1 or 2 , wherein the acquisition unit acquires the photographed image including the recess in the photographing range. The display control unit is characterized in that the display range is determined based on the deviation amount so that the object or the recess in which the object is housed in the captured image is located near the center of the display range. The image display device according to any one of claims 1 to 3 . The display control unit is characterized in that information related to the captured image or information related to the object is displayed on the display unit at a position within the display range that does not overlap with the display position of the object. The image display device according to any one of claims 1 to 4 . The aspect according to any one of claims 1 to 5 , wherein the acquisition unit acquires an image obtained by repeatedly photographing a plurality of objects by changing the position of the photographing unit for each object as the photographed image. Image display device. The image according to any one of claims 1 to 6 , wherein the display control unit transmits display information for displaying a part of the captured image to a display device having the display unit. Display device. A program run by a computer
An acquisition unit that acquires multiple captured images of an object at multiple focal positions at multiple times .
The computer is made to function as a display control unit for displaying a part of the captured image on the display unit.
The display control unit has a display range of the captured image based on the amount of deviation between the imaging range of the captured image corresponding to the focal position designated by the external input and the imaging range of the reference image set for each focal position. A program that determines and displays the determined images in the display range on the display unit in chronological order of shooting . It is a control method executed by the image display device.
An acquisition process for acquiring multiple captured images of an object at multiple focal positions at multiple times, respectively .
It has a display control step of displaying a part of the captured image on the display unit.
In the display control step , the display range of the captured image is based on the amount of deviation between the imaging range of the captured image corresponding to the focal position designated by the external input and the imaging range of the reference image set for each focal position. A control method for determining and displaying the determined image in the display range on the display unit in chronological order of shooting .

Images