WO2024128212A1 - Information processing device for outputting information based on point-of-gaze, and information processing method and program for outputting information based on point-of-gaze - Google Patents

Abstract

One embodiment of the present invention provides an information processing device. This information processing device: acquires a captured image of a surgical operation; obtains a point-of-gaze in relation to the surgical operation, on the basis of the captured image of the surgical operation; and outputs information based on the obtained point-of-gaze.

Description

Information processing device for outputting information based on gaze point, information processing method for outputting information based on gaze point, and program

The present invention relates to an information processing device, an information processing method, and a program.

There are systems that capture and record images of surgical sites during surgery for use in academic presentations, in medical education, and the like.
Patent Document 1 discloses a technology for distributing and recording video of a surgical site (operative field).

JP 2017-192043 A

When recording images of the surgical site, it is important to know where to focus on the surgical site.

According to one aspect of the present invention, an information processing device is provided. This information processing device acquires captured images of a surgery, determines a gaze point related to the surgery based on the captured images of the surgery, and outputs information based on the determined gaze point.

FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system 1000. FIG. 2 is a diagram illustrating an example of a hardware configuration of the server device 100. As shown in FIG. FIG. 3 is a diagram illustrating an example of a hardware configuration of the client device 110. As shown in FIG. FIG. 4 is an activity diagram showing an example of information processing for storing captured images. FIG. 5 is an activity diagram showing an example of main information processing. FIG. 6 is a diagram showing an example of an acquired captured image. FIG. 7 is a diagram for explaining how to determine the axis of an object. FIG. 8 is a diagram showing an example of a gaze point of a captured image (current frame) to be processed. FIG. 9 is a diagram showing an example of determining a comprehensive gaze point based on the gaze point of the current frame and gaze points of several past frames. FIG. 10 is a diagram showing an example of a gaze point of a captured image (current frame) to be processed in the fourth modification.

(Embodiment 1)
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Various features shown in the following embodiments can be combined with each other. In particular, in this specification, the term "part" may include, for example, a combination of hardware resources implemented by a circuit in a broad sense and software information processing that can be specifically realized by these hardware resources. In addition, in this embodiment, various information is handled, and communication and calculation can be performed on a circuit in a broad sense, regardless of whether this information is represented by the high and low signal values as a binary bit collection consisting of 0 or 1, represented by physical numerical values of signal values, or represented by quantum superposition.

In addition, a circuit in the broad sense is a circuit that is realized by at least appropriately combining a circuit, circuitry, a processor, and memory. In other words, it includes application specific integrated circuits (ASICs), programmable logic devices (e.g., simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

In addition, the programs for realizing the software appearing in the embodiments may be implemented in a form that allows them to be downloaded from a server, the programs may be executed on a cloud computer, or the programs may be stored in non-volatile or volatile non-transient storage media and distributed.

1. System Configuration Fig. 1 is a diagram showing an example of the system configuration of an information processing system 1000. As shown in Fig. 1, the information processing system 1000 includes, as a system configuration, a server device 100, a client device 110, and a camera 120. The server device 100, the client device 110, and the camera 120 are communicatively connected via a network 150.
The camera 120 is a device that captures an image of a surgical site (surgical field) in an operating room. In the case of an open surgery, the camera 120 is a Web camera or the like installed in the operating room. Based on control information from the server device 100, the camera 120 changes the imaging direction, imaging position, and camera settings (e.g., ISO sensitivity, white balance, shutter mode, microphone, self-timer, focus setting, camera shake reduction, flicker prevention, storage destination setting, time-lapse shooting interval, etc.) and changes the digital zoom setting. Digital zoom is a process of stretching and enlarging a part of an image by image processing.
The server device 100 is a device that executes the process according to the embodiment on captured images transmitted from the camera 120 or the like.
The client device 110 is a device that displays the results of the processing performed by the server device 100, etc.

Note that in FIG. 1, only one client device 110 is illustrated in the information processing system 1000 for the sake of simplicity, but the information processing system 1000 may include multiple client devices 110. Also, in FIG. 1, only one camera 120 is illustrated in the information processing system 1000 for the sake of simplicity, but the information processing system 1000 may include multiple cameras 120. When multiple cameras 120 are included in the information processing system 1000, the types of the multiple cameras 120 may be the same or different. Examples of the types of camera 120 include a web camera, an endoscope camera, etc. Also, the processes of the following embodiments and the like will be described as being executed by the server device 100, but instead of the server device 100, a PC (Personal Computer), a cloud system, etc. may execute the processes of the embodiments and the like.

2. Hardware Configuration (1) Hardware Configuration of Server Device 100 Fig. 2 is a diagram showing an example of the hardware configuration of the server device 100. The server device 100 includes, as its hardware configuration, a control unit 210, a storage unit 220, and a communication unit 230. The control unit 210 is a central processing unit (CPU) or the like, and controls the entire server device 100. The storage unit 220 is any one of a hard disk drive (HDD), a read only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or the like, or any combination thereof, and stores programs and data used when the control unit 210 executes processing based on the programs. The control unit 210 executes processes based on the programs stored in the storage unit 220, thereby realizing the functions of the server device 100. The communication unit 230 is a NIC (Network Interface Card) or the like, which connects the server device 100 to the network 150 and controls communication with other devices.

In the first embodiment, the data used by the control unit 210 to execute processing based on a program is described as being stored in the storage unit 220, but the data may be stored in a storage unit or the like of another device with which the server device 100 can communicate. Also, in FIG. 2, the control unit 210 is described as being one unit, but multiple control units may execute processing based on a program stored in a storage unit or the like.

(2) Hardware Configuration of the Client Device 110 FIG. 3 is a diagram showing an example of the hardware configuration of the client device 110. The client device 110 includes, as a hardware configuration, a control unit 310, a storage unit 320, an input unit 330, an output unit 340, and a communication unit 350. The control unit 310 is a CPU or the like, and controls the entire client device 110. The storage unit 320 is any one of an HDD, a ROM, a RAM, and an SSD, or any combination thereof, and stores a program, data used when the control unit 310 executes processing based on the program, and the like. The control unit 310 executes processing based on the program stored in the storage unit 320, thereby realizing the function of the client device 110. The input unit 330 is a keyboard and/or a mouse, and the like, and inputs user operation information, etc. to the control unit 310. The output unit 340 is a display, and the like, and displays the results of information processing by the control unit 310, etc. The communication unit 350 is a NIC or the like, which connects the client device 110 to the network 150 and controls communication with other devices.

The camera 120 also has at least a control unit, a memory unit, and an imaging unit. The imaging unit captures an image of a subject (the surgical site in this embodiment) based on the control of the control unit. The control unit controls the overall processing of the camera 120. The control unit realizes the functions of the camera 120 by executing processing based on a program stored in the memory unit. The memory unit stores the program for the camera 120 and data used when the control unit executes processing based on the program. The memory unit may also be configured to store images captured by the camera 120.

3. Information Processing Hereinafter, information processing according to the first embodiment will be described.
(1) Captured Image Storage Processing FIG. 4 is an activity diagram showing an example of information processing for storing captured images.
In activity A401, the control unit 210 waits to receive an image of the surgical site (hereinafter also simply referred to as an image) from the camera 120. If the control unit 210 receives an image, it proceeds to activity A402, and if the control unit 210 does not receive an image, it repeats the processing of activity A401.
In activity A402, the control unit 210 stores the received captured image in the storage unit 220 or the like.
During startup, the server device 100 repeatedly executes the process shown in Fig. 4. The camera 120 captures images of the surgical site in the operating room and transmits chronologically continuous captured images (i.e., video) to the server device 100. Thus, the server device 100 stores the video transmitted from the camera 120 in the storage unit 220 or the like.
In the following embodiment, the server device 100 is described as executing the information processing described below after storing the captured image received from the camera 120 in the storage unit 220, etc. However, the server device 100 may acquire the captured image from the predetermined data received from the camera 120 before storing the captured image in the storage unit 220, etc., and execute the information processing described below.

(2) Main processing (processing overview)
An overview of the main information processing of the first embodiment will be described with reference to Fig. 5. The control unit 210 acquires a captured image from data received from the storage unit 220 or the camera 120. The control unit 210 then determines a gaze point related to the surgery based on the acquired captured image. The control unit 210 then outputs information based on the determined gaze point. By performing such processing, it is possible to output information based on the gaze point related to the surgical site.
(Processing details)
FIG. 5 is an activity diagram showing an example of main information processing.
In activity A501, the control unit 210 acquires a captured image (frame) from data received from the storage unit 220 or the camera 120. Fig. 6 is a diagram showing an example of the acquired captured image. As shown in Fig. 6, the captured image includes a surgical site of an abdominal surgery. Also, as shown in Fig. 6, the captured image includes an operator's hand 610, a surgical instrument 620, a surgical instrument 630, and a surgical instrument 640. That is, the captured image of the surgery shown in Fig. 6 is an image of a surgical site during an abdominal surgery. The operator's hand 610, the surgical instrument 620, the surgical instrument 630, and the surgical instrument 640 included in the captured image are examples of objects.

In activity A502, the control unit 210 extracts objects from the captured image. To explain the processing of activity A502 in more detail, the control unit 210 inputs the captured image to the trained model. Here, the trained model is a trained model that has been trained using the captured image of the surgery as input data and the objects contained in the captured image of the surgery as output data. The control unit 210 extracts objects from the captured image by acquiring the objects output from the trained model. Note that, as another example, the control unit 210 may perform image analysis of the captured image and extract objects from the captured image based on the image analysis. When multiple objects are present, the control unit 210 extracts each of the multiple objects. In the following explanation, unless otherwise specified, it is assumed that multiple objects are present.

In activity A503, the control unit 210 obtains the axes of each of a plurality of objects extracted from the captured image, and obtains a gaze point from the intersection of the obtained axes.
A method for determining the axis of an object will be described with reference to Fig. 7. Fig. 7 is a diagram for explaining a method for determining the axis of an object. The control unit 210 determines a rectangular area that circumscribes an area from which an object included in a captured image is extracted, and determines a width W and a height H of the rectangular area. The control unit 210 also determines a center of gravity (X, Y) 710 of the rectangular area.

Next, the control unit 210 obtains the area and center of gravity of an extraction area existing within a circle with a radius of √(W^2+H^2)*3/8 at the coordinates of the following eight points.
1. ((X-W)/4, (Y-H)/4)
2. (X, (Y-H)/4)
3. ((X+W)/4, (Y-H)/4)
4. ((X-W)/4,Y)
5. ((X+W)/4,Y)
6. ((X-W)/4, (Y+H)/4)
7. (X, (Y+H)/4)
8. ((X-W)/4, (Y-H)/4)
7, eight circles are shown as the areas of interest. The control unit 210 calculates the area and center of gravity of the extraction areas that exist within these circles.

Next, the control unit 210 selects the area of interest (of the eight circles) that overlaps the largest area with the extracted area. In the example of FIG. 7, area of interest 720 is selected as the selected area of interest. The control unit 210 also selects three areas of interest that exist opposite the selected area of interest with respect to the center of gravity. In the example of FIG. 7, area of interest 721, area of interest 722, and area of interest 723 are selected as the three areas of interest.

The control unit 210 determines a center of gravity 730 of the area where the attention area 720, which is the selected attention area, overlaps with the extraction area. The control unit 210 also determines a center of gravity 731 of the area where the attention area 721 overlaps with the extraction area. The control unit 210 also determines a center of gravity 732 of the area where the attention area 722 overlaps with the extraction area. The control unit 210 also determines a center of gravity 733 of the area where the attention area 723 overlaps with the extraction area. The control unit 210 also determines a center of gravity 735 which is the center of gravity of the centers of gravity 731, 732, and 733.
Then, the control unit 210 sets the straight line connecting the center of gravity 730, the center of gravity 710, and the center of gravity 735 as the axis of the object to be processed among the extracted objects.

The control unit 210 determines the axes of each of the multiple objects extracted from the captured image, and determines the gaze point of the captured image to be processed from the intersection of the determined axes. More specifically, the control unit 210 removes outlier intersections from all intersections of the determined axes. The control unit 210 removes outliers using an outlier removal algorithm called LocalOutlierFactor, but is not limited to this method. The control unit 210 sets the average point of the intersections from which the outliers have been removed as the gaze point of the captured image to be processed (current frame). Figure 8 is a diagram showing an example of the gaze point of the captured image to be processed (current frame). Point 810 is an example of the determined gaze point.

The control unit 210 acquires the gaze points of the captured image (current frame) and the past captured images (past frames) that are chronologically consecutive from the captured image (current frame) from the storage unit 220, etc. The control unit 210 acquires gaze points for four frames as the gaze points of the past captured images (past frames), but is not limited to this. The control unit 210 calculates the average of the gaze points of the captured image (current frame) and the gaze points of the past captured images (past frames) to obtain a comprehensive gaze point. This process is an example of a process for obtaining a gaze point related to surgery based on the gaze point obtained based on the captured image (current frame) to be processed and the gaze point of the past captured images (past frames) that are chronologically consecutive to the captured image to be processed. FIG. 9 is a diagram showing an example of obtaining a comprehensive gaze point based on the gaze point of the current frame and the gaze points of several past frames.
The determined overall gaze point is the gaze point determined in activity A503.

In activity A504, the control unit 210 outputs information based on the gaze point obtained in activity A503. For example, the control unit 210 outputs information based on the obtained gaze point (for example, an object indicating the gaze point) by displaying information indicating the obtained gaze point superimposed on the captured image displayed on the output unit 340 of the client device 110. Also, for example, the control unit 210 outputs information based on the obtained gaze point by transmitting control information for controlling an imaging device that captures an image based on the obtained gaze point to the imaging device. The control information includes information for controlling the camera 120 so that the gaze point is included in the captured image. For example, if a part of the surgeon's body or the like gets between the camera 120 and the gaze point and the captured image does not include the gaze point, the control unit 210 transmits control information to the camera 120 for shifting the imaging orientation or position of the camera 120 so that the gaze point is included in the captured image. When the camera 120 receives the control information, the camera 120 moves the imaging orientation of the camera 120 or the position of the camera 120 based on the control information so that the gaze point is included in the captured image.

According to embodiment 1, the camera 120 can be controlled to record an image including the gaze point of the surgical site. As a result, an image including the gaze point of the surgical site is recorded in the camera 120 or the server device 100.

(Variation 1)
Modifications of the first embodiment will be described. Modification 1 will describe another example of a part of the processing of the first embodiment. Modification 1 is included in the first embodiment, and does not describe other embodiments. The same applies to the modifications described below.
Another example of the control information is information for controlling the imaging device to focus on the gaze point. For example, when the focus of the camera 120 deviates from the gaze point, the control unit 210 transmits control information to the camera 120 for changing the focus setting of the camera 120 so that the focus of the camera 120 coincides with the gaze point of the surgical site (surgical field). Upon receiving the control information, the camera 120 focuses the camera 120 on the gaze point of the surgical site (surgical field) based on the control information.

By combining the first modification with the first embodiment, the camera 120 can be controlled to record an image that includes the gaze point of the surgical site that is not out of focus. As a result, an image that includes the gaze point of the surgical site and is not out of focus is recorded in the camera 120 or the server device 100.

(Variation 2)
A modification of the first embodiment will be described.
The control unit 210 may output audio information as information based on the obtained gaze point. An example of the audio information is information such as "the gaze point is blocked by the surgeon's head."
According to the second modification, information regarding the point of gaze can be output by audio information.

(Variation 3)
A modification of the first embodiment will be described.
In the first embodiment, the description is given taking laparotomy as an example. However, the information processing system 1000 is also applicable to surgeries other than laparotomy. An example of surgery other than laparotomy is, for example, endoscopic surgery. Examples of endoscopic surgery include laparoscopic surgery, robot-assisted surgery, and microsurgery. In the third modification, for the sake of simplicity, laparoscopic surgery is described as an example. In the case of laparoscopic surgery, the camera 120 is a laparoscopic camera. The captured image of the laparoscopic surgery is an image of the surgical site during the laparoscopic surgery. In the case of the configuration of the third modification, as information based on the gaze point, the control unit 210 transmits control information to the laparoscopic camera for controlling the laparoscopic camera so that an image including the gaze point of the surgical site (operative field) is always captured during the laparoscopic surgery. Based on the control information, the laparoscopic camera moves, for example, automatically during the laparoscopic surgery so that an image including the gaze point of the surgical site (operative field) is always captured.
According to the third modification, the camera 120 can be controlled to record an image including the gaze point of the surgical site even in surgery other than open surgery, such as laparoscopic surgery. As a result, an image including the gaze point of the surgical site is recorded in the camera 120 or the server device 100.

(Variation 4)
A modification of the first embodiment will be described.
In the fourth modified example, the control unit 210 extracts surgical instruments and anatomical structures from the captured image in the activity A502. The surgical instruments are an example of a first object. The anatomical structures are an example of a second object. The anatomical structures are anatomical parts that constitute the human body, and examples of such anatomical structures include the gallbladder, liver, and heart. The control unit 210 inputs the acquired captured image into the trained model. Here, the trained model is a trained model trained using the surgical captured image as input data and the objects and object types included in the surgical captured image as output data. When the type of object output from the trained model indicates a surgical instrument and only one surgical instrument is extracted, the control unit 210 determines the axis of the surgical instrument. Then, the control unit 210 determines the intersection of the determined axis and the extracted anatomical structure, and sets the determined intersection as the gaze point.

10 is a diagram showing an example of a gaze point of a captured image (current frame) to be processed in Modification Example 4. An axis 1010 is an axis of a surgical instrument extracted from the captured image. A gaze point 1020 is an intersection point of an anatomical structure and the axis 1010.
According to the fourth modification, even if there is only one surgical instrument, the gaze point can be obtained based on the captured image. Therefore, even if there is only one surgical instrument, the camera 120 can be controlled to record an image including the gaze point of the surgical site. As a result, an image including the gaze point of the surgical site is recorded in the camera 120 or the server device 100.

Furthermore, it may be provided in the following forms:

(1) An information processing device that acquires captured images of a surgery, determines a gaze point related to the surgery based on the captured images of the surgery, and outputs information based on the determined gaze point.

(2) In the information processing device described in (1) above, an information processing device extracts multiple objects included in the captured image, determines the axes of the extracted multiple objects, and determines the gaze point from the intersection of the determined axes.

(3) In the information processing device described in (2) above, the object is an instrument related to the surgery.

(4) An information processing device according to (2) or (3) above, in which the captured image is input to a trained model, the trained model being trained using the captured image of the surgery as input data and the objects contained in the captured image of the surgery as output data, and the object is extracted from the captured image by obtaining the object output from the trained model.

(5) In the information processing device described in (1) above, a first object and a second object included in the captured image are extracted, an axis of the extracted first object is determined, and the gaze point is determined from the intersection of the axis and the second object.

(6) In the information processing device described in (5) above, the first object is an instrument related to the surgery, and the second object is an anatomical structure.

(7) An information processing device according to any one of (1) to (6) above, which determines a gaze point related to the surgery based on the gaze point determined based on the captured image and the gaze point of a past captured image that is chronologically consecutive to the captured image.

(8) An information processing device according to any one of (1) to (7) above, which outputs information based on the determined gaze point by displaying information indicating the determined gaze point superimposed on the captured image.

(9) An information processing device according to any one of (1) to (8) above, which outputs information based on the determined gaze point by transmitting control information to an imaging device that captures the captured image based on the determined gaze point.

(10) In the information processing device described in (9) above, the control information is information for controlling the imaging device so that the gaze point is included in the captured image.

(11) In the information processing device described in (9) or (10) above, the control information is information for controlling the imaging device so as to focus on the gaze point.

(12) An information processing device according to any one of (1) to (11) above, wherein the captured image of the surgery is a captured image of the surgical site during an open surgery.

(13) An information processing device according to any one of (1) to (11) above, wherein the captured image of the surgery is a captured image of the surgical site during laparoscopic surgery.

(14) An information processing method that acquires an image of a surgery, determines a gaze point related to the surgery based on the image of the surgery, and outputs information based on the determined gaze point.

(15) A program for causing a computer to function as the information processing device according to any one of (1) to (13) above.
Of course, this is not the case.

Finally, although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. New embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The embodiments and their modifications are within the scope and gist of the invention, and are also within the scope of the invention and its equivalents as set forth in the claims.

100: Server device, 110: Client device, 120: Camera, 150: Network, 210: Control unit, 220: Memory unit, 230: Communication unit, 310: Control unit, 320: Memory unit, 330: Input unit, 340: Output unit, 350: Communication unit, 610: Hand, 620: Instrument, 630: Instrument, 640: Instrument, 710: Center of gravity, 720: Area of interest, 721: Area of interest, 722: Area of interest, 723: Area of interest, 730: Center of gravity, 731: Center of gravity, 732: Center of gravity, 733: Center of gravity, 735: Center of gravity, 1000: Information processing system

Claims (15)

An information processing device,
acquiring a captured image of the surgery, determining a gaze point related to the surgery based on the captured image of the surgery, and outputting information based on the determined gaze point;
Information processing device. 2. The information processing device according to claim 1,
extracting a plurality of objects included in the captured image, determining axes of the extracted plurality of objects, and determining the gaze point from an intersection of the determined axes;
Information processing device. 3. The information processing device according to claim 2,
the object is an instrument related to the surgery;
Information processing device. 3. The information processing device according to claim 2,
The captured image is input into a trained model,
The trained model is a trained model trained using surgical images as input data and objects included in the surgical images as output data,
Extracting the object from the captured image by acquiring the object output from the trained model;
Information processing device. 2. The information processing device according to claim 1,
extracting a first object and a second object included in the captured image, determining an axis of the extracted first object, and determining the gaze point from an intersection of the axis and the second object;
Information processing device. 6. The information processing device according to claim 5,
the first object is an instrument related to the surgery;
the second object is an anatomical structure.
Information processing device. 2. The information processing device according to claim 1,
determining a gaze point related to the surgery based on the gaze point determined based on the captured image and a gaze point of a past captured image that is chronologically consecutive to the captured image;
Information processing device. 2. The information processing device according to claim 1,
and outputting information based on the obtained gaze point by superimposing information indicating the obtained gaze point on the captured image.
Information processing device. 2. The information processing device according to claim 1,
and outputting information based on the obtained gaze point by transmitting control information to the imaging device that captures the captured image based on the obtained gaze point.
Information processing device. 10. The information processing device according to claim 9,
The control information is information for controlling the imaging device so that the gaze point is included in the captured image.
Information processing device. 10. The information processing device according to claim 9,
The control information is information for controlling the imaging device so as to focus on the gaze point.
Information processing device. 2. The information processing device according to claim 1,
The captured image of the surgery is an image of a surgical site during an open surgery.
Information processing device. 2. The information processing device according to claim 1,
The captured image of the surgery is an image of a surgical site during laparoscopic surgery;
Information processing device. 1. An information processing method, comprising:
Acquire images of the surgery;
A surgical gaze point is obtained based on the captured image of the surgical procedure.
Output information based on the obtained gaze point.
Information processing methods. A program,
Computer,
A program for causing the information processing device according to any one of claims 1 to 13 to function as the information processing device.