[go: up one dir, main page]

WO2024195729A1 - Système de traitement d'informations, dispositif de traitement d'informations et procédé de génération de modèle d'apprentissage - Google Patents

Système de traitement d'informations, dispositif de traitement d'informations et procédé de génération de modèle d'apprentissage Download PDF

Info

Publication number
WO2024195729A1
WO2024195729A1 PCT/JP2024/010268 JP2024010268W WO2024195729A1 WO 2024195729 A1 WO2024195729 A1 WO 2024195729A1 JP 2024010268 W JP2024010268 W JP 2024010268W WO 2024195729 A1 WO2024195729 A1 WO 2024195729A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
unit
information processing
visual field
evaluation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/JP2024/010268
Other languages
English (en)
Japanese (ja)
Inventor
大輔 山田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Publication of WO2024195729A1 publication Critical patent/WO2024195729A1/fr
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/045—Control thereof
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis

Definitions

  • This disclosure relates to an information processing system, an information processing device, and a learning model generation method.
  • an endoscope captures images of the patient's abdominal cavity, and the captured images are displayed on a display.
  • the surgeon performs surgery while checking the captured images displayed on the display.
  • a robot that supports an endoscope is autonomously controlled, and technology has been developed to appropriately control the robot (see, for example, Patent Document 1).
  • Autonomous control of endoscopes requires control that allows users, such as surgeons, to see the field of view that they want to see.
  • autonomous control of endoscopes uses a control method based on a rule that positions the tip of the surgical tool at the center of the image (rule-based control). With this control method, it is difficult for users to see the field of view that they want to see.
  • this disclosure proposes an information processing system, an information processing device, and a learning model generation method that enable users to obtain the field of view they want to see.
  • An information processing system includes an imaging device that captures an image of an object, and an evaluation unit that quantitatively evaluates the quality of the field of view of the imaging device based on image information obtained by the imaging device.
  • An information processing device quantitatively evaluates the quality of the field of view of an imaging device based on image information obtained by the imaging device that captures an object.
  • a learning model generation method generates a learning model for quantitatively evaluating the quality of the field of view of an imaging device based on image information obtained by the imaging device that captures an object.
  • FIG. 1 is a diagram illustrating a configuration example of an endoscopic surgery system according to an embodiment of the present disclosure.
  • FIG. 2 is a diagram illustrating an example configuration of a camera head and a CCU according to an embodiment of the present disclosure.
  • 1 is a diagram illustrating an example of a detailed configuration of a support arm device according to an embodiment of the present disclosure.
  • FIG. 1A and 1B are diagrams for explaining an example of a drive system configuration of a support arm device according to an embodiment of the present disclosure.
  • FIG. 1 is a diagram for explaining the flow of a cholecystectomy surgery according to an embodiment of the present disclosure.
  • 1 is a diagram for explaining an example of a field of view (focus) that a user desires to see according to an embodiment of the present disclosure.
  • FIG. 1 is a diagram illustrating a configuration example of an endoscopic surgery system according to an embodiment of the present disclosure.
  • FIG. 2 is a diagram illustrating an example configuration of a camera head and a CCU according
  • FIG. 1A to 1C are diagrams for explaining an example of a field of view (zoom) that a user desires to see according to an embodiment of the present disclosure.
  • 1 is a diagram for explaining an example of a field of view (line of sight direction) that a user desires to see according to an embodiment of the present disclosure.
  • FIG. 1 is a diagram illustrating an example of the configuration of a medical observation system according to an embodiment of the present disclosure.
  • FIG. 1 is a diagram illustrating an example of a main configuration of an information processing device according to an embodiment of the present disclosure.
  • 10 is a flowchart illustrating a flow of an example of advance preparation processing of an information processing device according to an embodiment of the present disclosure.
  • 10 is a flowchart illustrating a flow of an example of a control process of an information processing device according to an embodiment of the present disclosure.
  • FIG. 2 illustrates an example of a hardware configuration.
  • each embodiment can be implemented independently. However, at least a portion of the following embodiments may be implemented in appropriate combination with at least a portion of the other embodiments. These embodiments may include novel features that are different from one another. Thus, each embodiment may contribute to solving a different purpose or problem, and may provide different effects.
  • Embodiments 1-1 Endoscopic surgery system 1-1-1.
  • Configuration example of an endoscopic surgery system 1-1-2 Configuration example of a support arm device 1-1-3.
  • Configuration example of a light source device 1-1-4 Configuration example of a camera head and CCU 1-1-5.
  • Detailed configuration example of a support arm device 1-1-6 Configuration example of a drive system of a support arm device 1-2.
  • Example of advance preparation processing of an information processing device 1-3-4 Example of control processing of an information processing device 1-4.
  • Fig. 1 is a diagram showing an example of the configuration of an endoscopic surgery system 5000 according to this embodiment.
  • an operator (doctor) 5067 is shown using an endoscopic surgery system 5000 to perform surgery on a patient 5071 on a patient bed 5069.
  • the endoscopic surgery system 5000 includes an endoscope 5001, other surgical tools 5017, a support arm device 5027 that supports the endoscope 5001, and a cart 5037 on which various devices for endoscopic surgery are mounted.
  • trocars 5025a to 5025d In endoscopic surgery, instead of cutting the abdominal wall to open the abdomen, for example, cylindrical opening instruments called trocars 5025a to 5025d are punctured into the abdominal wall in multiple places. Then, the endoscope 5003 of the endoscope 5001 and other surgical tools 5017 are inserted into the body cavity of the patient 5071 from the trocars 5025a to 5025d.
  • the other surgical tools 5017 such as an insufflation tube 5019, an energy treatment tool 5021, and forceps 5023, are inserted into the body cavity of the patient 5071.
  • the energy treatment tool 5021 is a treatment tool that uses high-frequency current or ultrasonic vibration to incise and dissect tissue, or seal blood vessels.
  • the surgical tool 5017 shown in FIG. 1 is merely an example, and various surgical tools generally used in endoscopic surgery, such as a tug and a retractor, may be used as the surgical tool 5017.
  • An image of the surgical site inside the body cavity of the patient 5071 captured by the endoscope 5001 is displayed on the display device 5041. While viewing the image of the surgical site displayed on the display device 5041 in real time, the surgeon 5067 performs treatment such as resecting the affected area using the energy treatment tool 5021 and forceps 5023. Although not shown in the figure, the insufflation tube 5019, energy treatment tool 5021, and forceps 5023 are supported by the surgeon 5067 or an assistant, for example, during surgery.
  • the support arm device 5027 includes an arm unit 5031 extending from a base unit 5029.
  • the arm unit 5031 is composed of joints 5033a, 5033b, and 5033c and links 5035a and 5035b, and is driven under the control of an arm control device 5045.
  • the arm unit 5031 supports the endoscope 5001, and controls its position and/or attitude. This allows the endoscope 5001 to be stably fixed in position.
  • the position of the endoscope 5001 indicates the position of the endoscope 5001 in space, and can be expressed as three-dimensional coordinates such as (x, y, z).
  • the attitude of the endoscope 5001 indicates the direction in which the endoscope 5001 is facing, and can be expressed as a three-dimensional vector, for example.
  • the endoscope 5001 is composed of a lens barrel 5003, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 5071, and a camera head 5005 connected to the base end of the lens barrel 5003.
  • the endoscope 5001 is configured as a so-called rigid lens barrel having a rigid lens barrel 5003, but the endoscope 5001 may be configured as a so-called flexible lens barrel having a flexible lens barrel 5003, and is not particularly limited.
  • the tip of the tube 5003 has an opening into which an objective lens is fitted.
  • a light source device 5043 mounted on a cart 5037 is connected to the endoscope 5001, and light generated by the light source device 5043 is guided to the tip of the tube by a light guide extending inside the tube 5003, and is irradiated via the objective lens towards the object to be observed inside the body cavity of the patient 5071.
  • the endoscope 5001 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope, and is not particularly limited.
  • An optical system and an imaging element are provided inside the camera head 5005, and reflected light (observation light) from the object being observed is focused onto the imaging element by the optical system.
  • the imaging element photoelectrically converts the observation light to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observed image.
  • the image signal is sent to a camera control unit (CCU: Camera Control Unit) 5039 as RAW data.
  • the camera head 5005 is equipped with a function for adjusting the magnification and focal length by appropriately driving the optical system.
  • the camera head 5005 may be provided with multiple imaging elements.
  • multiple relay optical systems are provided inside the lens barrel 5003 to guide observation light to each of the multiple imaging elements.
  • the cart 5037 is equipped with a CCU 5039, a light source device 5043, an arm control device 5045, an input device 5047, a treatment tool control device 5049, an insufflation device 5051, a recorder 5053, and a printer 5055.
  • the CCU 5039 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the overall operation of the endoscope 5001 and the display device 5041. Specifically, the CCU 5039 performs various image processing, such as development processing (demosaic processing), on the image signal received from the camera head 5005 in order to display an image based on the image signal. The CCU 5039 provides the image signal that has been subjected to the image processing to the display device 5041. The CCU 5039 also transmits a control signal to the camera head 5005 to control its drive. The control signal may include information regarding imaging conditions, such as magnification and focal length.
  • the display device 5041 under the control of the CCU 5039, displays an image based on an image signal that has been subjected to image processing by the CCU 5039.
  • the endoscope 5001 is compatible with high-resolution imaging, such as 4K (3840 horizontal pixels x 2160 vertical pixels) or 8K (7680 horizontal pixels x 4320 vertical pixels), and/or compatible with 3D display
  • the display device 5041 may be capable of high-resolution display and/or 3D display, respectively.
  • the endoscope is compatible with high-resolution imaging, such as 4K or 8K
  • a display device 5041 of 55 inches or more in size can be used to provide a more immersive experience.
  • multiple display devices 5041 with different resolutions and sizes may be provided depending on the application.
  • the light source device 5043 includes a light source, such as an LED (light emitting diode) and a drive circuit for driving the light source, and supplies illumination light to the endoscope 5001 when photographing the surgical site.
  • a light source such as an LED (light emitting diode)
  • a drive circuit for driving the light source, and supplies illumination light to the endoscope 5001 when photographing the surgical site.
  • the arm control device 5045 includes a processor such as a CPU, and operates according to a predetermined program to control the drive of the arm portion 5031 of the support arm device 5027 according to a predetermined control method.
  • the input device 5047 is an input interface for the endoscopic surgery system 5000.
  • a user can input various information and instructions to the endoscopic surgery system 5000 via the input device 5047.
  • a user inputs various information related to the surgery, such as the patient's physical information and information about the surgical procedure, via the input device 5047.
  • a user inputs via the input device 5047 an instruction to drive the arm portion 5031, an instruction to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) of the endoscope 5001, an instruction to drive the energy treatment tool 5021, etc.
  • the type of the input device 5047 is not limited, and the input device 5047 may be any of various known input devices.
  • a mouse, a keyboard, a touch panel, a switch, a foot switch 5057, and/or a lever may be applied as the input device 5047.
  • the touch panel may be provided on the display surface of the display device 5041.
  • the input device 5047 is a device worn by a user (e.g., the surgeon 5067), such as a glasses-type wearable device or an HMD (Head Mounted Display), and various inputs are made according to the user's gestures and line of sight detected by these devices.
  • a user e.g., the surgeon 5067
  • HMD Head Mounted Display
  • the input device 5047 also includes a camera capable of detecting the user's movements, and various inputs are made according to the user's gestures and line of sight detected from the image captured by the camera. Furthermore, the input device 5047 includes a microphone capable of picking up the user's voice, and various inputs are made by voice via the microphone. In this way, the input device 5047 is configured to be able to input various information in a non-contact manner, which enables a user (e.g., the surgeon 5067) in the clean area to operate equipment in the unclean area in a non-contact manner. In addition, the user can operate the equipment without removing their hands from the surgical tools they are holding, improving user convenience.
  • a user e.g., the surgeon 5067
  • the treatment tool control device 5049 controls the operation of the energy treatment tool 5021 for cauterizing tissue, incising, sealing blood vessels, etc.
  • the insufflation device 5051 sends gas into the body cavity of the patient 5071 via the insufflation tube 5019 in order to inflate the body cavity in order to ensure a clear field of view for the endoscope 5001 and to ensure working space for the surgeon 5067.
  • the recorder 5053 is a device capable of recording various types of information related to the surgery.
  • the printer 5055 is a device capable of printing various types of information related to the surgery in various formats such as text, images, or graphs.
  • the support arm device 5027 includes a base portion 5029, which is a base, and an arm portion 5031 extending from the base portion 5029.
  • the arm portion 5031 is composed of a plurality of joint portions 5033a, 5033b, and 5033c, and a plurality of links 5035a and 5035b connected by the joint portion 5033b.
  • the configuration of the arm portion 5031 is simplified in FIG. 1.
  • the shape, number, and arrangement of the joint portions 5033a to 5033c and the links 5035a and 5035b, as well as the direction of the rotation axis of the joint portions 5033a to 5033c can be appropriately set so that the arm portion 5031 has the desired degree of freedom.
  • the arm portion 5031 can be preferably configured to have six or more degrees of freedom. This allows the endoscope 5001 to be moved freely within the movable range of the arm portion 5031, making it possible to insert the lens barrel 5003 of the endoscope 5001 into the body cavity of the patient 5071 from the desired direction.
  • Actuators are provided in the joints 5033a to 5033c, and the joints 5033a to 5033c are configured to be rotatable around a predetermined rotation axis by driving the actuators.
  • the drive of the actuators is controlled by the arm control device 5045, thereby controlling the rotation angle of each joint 5033a to 5033c and controlling the drive of the arm unit 5031. This makes it possible to control the position and/or attitude of the endoscope 5001.
  • the arm control device 5045 can control the drive of the arm unit 5031 by various known control methods, such as force control or position control.
  • the surgeon 5067 may perform appropriate operation input via the input device 5047 (including the foot switch 5057), and the drive of the arm unit 5031 may be appropriately controlled by the arm control device 5045 in response to the operation input, thereby controlling the position and/or posture of the endoscope 5001.
  • the endoscope 5001 at the tip of the arm unit 5031 may be moved from any position to any position, and then fixedly supported at the position after the movement.
  • the arm unit 5031 may be operated in a so-called master-slave manner.
  • the arm unit 5031 (slave) may be remotely operated by the user via the input device 5047 (master console) installed in a location away from the operating room or in the operating room.
  • the arm control device 5045 may perform so-called power assist control, in which the actuators of the joints 5033a to 5033c are driven so that the arm section 5031 receives an external force from the user and moves smoothly in accordance with the external force.
  • power assist control in which the actuators of the joints 5033a to 5033c are driven so that the arm section 5031 receives an external force from the user and moves smoothly in accordance with the external force.
  • the endoscope 5001 is supported by a doctor called a scopist.
  • the position of the endoscope 5001 can be fixed more reliably without relying on human hands, so images of the surgical site can be obtained stably and the surgery can be performed smoothly.
  • the arm control device 5045 does not necessarily have to be provided on the cart 5037. Furthermore, the arm control device 5045 does not necessarily have to be a single device. For example, an arm control device 5045 may be provided on each of the joints 5033a to 5033c of the arm section 5031 of the support arm device 5027, and the drive control of the arm section 5031 may be realized by multiple arm control devices 5045 working together.
  • Configuration example of light source device> An example of the configuration of a light source device 5043 according to this embodiment will be described with reference to FIG.
  • the light source device 5043 supplies the endoscope 5001 with irradiation light when photographing the surgical site.
  • the light source device 5043 is composed of a white light source composed of, for example, an LED, a laser light source, or a combination of these.
  • the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the white balance of the captured image can be adjusted in the light source device 5043.
  • the light source device 5043 may be controlled to change the intensity of the light it outputs at predetermined time intervals.
  • the image sensor of the camera head 5005 may be controlled to acquire images in a time-division manner in synchronization with the timing of the change in the light intensity, and the images may be synthesized to generate an image with a high dynamic range that is free of so-called blackout and whiteout.
  • the light source device 5043 may also be configured to supply light of a predetermined wavelength band corresponding to special light observation.
  • special light observation for example, by utilizing the wavelength dependency of light absorption in body tissue, a narrow band of light is irradiated compared to the light irradiated during normal observation (i.e., white light), and a predetermined tissue such as blood vessels on the surface of the mucosa is photographed with high contrast, so-called narrow band imaging is performed.
  • fluorescent observation may be performed in which an image is obtained by fluorescence generated by irradiating excitation light.
  • excitation light is irradiated to body tissue and the fluorescence from the body tissue is observed (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue and excitation light corresponding to the fluorescent wavelength of the reagent is irradiated to the body tissue to obtain a fluorescent image.
  • the light source device 5043 may be configured to supply narrow band light and/or excitation light corresponding to such special light observation.
  • Fig. 2 is a diagram showing an example of the detailed configuration of the camera head 5005 and the CCU 5039 shown in Fig. 1.
  • the camera head 5005 has, as its functions, a lens unit 5007, an imaging unit 5009, a drive unit 5011, a communication unit 5013, and a camera head control unit 5015.
  • the CCU 5039 has, as its functions, a communication unit 5059, an image processing unit 5061, and a control unit 5063.
  • the camera head 5005 and the CCU 5039 are connected by a transmission cable 5065 so as to be able to communicate in both directions.
  • the lens unit 5007 is an optical system provided at the connection with the lens barrel 5003. Observation light taken in from the tip of the lens barrel 5003 is guided to the camera head 5005 and enters the lens unit 5007.
  • the lens unit 5007 is composed of a combination of multiple lenses including a zoom lens and a focus lens.
  • the optical characteristics of the lens unit 5007 are adjusted so as to focus the observation light on the light receiving surface of the image sensor of the imaging section 5009.
  • the zoom lens and focus lens are configured so that their positions on the optical axis can be moved to adjust the magnification and focus of the captured image.
  • the imaging unit 5009 is composed of an imaging element, and is arranged after the lens unit 5007.
  • the observation light that passes through the lens unit 5007 is focused on the light receiving surface of the imaging element, and an image signal corresponding to the observation image is generated by photoelectric conversion.
  • the image signal generated by the imaging unit 5009 is provided to the communication unit 5013.
  • the imaging element constituting the imaging unit 5009 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) type image sensor that is capable of color imaging with R (red), G (green), and B (blue) color filters arranged in a Bayer array.
  • the imaging element may be capable of capturing high-resolution images of, for example, 4K or higher. Obtaining a high-resolution image of the surgical site allows the surgeon 5067 to grasp the state of the surgical site in more detail, enabling the surgery to proceed more smoothly.
  • the imaging element constituting the imaging unit 5009 is configured to have a pair of imaging elements for acquiring image signals for the right eye and the left eye, respectively, corresponding to 3D display.
  • 3D display allows the surgeon 5067 to more accurately grasp the depth of the biological tissue in the surgical site. Note that when the imaging unit 5009 is configured as a multi-plate type, multiple lens units 5007 are also provided corresponding to each imaging element.
  • the imaging unit 5009 does not necessarily have to be provided in the camera head 5005.
  • the imaging unit 5009 may be provided inside the lens barrel 5003, immediately after the objective lens.
  • the driving unit 5011 is composed of an actuator, and moves the zoom lens and focus lens of the lens unit 5007 a predetermined distance along the optical axis under the control of the camera head control unit 5015. This allows the magnification and focus of the image captured by the imaging unit 5009 to be adjusted appropriately.
  • the communication unit 5013 is composed of a communication device for transmitting and receiving various information to and from the CCU 5039.
  • the communication unit 5013 transmits the image signal obtained from the imaging unit 5009 as RAW data to the CCU 5039 via the transmission cable 5065.
  • the image signal is transmitted by optical communication.
  • the surgeon 5067 performs surgery while observing the condition of the affected area using the captured image, so for a safer and more reliable surgery, it is required that the moving image of the surgical site be displayed as quickly as possible in real time.
  • the communication unit 5013 is provided with a photoelectric conversion module that converts an electrical signal into an optical signal.
  • the image signal is converted into an optical signal by the photoelectric conversion module and then transmitted to the CCU 5039 via the transmission cable 5065.
  • the communication unit 5013 also receives control signals from the CCU 5039 for controlling the operation of the camera head 5005.
  • the control signals include information related to the imaging conditions, such as information specifying the frame rate of the captured image, information specifying the exposure value during imaging, and/or information specifying the magnification and focus of the captured image.
  • the communication unit 5013 provides the received control signals to the camera head control unit 5015.
  • the control signals from the CCU 5039 may also be transmitted by optical communication.
  • the communication unit 5013 is provided with a photoelectric conversion module that converts optical signals into electrical signals, and the control signals are converted into electrical signals by the photoelectric conversion module before being provided to the camera head control unit 5015.
  • the above-mentioned frame rate, exposure value, magnification, focus, and other imaging conditions are automatically set by the control unit 5063 of the CCU 5039 based on the acquired image signal.
  • the endoscope 5001 is equipped with so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function.
  • the camera head control unit 5015 controls the driving of the camera head 5005 based on a control signal received from the CCU 5039 via the communication unit 5013. For example, the camera head control unit 5015 controls the driving of the image sensor of the imaging unit 5009 based on information specifying the frame rate of the captured image and/or information specifying the exposure during imaging. Also, for example, the camera head control unit 5015 appropriately moves the zoom lens and focus lens of the lens unit 5007 via the drive unit 5011 based on information specifying the magnification and focus of the captured image.
  • the camera head control unit 5015 may further include a function for storing information for identifying the lens barrel 5003 and the camera head 5005.
  • the camera head 5005 can be made resistant to autoclave sterilization.
  • the communication unit 5059 is composed of a communication device for transmitting and receiving various information to and from the camera head 5005.
  • the communication unit 5059 receives an image signal transmitted from the camera head 5005 via the transmission cable 5065.
  • the image signal can be suitably transmitted by optical communication.
  • the communication unit 5059 is provided with a photoelectric conversion module that converts an optical signal into an electrical signal.
  • the communication unit 5059 provides the image signal converted into an electrical signal to the image processing unit 5061.
  • the communication unit 5059 also transmits a control signal to the camera head 5005 for controlling the operation of the camera head 5005.
  • This control signal may also be transmitted by optical communication.
  • the image processing unit 5061 performs various types of image processing on the image signal, which is RAW data sent from the camera head 5005.
  • the image processing includes various known signal processing such as development processing, high image quality processing and/or enlargement processing (electronic zoom processing).
  • the high image quality processing includes, for example, band enhancement processing, super-resolution processing, NR (Noise reduction) processing and/or camera shake correction processing.
  • the image processing unit 5061 also performs detection processing on the image signal to perform AE, AF and AWB.
  • the image processing unit 5061 is configured with a processor such as a CPU or GPU, and the above-mentioned image processing and detection processing can be performed by the processor operating according to a specified program. Note that if the image processing unit 5061 is configured with multiple GPUs, the image processing unit 5061 divides the information related to the image signal appropriately and performs image processing in parallel using these multiple GPUs.
  • the control unit 5063 performs various controls related to the imaging of the surgical site by the endoscope 5001 and the display of the captured images. For example, the control unit 5063 generates a control signal for controlling the driving of the camera head 5005. At this time, if the imaging conditions have been input by the user, the control unit 5063 generates a control signal based on the input by the user. Alternatively, if the endoscope 5001 is equipped with an AE function, an AF function, and an AWB function, the control unit 5063 appropriately calculates the optimal exposure value, focal length, and white balance according to the results of the detection processing by the image processing unit 5061, and generates a control signal.
  • the control unit 5063 also causes the display device 5041 to display an image of the surgical site based on the image signal that has been image-processed by the image processing unit 5061. At this time, the control unit 5063 uses various image recognition techniques to recognize various objects in the surgical site image. For example, the control unit 5063 can recognize surgical tools such as forceps, specific body parts, bleeding, mist generated when the energy treatment tool 5021 is used, and the like, by detecting the shape and color of the edges of objects contained in the surgical site image.
  • the control unit 5063 causes the display device 5041 to display an image of the surgical site, it uses the recognition results to superimpose various types of surgical support information on the image of the surgical site. The surgical support information is superimposed and presented to the surgeon 5067, making it possible to proceed with the surgery more safely and reliably.
  • the transmission cable 5065 connecting the camera head 5005 and the CCU 5039 is an electrical signal cable for electrical signal communication, an optical fiber for optical communication, or a composite cable of these.
  • communication is performed wired using the transmission cable 5065, but communication between the camera head 5005 and the CCU 5039 may be performed wirelessly. If communication between the two is performed wirelessly, there is no need to lay the transmission cable 5065 in the operating room, which can eliminate the situation where the transmission cable 5065 impedes the movement of medical staff in the operating room.
  • Fig. 3 is a diagram showing a detailed configuration example of the support arm device 400 according to this embodiment.
  • the support arm device 400 corresponds to, for example, the support arm device 5027 described above.
  • the support arm device 400 includes a base unit 410 and an arm unit 420.
  • the base unit 410 is the base of the support arm device 400, and the arm unit 420 extends from the base unit 410.
  • a control unit that controls the support arm device 400 in an integrated manner may be provided within the base unit 410, and the drive of the arm unit 420 may be controlled by the control unit.
  • the control unit is composed of various signal processing circuits, such as a CPU, DSP, etc.
  • the arm section 420 has a plurality of active joints 421a-421f, a plurality of links 422a-422f, and an endoscope device 423 as a tip unit provided at the tip of the arm section 420.
  • the links 422a-422f are roughly rod-shaped members. One end of the link 422a is connected to the base section 410 via the active joint 421a, the other end of the link 422a is connected to one end of the link 422b via the active joint 421b, and the other end of the link 422b is connected to one end of the link 422c via the active joint 421c.
  • the other end of the link 422c is connected to the link 422d via the passive slide mechanism 431, and the other end of the link 422d is connected to one end of the link 422e via the passive joint 433.
  • the other end of link 422e is connected to one end of link 422f via active joints 421d and 421e.
  • the endoscope device 423 is connected to the tip of the arm unit 420, i.e., the other end of link 422f, via active joint 421f. In this way, the ends of the multiple links 422a to 422f are connected to each other by active joints 421a to 421f, passive slide mechanism 431, and passive joint 433, with the base unit 410 as the fulcrum, forming an arm shape extending from the base unit 410.
  • the actuators provided at each of the active joints 421a to 421f of the arm section 420 are driven and controlled to control the position and attitude of the endoscope device 423.
  • the endoscope device 423 enters the patient's body cavity, which is the treatment site, with its tip, and captures an image of a partial area of the treatment site.
  • the tip unit provided at the tip of the arm section 420 is not limited to the endoscope device 423, and various medical instruments may be connected to the tip of the arm section 420 as a tip unit.
  • the support arm device 400 according to this embodiment is configured as a medical support arm device equipped with medical instruments.
  • the support arm device 400 will be described by defining the coordinate axes as shown in FIG. 3. Additionally, the up-down direction, front-back direction, and left-right direction are defined in accordance with the coordinate axes. That is, the up-down direction with respect to the base unit 410 placed on the floor surface is defined as the z-axis direction and up-down direction. Additionally, the direction perpendicular to the z-axis and in which the arm unit 420 extends from the base unit 410 (i.e., the direction in which the endoscope device 423 is positioned relative to the base unit 410) is defined as the y-axis direction and front-back direction. Additionally, the direction perpendicular to the y-axis and z-axis is defined as the x-axis direction and left-right direction.
  • the active joints 421a to 421f connect the links to each other so that they can rotate.
  • the active joints 421a to 421f have actuators and have rotation mechanisms that are driven to rotate about a predetermined rotation axis by the actuators.
  • By controlling the rotation drive of each of the active joints 421a to 421f it is possible to control the drive of the arm 420, for example, to extend or retract (fold) the arm 420.
  • the drive of the active joints 421a to 421f can be controlled by, for example, known whole-body cooperative control and ideal joint control.
  • the drive control of the active joints 421a to 421f specifically means that the rotation angle and/or generated torque (torque generated by the active joints 421a to 421f) of the active joints 421a to 421f are controlled.
  • the passive sliding mechanism 431 is one aspect of the passive shape changing mechanism, and connects the links 422c and 422d so that they can move back and forth relative to each other along a predetermined direction.
  • the passive sliding mechanism 431 may connect the links 422c and 422d so that they can move linearly relative to each other.
  • the forward and backward movement of the links 422c and 422d is not limited to linear movement, and may be forward and backward movement in a direction forming an arc.
  • the passive sliding mechanism 431 is operated to move forward and backward by a user, for example, and changes the distance between the active joint 421c on one end side of the link 422c and the passive joint 433. This allows the overall shape of the arm unit 420 to change.
  • the passive joint 433 is one aspect of a passive shape changing mechanism, and connects the links 422d and 422e to each other so that they can rotate.
  • the passive joint 433 is rotated by a user, for example, to change the angle between the links 422d and 422e. This allows the overall shape of the arm 420 to change.
  • the "posture of the arm unit” refers to the state of the arm unit that can be changed by the control unit driving the actuators provided in the active joint units 421a to 421f when the distance between adjacent active joint units across one or more links is constant.
  • the "posture of the arm unit” is not limited to the state of the arm unit that can be changed by the control unit driving the actuators.
  • the “posture of the arm unit” may be the state of the arm unit that is changed by the cooperative operation of the joint units.
  • the arm unit does not necessarily need to include joint units.
  • the "posture of the arm unit” is the position with respect to the target object or the relative angle with respect to the target object.
  • the "shape of the arm unit” refers to the state of the arm unit that can change as the passive shape change mechanism is operated, changing the distance between adjacent active joint units across a link, or the angle between the links connecting adjacent active joint units.
  • the "shape of the arm unit” is not limited to the state of the arm unit that can change as the distance between adjacent active joint units across a link, or the angle between the links connecting adjacent active joint units, changes.
  • the “shape of the arm unit” may be the state of the arm unit that can change as the positional relationship between the joint units or the angle changes as the joint units operate in a coordinated manner.
  • the "shape of the arm unit” may be the state of the arm unit that can change as the position relative to an object or the relative angle to the object changes.
  • the support arm device 400 has six active joints 421a to 421f, and six degrees of freedom are achieved with respect to the drive of the arm section 420.
  • the drive control of the support arm device 400 is achieved by the control section controlling the drive of the six active joints 421a to 421f, while the passive slide mechanism 431 and the passive joint section 433 are not subject to drive control by the control section.
  • the active joints 421a, 421d, and 421f are arranged so that the long axis direction of each connected link 422a, 422e and the imaging direction of the connected endoscope device 423 are the rotation axis direction.
  • the active joints 421b, 421c, and 421e are arranged so that the rotation axis direction is the x-axis direction, which is the direction in which the connection angle of each connected link 422a to 422c, 422e, and 422f and the endoscope device 423 is changed within the y-z plane (the plane defined by the y-axis and z-axis).
  • the active joints 421a, 421d, and 421f have a function of performing so-called yawing
  • the active joints 421b, 421c, and 421e have a function of performing so-called pitching.
  • the support arm device 400 realizes six degrees of freedom for driving the arm unit 420, so that the endoscope device 423 can be moved freely within the movable range of the arm unit 420.
  • a hemisphere is illustrated as an example of the movable range of the endoscope device 423.
  • the endoscope device 423 can be moved on the spherical surface of the hemisphere with the imaging center of the endoscope device 423 fixed to the center point of the hemisphere, thereby allowing the treatment site to be imaged from various angles.
  • the arm unit 420 of the support arm device 400 has been described as having multiple joints and six degrees of freedom, the present disclosure is not limited to this.
  • the arm unit 420 may have a structure in which an endoscope 5001 or an endoscope is provided at the tip.
  • the arm unit 420 may be configured with only one degree of freedom to drive the endoscope 5001 to move in the direction of entering the patient's body cavity and in the direction of retracting.
  • Fig. 4 is a diagram for explaining an example of the drive system configuration of the support arm device 400 according to this embodiment.
  • the arm unit 420 of the support arm device 400 includes a first joint unit 111-1 , a second joint unit 111-2 , a third joint unit 111-3 , and a fourth joint unit 111-4 .
  • the first joint unit 111-1 , the second joint unit 111-2 , the third joint unit 111-3 , and the fourth joint unit 111-4 function as actuators.
  • an endoscope device 423 is attached to the first joint unit 111-1 .
  • a posture control unit 550 is connected to the support arm device 400.
  • a user interface unit 570 is connected to the posture control unit 550. Note that in the example of Fig. 4, the arm unit 420 is shown in a simplified form of the arm unit 420 shown in Fig. 3.
  • the endoscope device 423 corresponds to, for example, the endoscope 5001 described above.
  • the attitude control unit 550 is included, for example, in the arm control device 5045 described above, and the user interface unit 570 is included, for example, in the input device 5047 described above.
  • the first joint unit 111-1 has a motor 501-1 , an encoder 502-1 , a motor controller 503-1 , and a motor driver 504-1 .
  • the second joint unit 111-2 to the fourth joint unit 111-4 have the same configuration as the first joint unit 111-1 . That is, the second joint unit 111-2 has a motor 501-2 , an encoder 502-2 , a motor controller 503-2 , and a motor driver 504-2 .
  • the third joint unit 111-3 has a motor 501-3 , an encoder 502-3 , a motor controller 503-3 , and a motor driver 504-3 .
  • the fourth joint unit 111-4 has a motor 501-4 , an encoder 502-4 , a motor controller 503-4 , and a motor driver 504-4 .
  • the first joint unit 111-1 will be described as an example.
  • the motor 501 1 is driven under the control of the motor driver 504 1 to drive the first joint unit 111 1.
  • the motor 501 1 drives the first joint unit 111 1, for example, in the direction of an arrow attached to the first joint unit 111 1.
  • the motor 501 1 changes the configuration (for example, the position and/or posture) of the arm unit 420 and adjusts the position and/or posture of the endoscope device 423 supported by the arm unit 420.
  • the encoder 502 1 detects information relating to the rotation angle of the first joint unit 111 1 under the control of the motor controller 503 1. That is, the encoder 502 1 obtains information relating to the attitude of the first joint unit 111 1 .
  • the attitude control unit 550 changes the form of the arm unit 420 and controls the position and/or attitude of the endoscope device 423 supported by the arm unit 420. Specifically, the attitude control unit 550 controls the motor controllers 503.sub.1 to 503.sub.4 and the motor drivers 504.sub.1 to 504.sub.4 , etc. , to control the first joint unit 111.sub.1 to the fourth joint unit 111.sub.4 . In this way, the attitude control unit 550 controls the position and/or attitude of the endoscope device 423 supported by the arm unit 420.
  • the user interface unit 570 accepts various operations from the user.
  • the user interface unit 570 accepts an operation for controlling the position and/or attitude of the endoscope device 423 supported by the arm unit 420.
  • the user interface unit 570 outputs an operation signal corresponding to the accepted operation to the attitude control unit 550.
  • the attitude control unit 550 controls the first joint unit 111-1 to the fourth joint unit 111-4 in accordance with the operation accepted from the user interface unit 570 to control the position and/or attitude of the endoscope device 423 supported by the arm unit 420.
  • the electronic degrees of freedom for changing the line of sight and the degrees of freedom provided by the actuator of the arm unit 420 are all treated as degrees of freedom of the robot. This makes it possible to realize motion control that links the electronic degrees of freedom for changing the line of sight and the degrees of freedom provided by the actuator.
  • an endoscopic surgery system 5000 to which the technology disclosed herein can be applied.
  • the endoscopic surgery system 5000 has been described here as an example, systems to which the technology disclosed herein can be applied are not limited to this example.
  • the technology disclosed herein may be applied to a flexible endoscopic surgery system for inspection or a microsurgery system.
  • FIG. 5 is a diagram for explaining the flow of a gallbladder removal surgery according to this embodiment.
  • FIGs 6 to 8 is a diagram for explaining an example of a field of view that a user wants to see according to this embodiment.
  • a cholecystectomy the surgeon, who is an example of a user, sequentially performs the following procedures: development of Calot's triangle S1, preparation of the cystic artery S2, preparation of the cystic duct S3, resection of the gallbladder S4, removal of the cholecystectomy S5, hemostasis of the gall bed S6, removal of the cholecystectomy S7, and removal of the trocha and placement of a drain S8.
  • the surgeon performs the cholecystectomy while viewing the images and videos displayed on the display device 5041.
  • Calot's triangle S1 includes the procedures of "a1: trocar placement and pneumoperitoneum”, “a2: insertion (insertion of endoscope 5001) and observation of internal conditions”, “a3: insertion of left and right surgical tools (insertion of surgical tool 5017) and traction of the gallbladder”, “a4: movement to Calot's triangle”, “a5: insertion of assistant tool”, “a6: traction of the gallbladder with left and right surgical tools", “a7: dissection”, “a8: hemostasis with electric scalpel”, and “a9: continuation of dissection".
  • the cystic artery treatment S2 includes the procedures of "b1: exposure of the cystic artery and cystic duct", "b2: cystic artery clip (multiple times)", “b3: dissection”, and “b4: confirmation of dissection and removal of instruments”.
  • the cystic duct treatment S3 includes the procedures of "c1: cystic duct clip (multiple times)", “c2: dissection", and "c3: change of instruments”.
  • Gallbladder resection S4 includes the procedures “d1: grasp and ignite target (multiple times)", “d2: resection (multiple times)", “d3: remove clip”, “d4: suction”, “d5: grasp gallbladder”, and “d6: resection”.
  • Cholecystectomy S5 includes the procedures “e1: put gallbladder into bag”, “e2: close mouth of bag”, and “e3: remove bag”.
  • Gallbed hemostasis S6 includes the procedures “f1: hemostasis” and "f2: suction”. The individual steps of "f1: hemostasis” and “f2: suction” are repeated several times.
  • Cholecystectomy S7 includes the procedures “g1: insert main instrument” and “g2: grasp target and pull in one go”.
  • Trocar removal and drain placement S8 includes the procedures “h1: suction”, “h2: trocar removal”, “h3: drain placement”, and "h4: suturing”.
  • autonomous control of the endoscope 5001 which is moved by the support arm device 400, requires control that allows the user, the surgeon, to see the field of view that he or she wants to see.
  • autonomous control employs a control method based on a rule that aligns the tip of the surgical tool with the center of the image (rule-based control).
  • rule-based control there are cases where the surgeon is unable to see the field of view that he or she wants to see. Three specific examples are given below.
  • rule-based control only looks at the tip of the surgical tool.
  • Figure 6 there are cases where the surgeon wants to see not only the position A1 of the surgical tool tip, but also the position A2 to be treated/confirmed.
  • rule-based control determines the camera position so as to maintain a certain distance from the tip of the surgical tool.
  • Figure 7 there are cases where the surgeon wants to clearly see the part B1 to be processed/checked, not too close or too far, but at an appropriate size.
  • rule-based control views the object based on position (point) information, so direction is not taken into consideration in the first place, and the object is basically viewed from the front.
  • position (point) information so direction is not taken into consideration in the first place, and the object is basically viewed from the front.
  • Figure 8 there are cases where the surgeon wants to view the object not only from the front, but also from a direction in which he or she is peering in by changing the line of sight of camera C1.
  • the abdominal cavity being an unstructured environment means that the environment inside the abdominal cavity changes dynamically, for example, as organs move, blood flows, and smoke is produced when an electric scalpel is used.
  • the abdominal cavity being an unstructured environment also means that, for example, the shape and size of organs differ from person to person, and furthermore, the lighting conditions in the operating room differ, and the way shadows are cast during surgery also differ, so the way objects are seen can change.
  • the need to take into account the flow of surgery means, for example, that even with the same field of view, the quality of the field of view will vary depending on the surgical procedure (technique). Also, even if the same field of view has been seen before, there is no need to look at it, but if it has not been seen before, it is necessary to look at it; this kind of chronological information must be taken into consideration. It is also necessary to take into account chronological information and predict what the field of view should be for the next procedure once the current procedure is completed. Also, although the field of view needs to be considered depending on the procedure, it is difficult to clearly distinguish between procedures in the first place.
  • autonomous control of the imaging device 12 of the endoscope 5001 requires control to obtain the field of view that a user, such as a surgeon, wants to see, but with rule-based control, it is difficult to obtain the field of view that the user wants to see. Therefore, the following describes a medical observation system 1 that enables the user to obtain the field of view that he or she wants to see.
  • Fig. 9 is a diagram showing an example of the configuration of the medical observation system 1 according to this embodiment.
  • the medical observation system 1 is applied to the above-mentioned endoscopic surgery system 5000.
  • the medical observation system 1 is an example of an information processing system.
  • the medical observation system 1 includes a robot arm device 10, an imaging device 12, a light source device 13, an operation device 14, an information processing device 20, a display device 40, and a storage device 60.
  • the robot arm device 10 corresponds to, for example, the support arm device 400 described above.
  • the imaging device 12 corresponds to, for example, the endoscope 5001 described above.
  • the light source device 13 corresponds to, for example, the light source device 5043 described above.
  • the operation device 14 corresponds to, for example, the input device 5047 described above.
  • the information processing device 20 includes, for example, the CCU 5039 and the arm control device 5045 described above.
  • the display device 40 corresponds to, for example, the display device 5041 described above.
  • an imaging device 12 is inserted into the patient's body through a medical puncture device called a trocar, and the surgeon 5067 performs laparoscopic surgery while photographing the area of interest. At this time, the imaging device 12 can freely change the photographing position by driving the robot arm device 10.
  • the medical observation system 1 captures images of the patient's abdominal cavity using the imaging device 12, recognizes the environment within the abdominal cavity, and drives the robot arm device 10 based on the recognition result of the abdominal environment. Driving the robot arm device 10 changes the imaging range within the abdominal cavity. When the imaging range within the abdominal cavity changes, the medical observation system 1 recognizes the changed environment and drives the robot arm device 10 based on the recognition result.
  • the medical observation system 1 repeats image recognition of the intraperitoneal environment and driving the robot arm device 10. In other words, the medical observation system 1 executes a process that combines image recognition processing and processing for controlling the position and/or posture of the robot arm device 10.
  • the robot arm device 10 has an arm section 11 which is a multi-link structure, and controls the position and/or posture of a tip unit (e.g., an image capture device 12) provided at the tip of the arm section 11 which is a multi-joint arm, by driving the arm section 11 within a movable range.
  • the arm section 11 corresponds to, for example, the arm section 420 described above.
  • the arm unit 11 is composed of, for example, a plurality of joints (e.g., active joints 421a to 421f) and a plurality of links (e.g., links 422a to 422f).
  • This arm unit 11 is controlled by the arm control unit 23.
  • one joint 11a is shown as a representative of the plurality of joints.
  • the joint 11a rotatably connects the links in the arm unit 11, and drives the arm unit 11 by controlling its rotational drive by the arm control unit 23.
  • the arm unit 11 may also have a motion sensor (not shown) including an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc., to obtain information on the configuration (e.g., position and/or posture) of the arm unit 11.
  • the electronic degree of freedom to change the line of sight by cutting out a specific area from the captured image using the wide-angle/cut-out function, and the degree of freedom provided by the actuator of the arm unit 11 are all treated as the degrees of freedom of the robot. This makes it possible to realize motion control that links the electronic degree of freedom to change the line of sight with the degree of freedom of the joints provided by the actuator.
  • the imaging device 12 is provided at the tip of the arm 11 and captures various imaging objects such as parts of a patient.
  • the imaging device 12 captures an image of the patient's abdominal cavity and obtains an image including various medical instruments and organs in the abdominal cavity of the patient.
  • the imaging device 12 may be, for example, a direct endoscope, an oblique endoscope, a monocular endoscope, a stereo endoscope, or a microscope, and is not particularly limited.
  • the imaging device 12 is a stereo endoscope
  • distance information can be obtained from the stereo images obtained by stereo matching.
  • the imaging device 12 is a monocular endoscope
  • distance information can be obtained by performing distance estimation based on machine learning or using a method such as SfM (Structure from Motion).
  • the imaging device 12 includes a camera capable of capturing an image of an object in the form of a video or still image.
  • the camera is, for example, a wide-angle camera configured with a wide-angle optical system.
  • the angle of view of a normal camera is about 80°, whereas the angle of view of the camera according to this embodiment may be 140°.
  • the angle of view of the camera may be less than 140° as long as it exceeds 80°, or may be 140° or more.
  • the imaging device 12 transmits an electrical signal (pixel signal) corresponding to the captured image to the information processing device 20.
  • the arm portion 11 may support a medical instrument such as forceps 5023.
  • an endoscope other than a stereo endoscope may be used as the imaging device 12, and a depth sensor (distance measuring device) may be provided separately from the imaging device 12.
  • the imaging device 12 may be a monocular endoscope.
  • the depth sensor may be, for example, a sensor that measures distance using a ToF (Time of Flight) method that measures distance using the return time of reflection of pulsed light from a subject, or a structured light method that measures distance by irradiating a lattice-shaped pattern light and measuring distance by distortion of the pattern.
  • the imaging device 12 itself may also be provided with a depth sensor. In this case, the imaging device 12 can perform distance measurement using the ToF method at the same time as capturing an image.
  • the imaging device 12 includes multiple light receiving elements, and can generate an image and calculate distance information based on pixel signals obtained from the light receiving elements.
  • the light source device 13 irradiates light onto an object to be imaged by the imaging device 12.
  • the light source device 13 can be realized, for example, by an LED (Light Emitting Diode) for a wide-angle lens.
  • the light source device 13 may be configured, for example, by combining a normal LED with a lens to diffuse light.
  • the light source device 13 may also be configured to diffuse (widen the angle of) light transmitted by an optical fiber (for example, a light guide) with a lens.
  • the light source device 13 may also widen the irradiation range by irradiating light in multiple directions from the optical fiber itself.
  • the operation device 14 has, for example, one or more operators, and outputs operation information according to the operation of a user (for example, a doctor) on the operator.
  • a user for example, a doctor
  • the operator of the operation device 14 for example, a switch, a lever (including a joystick), a foot switch, a touch panel, etc. that the user operates by directly or indirectly touching it can be applied.
  • a microphone that detects voice, a gaze sensor that detects gaze, etc. can also be applied as the operator.
  • the information processing device 20 includes a processing unit 21, an imaging control unit 22, an arm control unit 23, an input unit 24, a display control unit 25, and an action planning unit 26.
  • the imaging control unit 22 corresponds to, for example, the above-mentioned CCU
  • the arm control unit 23 corresponds to, for example, the above-mentioned arm control device 5045.
  • the information processing device 20 may be realized by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing a program using a RAM (Random Access Memory) or the like as a working area.
  • the information processing device 20 may also be a controller, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
  • ASIC Application Specific Integrated Circuit
  • FPGA Field Programmable Gate Array
  • the processing unit 21 has an acquisition unit 211, a processing unit 212, an evaluation unit 213, and a generation unit 214.
  • the acquisition unit 211 acquires image information related to the captured image obtained by the imaging device 12. For example, the acquisition unit 211 sends the acquired image information to the processing unit 212, the evaluation unit 213, etc.
  • the acquisition unit 211 may be configured, for example, by a capture board or a memory.
  • the processing unit 212 performs various processes on the captured image obtained by the imaging device 12, and generates various images from the captured image.
  • the processing unit 212 can generate a surgical field image by extracting an image of a display target area from the captured image.
  • the display target area is, for example, an area of interest to a user such as a doctor (ROI: Region of Interest).
  • the processing unit 212 may also change the display target area in accordance with, for example, a user's operation on the operation device 14 or action plan information obtained by the action plan unit 26, and extract the display target area.
  • the processing unit 212 generates a surgical field image by cutting out and enlarging a display target area from the captured image.
  • the processing unit 212 may change the cut-out position depending on the position and/or posture of the imaging device 12 supported by the arm unit 11. For example, when the position and/or posture of the imaging device 12 changes, the processing unit 212 can change the cut-out position so that the surgical field image displayed on the display screen of the display device 40 does not change.
  • the processing unit 212 can also perform, for example, image quality improvement processing on the operative field image.
  • the processing unit 212 can perform super-resolution processing, band enhancement processing, noise removal processing, camera shake correction processing, brightness correction processing, etc., as image quality improvement processing on the operative field image.
  • the processing unit 212 can also perform low-resolution processing on the operative field image to reduce the size of the operative field image.
  • the processing unit 212 can also perform distortion correction, etc., on the operative field image. By performing distortion correction on the operative field image, it is possible to improve the accuracy of visual field evaluation in the evaluation unit 213, which will be described later.
  • the evaluation unit 213 performs visual field evaluation to quantitatively evaluate the quality of the visual field based on the image information sent from the acquisition unit 211 or the processing unit 212. For example, the evaluation unit 213 obtains a visual field evaluation value indicating the quality of the visual field and sends it to the action planning unit 26.
  • the evaluation unit 213 recognizes, for example, the position, trajectory, area, type, etc. of surgical tools, as well as the area and type of organs, based on the image information, makes a task judgment by referring to the surgical information based on the recognition result (for example, feature information), and quantitatively evaluates the quality of the visual field by referring to attribute information based on the recognition result and the task judgment result.
  • the evaluation unit 213 will be described in detail later.
  • the generating unit 214 generates various information required by the evaluating unit 213, the behavior planning unit 26, etc.
  • the various information includes, for example, any or all of the following information: surgery information, attribute information, evaluation model (learning model), behavior planning model (learning model), and environmental information. Such various information is, for example, generated in advance. This various information may be stored in the storage device 60.
  • the various information generated by the generating unit 214 i.e., the various databases, will be described in detail later.
  • the imaging control unit 22 controls the imaging device 12.
  • the imaging control unit 22 controls the imaging device 12 to capture an image of the surgical field.
  • the imaging control unit 22 controls, for example, the magnification ratio of the image captured by the imaging device 12.
  • the imaging control unit 22 may control the magnification ratio of the imaging device 12, or may control the focus (focal length) of the imaging device 12, or may control the gain (sensitivity) of the imaging device 12 (more specifically, the image sensor of the imaging device 12) based on operation information input to the input unit 24.
  • the imaging control unit 22 also controls the light source device 13. For example, the imaging control unit 22 controls the brightness of the light source device 13 when the imaging device 12 images the surgical field.
  • the imaging control unit 22 may control the brightness of the light source device 13 based on operation information input to the input unit 24, for example.
  • the operation information is generated when a user, such as a doctor, operates the operation device 14.
  • the arm control unit 23 comprehensively controls the robot arm device 10 and also controls the drive of the arm unit 11. Specifically, the arm control unit 23 controls the drive of the joint unit 11a, thereby controlling the drive of the arm unit 11. More specifically, the arm control unit 23 controls the amount of current supplied to the motor in the actuator of the joint unit 11a, thereby controlling the number of rotations of the motor, and thereby controlling the rotation angle and generated torque at the joint unit 11a.
  • the arm control unit 23 can control the shape of the arm unit 11 and control the position and/or posture of the imaging device 12 supported by the arm unit 11.
  • the arm control unit 23 can control the shape of the arm unit 11 according to, for example, operation information input to the input unit 24 and action plan information obtained by the action plan unit 26.
  • the input unit 24 accepts various types of operation information output from the operation device 14, and outputs it to, for example, the processing unit 21, the imaging control unit 22, the arm control unit 23, the action planning unit 26, etc.
  • the operation information may be input by a physical mechanism (for example, an operator) or may be input by voice.
  • the operation information from the operation device 14 is instruction information for changing the magnification ratio (zoom amount) of the imaging device 12, or the position and/or posture of the arm unit 11.
  • the display control unit 25 causes various images to be displayed on the display device 40.
  • the display control unit 25 outputs the operative field image generated by the processing unit 212 and the captured image obtained by the acquisition unit 211 to the display device 40 for display.
  • the display control unit 25 generates a display signal that can be displayed by the display device 40 based on the operative field image and the captured image, and supplies the display device 40 with the display signal.
  • the action planning unit 26 plans the action of the robot arm device 10 based on the visual field evaluation information (e.g., visual field evaluation value) obtained by the evaluation unit 213, and obtains action plan information related to the action plan.
  • Action planning for example, involves determining a target position and/or target posture of the imaging device 12.
  • the action plan information includes, for example, information related to the determined target position and/or target posture (e.g., coordinate position, posture angle, etc.).
  • the arm control unit 23 can control the shape of the arm unit 11, for example, according to the action plan information obtained by the action planning unit 26. In response to this control, the robot arm device 10 changes the shape of the arm unit 11 so as to position the imaging device 12 at the target position and/or target posture based on the action plan information.
  • the action planning unit 26 will be described in detail later.
  • the display device 40 displays various images.
  • the display device 40 displays images such as an image captured by the imaging device 12 and an operative field image captured by the processing unit 212 on a display screen.
  • the display device 40 may be a display such as a liquid crystal display or an organic EL (Electro-Luminescence) display.
  • a plurality of display devices 40 may be provided depending on the application.
  • a user such as a doctor can perform, for example, an endoscopic surgery while viewing the images and videos displayed on the display device 40.
  • the storage device 60 stores various types of information.
  • the storage device 60 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.
  • the storage device 60 is, for example, a storage device, and may include various databases such as a cloud-type database.
  • Fig. 10 is a diagram showing an example of the main configuration of the information processing device 20.
  • the evaluation unit 213 has a feature extraction unit 213a, a task judgment unit 213b, and a visual field evaluation unit 213c.
  • a surgery information DB (database) 61, an attribute information DB 62, an evaluation model DB 63, an action plan model DB 64, and an environmental information DB 65 are connected to the information processing device 20. These databases 61 to 65 may be included in the storage device 60, for example.
  • the feature extraction unit 213a extracts features from the image information and obtains feature information related to the features. For example, the feature extraction unit 213a performs processing such as dimensional compression on the image information and extracts features from the image information.
  • the features include, for example, any or all of the following information: the position, trajectory, area, and type of a surgical tool, and the area and type of an organ. This information may be obtained, for example, by machine learning.
  • each of the above information can be obtained by semantic segmentation (SemaSegment) using machine learning.
  • SemaSegment obtains area information and semantic information, but in this embodiment, for example, surgical tool area information and surgical tool type information, as well as organ area information and organ type information are obtained.
  • the tip position can also be obtained.
  • the trajectory of the surgical tool can also be obtained based on image information from multiple frames. This information is obtained as feature information.
  • the work judgment unit 213b performs work judgment based on the feature information acquired by the feature extraction unit 213a, and acquires work information related to the work.
  • Work judgment is information about which procedure is currently being performed within the flow of a surgical procedure. For example, work judgment is determining where the current work is located within the flow of a gallbladder removal surgery shown in Figure 5.
  • the granularity of the work may be finer, or conversely, coarser.
  • the visual field evaluation unit 213c which performs visual field evaluation in the subsequent stage, can perform work judgment with a granularity necessary and sufficient for evaluation.
  • the work judgment unit 213b may refer to the surgery information DB 61 and make a work judgment based on the surgery information obtained from the surgery information DB 61.
  • the surgery information DB 61 holds surgery information related to surgery.
  • the surgery information includes, for example, surgical knowledge information.
  • the surgical knowledge information is general surgical knowledge information held by a surgeon, such as, for example, a surgical procedure or rules for surgery.
  • the work determination unit 213b may perform work determination based on, for example, either the characteristic information or the surgical information, or may perform work determination based on both the characteristic information and the surgical information.
  • the visual field evaluation unit 213c performs visual field evaluation based on the feature information acquired by the feature extraction unit 213a and the task information acquired by the task determination unit 213b, and obtains visual field evaluation information related to the visual field evaluation value.
  • the visual field evaluation is performed, for example, by machine learning. A learning model is used at that time.
  • the visual field evaluation unit 213c calculates a visual field evaluation value (e.g., a visual field evaluation result score) indicating the quality of the visual field based on the evaluation model obtained from the evaluation model DB 63.
  • the evaluation model DB 63 holds, for example, evaluation models.
  • the evaluation model is, for example, a learning model for evaluating the quality of the visual field.
  • the evaluation model is configured by a DNN (Deep Neural Network), and takes feature information and task information as input, and outputs a normalized visual field evaluation scalar value.
  • DNN Deep Neural Network
  • the visual field evaluation unit 213c may also determine the evaluation model to be used for visual field evaluation based on the work information and attribute information.
  • the visual field evaluation unit 213c may refer to the attribute information DB 62 and determine the evaluation model to be used from a plurality of evaluation models based on the attribute information obtained from the attribute information DB 62 and the work information acquired by the work judgment unit 213b.
  • the attribute information DB 62 holds, for example, attribute information related to attributes.
  • Attribute information includes, for example, information about a person's personality and the use of the field of view.
  • a person's personality is, for example, the proficiency of the surgeon. An experienced surgeon will want to keep the field of view fixed when withdrawing surgical tools, but a junior surgeon will want to take a wider field of view for a while, so the best field of view varies depending on the surgeon's level of proficiency.
  • the hospital to which the surgeon belongs is also included in the personality. For example, when treating the cystic artery, the task of whether to approach the surgical tools from above or below will differ depending on the hospital. Other personalities include the surgeon's preferences for surgical methods.
  • the target of use of the visual field can be, for example, whether the surgeon is a human or whether the system includes the surgeon as an autonomous system. If the surgeon is a human, it is necessary to increase the visual field evaluation value of the visual field that is easy for humans to see, but if the surgeon is also an autonomous system, i.e. a robot, it is necessary to increase the evaluation value of the visual field that is easy for the robot control recognizer to recognize. In this way, the visual field evaluation unit 213c switches the evaluation model to be used based on the task information and attribute information.
  • rule-based methods may be used, or machine learning-based methods and rule-based methods may be used in combination.
  • An example of mixing machine learning-based methods and rule-based methods is to use rules globally but machine learning locally.
  • this method uses a rule-based method to ensure that surgical tools fit within the image, but uses machine learning to evaluate how they should appear in the image.
  • a rule-based method uses machine learning to evaluate how they should appear in the image.
  • machine learning it is also possible to perform evaluation using one large learning model without switching learning models.
  • a visual field evaluation is performed directly based on feature information, task information, and attribute information.
  • the behavior planning unit 26 plans the behavior of the robot arm device 10 based on the visual field evaluation information related to the visual field evaluation value, and generates behavior plan information related to the behavior plan.
  • behavior planning means, for example, determining the target position and/or target posture of the imaging device 12.
  • the method using reinforcement learning considers the visual field evaluation value input from the visual field evaluation unit 213c, i.e., the visual field evaluation result score, as a reward, and plans actions to acquire a policy that maximizes the profit, which is the total value of this visual field evaluation result score to be received in the future.
  • a deep reinforcement learning approach may be taken by using deep learning to model the policy.
  • the behavior planning unit 26 refers to the behavior planning model DB 64, and performs learning and inference based on the behavior planning model obtained from the behavior planning model DB 64 to generate behavior planning information.
  • the behavior planning model DB 64 holds, for example, a behavior planning model.
  • the behavior planning model is, for example, a learning model for planning the behavior of the robot arm device 10.
  • the action planning unit 26 may use environmental information.
  • the action planning unit 26 generates the action plan information related to the action plan based on the action plan model obtained from the action plan model DB 64 and the environmental information obtained from the environmental information DB 65.
  • the environmental information DB 65 holds, for example, environmental information related to the environment.
  • the environmental information includes, for example, structural information about the three-dimensional structure inside the abdominal cavity.
  • the environmental information DB 65 There are three ways to create the environmental information DB 65 that holds this structural information. The first is a method in which the three-dimensional structure inside the abdominal cavity is created in advance by simulation, and is not updated at runtime. The second is a method in which the three-dimensional structural information is updated at runtime based on the three-dimensional structure inside the abdominal cavity created in advance by simulation. The third is a method in which the three-dimensional structure inside the abdominal cavity is created only at runtime, without any prior data.
  • a three-dimensional structure When creating a three-dimensional structure in advance by simulation, it may be created from data of an existing three-dimensional structure, or it may be created by acquiring distance information using a depth sensor such as a depth camera, or it may be created using a method such as SfM based on image information, or other methods may be used.
  • a method for creating a three-dimensional structure at runtime is to create a three-dimensional structure based on images acquired by the imaging device 12.
  • the imaging device 12 e.g., the endoscope 5001
  • the doctor first manually looks around the abdominal cavity with the imaging device 12, updating the overall structural information of the abdominal cavity at that time, and thereafter continuing to update based on the information captured by the imaging device 12 each time.
  • the visual field evaluation result score is regarded as a reward, and an approach is taken to maximize profits.
  • the output of the visual field evaluation unit 213c may be output separately for a number of different elements.
  • the action planning unit 26 determines an action to make the tip of the surgical tool the center of the image.
  • the action planning unit 26 determines an action to make the range to be viewed fall within the visual field. For example, if the range to be viewed is not within the visual field, this means that the distance to the object is too close, so it indicates that the action should be moved in a direction away from the object. In this way, when a rule-based method is used, the action to be taken is planned by designing in advance what action to take based on the value of one or more visual field evaluation result scores.
  • reinforcement learning is basically used to utilize the visual field evaluation result score, but when certain conditions are met, the rule-based method is forcibly prioritized for control.
  • the imaging device 12 interferes with the abdominal cavity or surgical tools, or when clearly abnormal movements are planned, the action plan based on reinforcement learning is ignored and danger is avoided using a rule-based method.
  • a method may be used in which either reinforcement learning or the rule-based method is prioritized when certain conditions are met, or a method may be used in which the results of the action plan obtained from each of the reinforcement learning and rule-based methods are used in a hybrid manner.
  • the arm control unit 23 controls the operation of the robot arm device 10 so that the imaging device 12 captures an image of the external environment based on the action plan information obtained from the action plan unit 26.
  • the external environment is the patient's abdominal cavity
  • the arm control unit 23 controls the position and/or posture of the imaging device 12 to obtain the desired field of view based on the action plan information regarding the action planned by the action plan unit 26.
  • the arm control unit 23 performs coordinate transformation processing to control the position and/or posture of the imaging device 12.
  • the image captured by the imaging device 12 is in an image coordinate system
  • the robot arm device 10 to be controlled is in a robot coordinate system, so in order to obtain an image of the desired field of view, the image coordinate system is transformed into the camera coordinate system (the coordinate system of the imaging device 12), the camera coordinate system is transformed into the tool (robot tip) coordinate system, and then the tool coordinate system is transformed into the robot coordinate system.
  • the imaging device 12 can be controlled to any position and/or posture to obtain the desired field of view.
  • the operation device 14 is an interface that allows the user to directly intervene during runtime with the medical observation system 1.
  • the medical observation system 1 basically performs autonomous control, but the visual field evaluation and action plan can be modified by the user's intervention.
  • the input unit 24 inputs visual field evaluation intervention information related to visual field evaluation intervention in response to the user's operation on the operation device 14.
  • the visual field evaluation intervention information is information for the user to intervene in the visual field evaluation.
  • the visual field evaluation intervention information includes any or all of correction information for correcting the evaluation result of whether the visual field is good or bad, instruction information for indicating the visual field that the user wants to see, instruction information for indicating work information, etc.
  • the visual field evaluation unit 213c executes a process of updating the evaluation model based on the visual field evaluation intervention information sent from the input unit 24.
  • the visual field evaluation unit 213c issues an instruction to the generation unit 214.
  • the generation unit 214 performs additional learning using the correction information or instruction information included in the visual field evaluation intervention information as corrective data, and updates the evaluation model stored in the evaluation model DB 63. In this way, the inference accuracy can be improved.
  • the user operates the operation device 14 to modify the behavior of the imaging device 12 (e.g., trajectory, position, and/or posture).
  • the input unit 24 inputs behavior plan intervention information related to the behavior plan intervention in response to the user's operation on the operation device 14.
  • the behavior plan intervention information is information for the user to intervene in the behavior plan.
  • the behavior plan intervention information is modification information for modifying the behavior of the imaging device 12.
  • the behavior planning unit 26 executes a process of updating the behavior plan model based on the behavior plan intervention information sent from the input unit 24.
  • the action planning unit 26 issues an instruction to the generation unit 214.
  • the generation unit 214 performs additional learning using the correction information included in the action plan intervention information as corrective data, and updates the action plan model stored in the action plan model DB 64. In this way, the inference accuracy can be improved.
  • an action plan is made to improve the visual field evaluation result score, but if there is user intervention, the generator 214 assumes that a more appropriate action plan exists and performs additional learning, updating the action plan model stored in the action plan model DB 64.
  • surgeon may directly operate the imaging device 12 manually without performing the autonomous control described above.
  • autonomous control may not be performed at the start of surgery, and the user (scopist) may first manually move the imaging device 12 to look around the abdominal cavity and check the intraperitoneal environment, or when emergency treatment is required, the autonomous control may be forcibly stopped and the imaging device 12 may be pulled out.
  • user intervention IF is basically intended to be used at runtime as described above, it may also be used as a tool for users to give instructions in natural language offline.
  • Fig. 11 is a flowchart showing the flow of an example of advance preparation processing of the information processing device 20 according to the present embodiment. This flowchart is an example of a workflow when offline.
  • step S11 the processing unit 21 of the information processing device 20 executes an initialization process.
  • the processing unit 21 is a CPU
  • the CPU loads a computer program stored in either the ROM or the external memory, expands it on the RAM, and makes it executable.
  • the CPU also reads the parameters of each device of the information processing device 20 and returns it to its initial position, making each device ready for use.
  • the generation unit 214 of the processing unit 21 creates the surgery information DB 61.
  • the surgery information includes information such as prior knowledge that a doctor has when performing surgery, such as the procedure and rules of the surgery.
  • the surgery information includes contents described in medical school textbooks and skill certification systems.
  • the generation unit 214 stores the surgery information in the surgery information DB 61 so that the work judgment unit 213b can interpret it.
  • step S13 the generation unit 214 of the processing unit 21 creates an attribute information DB 62.
  • the attribute information includes, for example, information such as a person's personality and the intended use of the visual field.
  • the attribute information includes information such as the proficiency of each doctor, the practices of the affiliated hospital, and preferences.
  • the generation unit 214 stores the attribute information in the attribute information DB 62 so that the visual field evaluation unit 213c can interpret it.
  • step S14 the generation unit 214 of the processing unit 21 learns the evaluation model and creates the evaluation model DB 63.
  • the generation unit 214 creates an evaluation model by learning an evaluation model for evaluating the quality of the visual field based on, for example, the endoscopic operation data of an expert (scopist), and stores the evaluation model in the evaluation model DB 63 so that the visual field evaluation unit 213c can use it.
  • step S15 the generation unit 214 of the processing unit 21 creates the behavior plan model DB 64.
  • the generation unit 214 creates a behavior plan model by acquiring a policy through reinforcement learning using a simulator, for example, and stores the behavior plan model in the behavior plan model DB 64 so that the behavior planning unit 26 can use it.
  • step S16 the generation unit 214 of the processing unit 21 creates the environmental information DB 65.
  • the generation unit 214 creates environmental information (e.g., intraperitoneal structure) by, for example, simulation, and stores the environmental information in the environmental information DB 65 so that the action planning unit 26 can interpret the environmental information.
  • environmental information e.g., intraperitoneal structure
  • Fig. 12 is a flowchart showing the flow of an example of a control process of the information processing device 20 according to the present embodiment. This flowchart is an example of a workflow at runtime.
  • step S21 the processing unit 21 of the information processing device 20 performs processing.
  • the same processing as in step S11 described above is performed.
  • the input unit 24 inputs the surgical information.
  • the surgical information is, for example, information that a cholecystectomy will be performed. Based on this information, surgical information such as the surgical procedure for the cholecystectomy stored in the surgical information DB 61 is referenced and used when performing the visual field evaluation.
  • the input of the surgical information may be, for example, a user operation using the operation device 14.
  • the operation may be, for example, a voice instruction, an operation using a GUI, or any other method.
  • the input unit 24 inputs attribute information.
  • the attribute information DB 62 is referenced and used to evaluate what type of field of view is good during visual field evaluation.
  • the attribute information is, for example, information that the user is an experienced doctor. For example, if the user is an experienced doctor, an evaluation criterion is set such that a good field of view is one in which the field of view at the time the surgical tool is removed is maintained as it is, rather than processing to pull back the endoscope 5001 (an example of the imaging device 12) to widen the field of view when inserting or removing the surgical tool.
  • the input of the attribute information may be, for example, a user operation using the operation device 14, as described above.
  • the operation may be, for example, a voice command, an operation using a GUI, or any other method.
  • step S24 the processing unit 21 waits for the surgeon to place a trocar in the patient's affected area and insert the endoscope 5001.
  • the system is in manual operation, and the procedure is performed in the same way as in normal endoscopic surgery.
  • the start and end of the insertion of the endoscope 5001 may be triggered, for example, by a user operation using the operation device 14.
  • the operation may be, for example, a voice command, an operation using a GUI, or any other method.
  • step S25 the processing unit 21 waits for the surgeon to use the endoscope 5001 to look around the patient's abdominal cavity. By performing this look around, the surgeon can understand the structure inside the abdominal cavity.
  • the start and end of the look around the abdominal cavity with the endoscope 5001 may be triggered by, for example, a user operation using the operation device 14.
  • the operation may be, for example, a voice command, an operation using a GUI, or any other method.
  • step S26 the generation unit 214 updates the intraperitoneal structure (an example of environmental information). That is, the generation unit 214 updates the environmental information DB 65.
  • the environmental information DB 65 is updated during a look-around operation, but it may be updated at another time, or may not be updated at all.
  • step S27 the processing unit 21 waits for the surgeon to place the endoscope 5001 in the initial position.
  • the trigger for starting and ending the setting of the endoscope 5001 in the initial position may be, for example, a user operation using the operation device 14.
  • the operation may be, for example, a voice instruction, an operation using a GUI, or any other method.
  • the surgeon When setting the initial position, the surgeon basically aligns the trocar position with the tip position of the endoscope 5001, and the position where they align becomes the initial position. This determines the positional relationship of the robot arm device 10 to the patient, and the robot arm device 10 recognizes the position of the trocar point. Subsequent operations of the robot arm device 10 are always performed while maintaining this trocar point position, allowing autonomous control to be performed without injuring the patient.
  • step S28 the robot arm device 10 starts autonomous control.
  • the trigger for starting autonomous control may be, for example, a user operation using the operation device 14.
  • the operation may be, for example, a voice command, an operation using a GUI, or any other method.
  • step S29 the endoscope 5001 captures images.
  • the endoscope 5001 basically captures images all the time, so it transmits image information to the processing unit 21 at arbitrary intervals.
  • the arbitrary intervals are determined depending on, for example, the processing speed of the visual field evaluation in the subsequent stage.
  • the transmission interval may be determined on the endoscope 5001 side, or the endoscope 5001 may constantly send image information and the information processing device 20 may receive it at the required timing, or both may be used.
  • step S30 the acquisition unit 211 of the processing unit 21 acquires image information.
  • the acquisition unit 211 receives the image captured by the imaging device 12.
  • the acquisition unit 211 transmits the acquired image to the feature extraction unit 213a.
  • step S31 the feature extraction unit 213a of the processing unit 21 performs dimensional compression and extracts features. Details of feature extraction are as described above.
  • step S32 the work determination unit 213b of the processing unit 21 determines the work.
  • the details of the work determination are as described above.
  • step S33 the processing unit 21 determines whether the surgery has ended. If the surgery has not ended based on the determined work result (Yes in step S33), the processing unit 21 continues processing. On the other hand, if the surgery has ended (No in step S33), the processing unit 21 ends processing.
  • step S34 the visual field evaluation unit 213c of the processing unit 21 performs visual field evaluation. Details of the visual field evaluation are as described above.
  • step S35 the action planning unit 26 of the processing unit 21 performs action planning.
  • the details of the action planning are as described above.
  • step S36 the arm control unit 23 controls the robot.
  • the details of the robot control are as described above.
  • the processing unit 21 returns the process to step S29, and repeats the processes from step S29 onwards.
  • the medical observation system 1 includes the imaging device 12 that images an object, and the evaluation unit 213 that quantitatively evaluates the quality of the visual field of the imaging device 12 based on image information obtained by the imaging device 12. This makes it possible to quantitatively evaluate the quality of the visual field of the imaging device 12, allowing the user to obtain the visual field that he or she desires to see.
  • the evaluation unit 213 may also obtain a visual field evaluation value that indicates the quality of the visual field of the imaging device 12. This allows a quantitative evaluation to be performed reliably.
  • the evaluation unit 213 may also have a feature extraction unit 213a that extracts features from image information, a task determination unit 213b that determines the task based on one or both of feature information related to the features and surgery information related to the surgery, and generates task information related to the task, and a visual field evaluation unit 213c that obtains a visual field evaluation value based on the feature information and task information. This allows for reliable quantitative evaluation.
  • the medical observation system 1 may further include a surgery information DB 61 that stores surgery information. This allows the surgery information to be used reliably.
  • the visual field evaluation unit 213c may also obtain a visual field evaluation value based on an evaluation model that is a learning model. This allows the visual field evaluation value to be obtained reliably.
  • the visual field evaluation unit 213c may also change the evaluation model used based on the feature information and attribute information related to the user. This makes it possible to obtain a visual field evaluation value more reliably.
  • the medical observation system 1 may further include an attribute information DB 62 that stores attribute information. This allows the attribute information to be used reliably.
  • the medical observation system 1 may further include an evaluation model DB 63 that stores the evaluation models. This allows the evaluation models to be used reliably.
  • the medical observation system 1 may further include a generation unit 214 that generates an evaluation model. This allows the evaluation model to be used reliably.
  • the generating unit 214 may also update the evaluation model based on visual field evaluation intervention information regarding visual field evaluation intervention by the user. This allows the evaluation model to be appropriately updated.
  • the medical observation system 1 may further include a robot arm device 10 that moves the imaging device 12, and a behavior planning unit 26 that plans behavior for the robot arm device 10 based on visual field evaluation value information related to the visual field evaluation value, and generates behavior plan information related to the behavior plan. This makes it possible to obtain the behavior plan information.
  • the action planning unit 26 may also perform action planning based on environmental information related to the environment. This makes it possible to reliably obtain action planning information.
  • the behavior planning unit 26 may also perform behavior planning based on a behavior planning model, which is a learning model. This makes it possible to reliably obtain behavior planning information.
  • the medical observation system 1 may further include an evaluation model database that holds the behavioral planning model. This allows the behavioral planning model to be used reliably.
  • the medical observation system 1 may further include a generation unit 214 that generates a behavioral plan model. This allows the behavioral plan model to be used reliably.
  • the generation unit 214 may also update the behavior plan model based on behavior plan intervention information regarding the user's intervention in the behavior plan. This allows the behavior plan model to be appropriately updated.
  • the medical observation system 1 may further include an arm control unit 23 that controls the robot arm device 10 based on the action plan information. This allows the robot arm device 10 to be controlled based on the action plan information.
  • the imaging device 12 may also be an endoscopic device (e.g., the endoscope 5001). This makes it possible to quantitatively evaluate the quality of the field of view of the imaging device 12 even when the imaging device 12 is an endoscopic device.
  • an endoscopic device e.g., the endoscope 5001.
  • each configuration and each process according to the above-mentioned embodiment may be implemented in various different forms (modified examples) other than the above-mentioned embodiment.
  • the configuration and process may be various forms without being limited to the above-mentioned example.
  • all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method.
  • the configurations, processing procedures, specific names, or information including various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.
  • the various information shown in each figure is not limited to the information shown in the figure.
  • each configuration and each process in the above-mentioned embodiment does not necessarily have to be physically configured as illustrated.
  • the specific form of distribution and integration of each device is not limited to that illustrated, and all or part of it can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.
  • a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.
  • a cloud computing configuration can be adopted in which one function is shared and processed jointly by multiple devices via a network.
  • each step described in the above-described process flow can be executed by one device, or can be shared and executed by multiple devices.
  • the multiple processes included in that one step can be executed by one device, or can be shared and executed by multiple devices.
  • the user such as the surgeon
  • the user is mainly described as a human, but the user does not necessarily have to be a human and can be a robot.
  • both the surgeon and the scopist are robots, and the surgery can be performed in part or in whole by a fully autonomous robot system.
  • the attribute information in the visual field evaluation is that of a robot rather than a human, and a visual field that is easy for the robot to recognize is evaluated as a good visual field.
  • learning the evaluation model learning is also performed according to this guideline.
  • the task may be a task such as cooking or cleaning a room, rather than surgery.
  • a common feature with these tasks is that it is difficult to quantitatively define a good visual field state.
  • an application example can be given in which the quality of the visual field for cooking or cleaning is quantitatively evaluated.
  • the quality of the visual field for the task of frying or plating the food can be quantitatively evaluated. In plating, it is also possible to evaluate whether the plating is in a good state.
  • an application example can be given in which the evaluation results are fed back to correct the plating to a better state.
  • the technology according to the present disclosure may be used in an application that improves the efficiency and success rate of task execution by evaluating the visual field state.
  • FIG. 13 is a diagram showing an example of the hardware configuration of the computer 1000.
  • the computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input/output interface 1600. Each part of the computer 1000 is connected by a bus 1050.
  • the CPU 1100 operates based on the programs stored in the ROM 1300 or the HDD 1400 and controls each component. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.
  • the ROM 1300 stores boot programs such as the Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000.
  • BIOS Basic Input Output System
  • HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records program data 1450.
  • the communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet).
  • the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.
  • the input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000.
  • the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input/output interface 1600.
  • the CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input/output interface 1600.
  • the input/output interface 1600 may also function as a media interface that reads programs and the like recorded on a specific recording medium.
  • Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase change rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, and semiconductor memories.
  • optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase change rewritable Disks)
  • magneto-optical recording media such as MOs (Magneto-Optical Disks)
  • tape media magnetic recording media
  • magnetic recording media and semiconductor memories.
  • the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of each part of the information processing device 20.
  • the information processing program and various data are stored in the HDD 1400.
  • the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from other devices via the external network 1550.
  • the present technology can also be configured as follows.
  • An imaging device that captures an image of an object; an evaluation unit that quantitatively evaluates the quality of the field of view of the imaging device based on image information obtained by the imaging device;
  • An information processing system comprising: (2) The evaluation unit obtains a visual field evaluation value indicating the quality of the visual field of the imaging device.
  • the information processing system according to (1) above.
  • the evaluation unit is a feature extraction unit that extracts features from the image information; a task determination unit that determines a task based on one or both of feature information related to the feature amount and surgery information related to the surgery, and generates task information related to the task; a visual field evaluation unit that obtains the visual field evaluation value based on the characteristic information and the task information; having The information processing system according to (2) above. (4) Further comprising a surgery information database for storing the surgery information. The information processing system according to (3) above. (5) The visual field evaluation unit obtains the visual field evaluation value based on an evaluation model which is a learning model. The information processing system according to (3) or (4). (6) The visual field evaluation unit changes the evaluation model to be used based on the feature information and attribute information related to the user.
  • a generating unit that generates the evaluation model is further provided.
  • the generation unit updates the evaluation model based on visual field evaluation intervention information regarding a visual field evaluation intervention by a user.
  • (11) a robot arm device that moves the imaging device; a behavior planning unit that performs a behavior plan for the robot arm device based on visual field evaluation value information regarding the visual field evaluation value, and generates behavior plan information regarding the behavior plan; Further comprising: The information processing system according to any one of (2) to (10). (12) The action planning unit performs the action plan based on environmental information related to the environment. The information processing system according to (11) above. (13) The action planning unit performs the action planning based on a action planning model which is a learning model. The information processing system according to (11) or (12). (14) Further comprising an evaluation model database for storing the behavioral planning model. The information processing system according to (13) above. (15) A generator for generating the behavioral planning model is further provided.
  • the generation unit updates the behavior plan model based on behavior plan intervention information regarding a user's intervention in the behavior plan.
  • (17) further comprising an arm control unit that controls the robot arm device based on the action plan information.
  • the imaging device is an endoscope device.

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Veterinary Medicine (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Physics & Mathematics (AREA)
  • Robotics (AREA)
  • Biophysics (AREA)
  • Optics & Photonics (AREA)
  • Pathology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Endoscopes (AREA)

Abstract

Un système de traitement d'informations selon un mode de réalisation de la présente divulgation comprend : un dispositif d'imagerie qui capture l'image d'un objet ; et une unité d'évaluation qui, sur la base d'informations d'image obtenues par le dispositif d'imagerie, évalue quantitativement le fait que le champ visuel du dispositif d'imagerie soit bon ou mauvais.
PCT/JP2024/010268 2023-03-23 2024-03-15 Système de traitement d'informations, dispositif de traitement d'informations et procédé de génération de modèle d'apprentissage Pending WO2024195729A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2023-046910 2023-03-23
JP2023046910 2023-03-23

Publications (1)

Publication Number Publication Date
WO2024195729A1 true WO2024195729A1 (fr) 2024-09-26

Family

ID=92841559

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/010268 Pending WO2024195729A1 (fr) 2023-03-23 2024-03-15 Système de traitement d'informations, dispositif de traitement d'informations et procédé de génération de modèle d'apprentissage

Country Status (1)

Country Link
WO (1) WO2024195729A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007152014A (ja) * 2005-12-08 2007-06-21 Olympus Medical Systems Corp 内視鏡用検具、及び内視鏡システム
WO2016111178A1 (fr) * 2015-01-05 2016-07-14 オリンパス株式会社 Système d'endoscope
WO2021006228A1 (fr) * 2019-07-10 2021-01-14 Sony Corporation Système d'observation médicale, dispositif de commande et procédé de commande

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007152014A (ja) * 2005-12-08 2007-06-21 Olympus Medical Systems Corp 内視鏡用検具、及び内視鏡システム
WO2016111178A1 (fr) * 2015-01-05 2016-07-14 オリンパス株式会社 Système d'endoscope
WO2021006228A1 (fr) * 2019-07-10 2021-01-14 Sony Corporation Système d'observation médicale, dispositif de commande et procédé de commande

Similar Documents

Publication Publication Date Title
JP7480477B2 (ja) 医療用観察システム、制御装置及び制御方法
JP7003985B2 (ja) 医療用支持アームシステムおよび制御装置
JP7722365B2 (ja) 手術中に画像取込装置を制御するためのシステム、方法、および、コンピュータプログラム製品
WO2020196338A1 (fr) Système de bras médical, dispositif de commande et procédé de commande
US20230017738A1 (en) Method, apparatus and system for controlling an image capture device during surgery
US20240282065A1 (en) Mixed reality systems and methods for indicating an extent of a field of view of an imaging device
US20250117073A1 (en) Systems and methods for facilitating optimization of an imaging device viewpoint during an operating session of a computer-assisted operation system
WO2020045015A1 (fr) Système médical, dispositif de traitement d'informations et méthode de traitement d'informations
CN113993478B (zh) 医疗工具控制系统、控制器和非暂时性计算机可读存储器
US20230172438A1 (en) Medical arm control system, medical arm control method, medical arm simulator, medical arm learning model, and associated programs
US20230142404A1 (en) Medical imaging apparatus, learning model generation method, and learning model generation program
JPWO2018221041A1 (ja) 医療用観察システム及び医療用観察装置
CN115916482A (zh) 信息处理装置、程序、学习模型以及学习模型生成方法
WO2022054498A1 (fr) Système de commande de bras médical, dispositif de bras médical, procédé de commande de bras médical, et programme
WO2021049220A1 (fr) Bras de support médical et système médical
WO2024195729A1 (fr) Système de traitement d'informations, dispositif de traitement d'informations et procédé de génération de modèle d'apprentissage
WO2018043205A1 (fr) Dispositif de traitement d'image médicale, procédé de traitement d'image médicale, et programme
WO2024162096A1 (fr) Système de commande et procédé de commande, et procédé de génération de modèle d'apprentissage
WO2021044900A1 (fr) Système opératoire, dispositif de traitement d'image, procédé de traitement d'image et programme

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24774865

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE