[go: up one dir, main page]

WO2021116846A1 - Système de commande pour un système endoscopique, système et procédé de commande d'un système endoscopique - Google Patents

Système de commande pour un système endoscopique, système et procédé de commande d'un système endoscopique Download PDF

Info

Publication number
WO2021116846A1
WO2021116846A1 PCT/IB2020/061440 IB2020061440W WO2021116846A1 WO 2021116846 A1 WO2021116846 A1 WO 2021116846A1 IB 2020061440 W IB2020061440 W IB 2020061440W WO 2021116846 A1 WO2021116846 A1 WO 2021116846A1
Authority
WO
WIPO (PCT)
Prior art keywords
control
images
processing unit
acquisition means
control signal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/IB2020/061440
Other languages
English (en)
Inventor
Marco Raoul Marini
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Advanced Vision And Inspection Systems Avais Srl
Original Assignee
Advanced Vision And Inspection Systems Avais Srl
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Advanced Vision And Inspection Systems Avais Srl filed Critical Advanced Vision And Inspection Systems Avais Srl
Publication of WO2021116846A1 publication Critical patent/WO2021116846A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00006—Operational features of endoscopes characterised by electronic signal processing of control signals
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00039—Operational features of endoscopes provided with input arrangements for the user
    • A61B1/00042—Operational features of endoscopes provided with input arrangements for the user for mechanical operation
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00045—Display arrangement
    • A61B1/00048—Constructional features of the display
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00147—Holding or positioning arrangements
    • A61B1/0016—Holding or positioning arrangements using motor drive units
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00163—Optical arrangements
    • A61B1/00188—Optical arrangements with focusing or zooming features
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00064—Constructional details of the endoscope body
    • A61B1/00066—Proximal part of endoscope body, e.g. handles
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods
    • A61B2017/00017—Electrical control of surgical instruments
    • A61B2017/00203—Electrical control of surgical instruments with speech control or speech recognition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods
    • A61B2017/00017—Electrical control of surgical instruments
    • A61B2017/00207—Electrical control of surgical instruments with hand gesture control or hand gesture recognition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods
    • A61B2017/00017—Electrical control of surgical instruments
    • A61B2017/00216—Electrical control of surgical instruments with eye tracking or head position tracking control
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods
    • A61B2017/00973—Surgical instruments, devices or methods pedal-operated
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2048—Tracking techniques using an accelerometer or inertia sensor
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2051—Electromagnetic tracking systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/371—Surgical systems with images on a monitor during operation with simultaneous use of two cameras
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/372—Details of monitor hardware
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
    • A61B2090/397—Markers, e.g. radio-opaque or breast lesions markers electromagnetic other than visible, e.g. microwave
    • A61B2090/3975—Markers, e.g. radio-opaque or breast lesions markers electromagnetic other than visible, e.g. microwave active
    • A61B2090/3979—Markers, e.g. radio-opaque or breast lesions markers electromagnetic other than visible, e.g. microwave active infrared
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0132—Head-up displays characterised by optical features comprising binocular systems
    • G02B2027/0134—Head-up displays characterised by optical features comprising binocular systems of stereoscopic type
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B23/00—Telescopes, e.g. binoculars; Periscopes; Instruments for viewing the inside of hollow bodies; Viewfinders; Optical aiming or sighting devices
    • G02B23/24—Instruments or systems for viewing the inside of hollow bodies, e.g. fibrescopes
    • G02B23/2476—Non-optical details, e.g. housings, mountings, supports
    • G02B23/2484—Arrangements in relation to a camera or imaging device

Definitions

  • This invention relates to a control system for an endoscopic device, a control method for an endoscopic system and an endoscopic system comprising the control system and an endoscopic device. More specifically the invention relates to an endoscopy control system that enables one or more control signals to be acquired without use of the hands.
  • both flexible and rigid or semi-rigid endoscopes are in common use.
  • the two types differ in mechanical properties, which do or do not allow the endoscope to adapt to some extent to the conformation of the part of the body under examination.
  • Flexible endoscopes are often characterised by openings that enable the operator to introduce other instruments and perform surgical procedures; while rigid endoscopes, although generally of high precision, are normally limited to image acquisition.
  • semi-rigid endoscopic devices are fitted with an articulated operating end (an operating segment at the head of the endoscope) which can perform movements in different directions when controlled by appropriate actuators connected to the other end of the endoscope.
  • the operating end is also equipped with image acquisition means such as a pair of cameras.
  • the present invention is particularly suitable for the control of endoscopic devices having an operational image acquisition end whose movement can be controlled in (about) at least three axes of rotation, conventionally defined as the X, Y and Z axes.
  • rotation about the X axis identifies a so- called “pan” or “pitch” movement
  • rotation about the Y axis identifies a so-called “tilt” or “yaw” movement
  • rotation about the Z axis identifies a so-called “roll” movement.
  • these three axes of movement are therefore generally called the pitch, yaw and roll axes respectively.
  • endoscopy devices are the semi-rigid Endoeye Flex 3D endoscope from Olympus.
  • this endoscope the operator has to operate movements of the flexible segment at the head of the endoscope using appropriate mechanisms at the opposite end while looking at a monitor. This means that the operator deciding how to direct the endoscope inside the patient will always have at least one hand engaged.
  • Viewers that reproduce a virtual reality in response to movements of the person wearing the viewer are also known.
  • Tracking systems that capture movements of the viewer and accordingly modify the images that can be seen inside it are also known. The effect of this technology is to immerse the individual in a three-dimensional reality that he or she can only see when wearing the viewer.
  • Italian Patent No. 102015000055827 describes an endoscopy device including a wearable viewer, a processor and an endoscope.
  • the wearable viewer is equipped with a movement tracking system and is connected to a processor that receives position information in relation to a given reference or horizon.
  • the tracking system is based on the transmission/reception of a three-axis spatial location system (accelerometer, gyroscope and magnetometer).
  • the viewer transfers position information to a processor via cable or wireless communication and the processor transforms the position information into a motion control signal at the operating end of the endoscope, resulting in hands-free control.
  • the viewer is also equipped with a limb recognition system, also known as Leap Motion, which can be used to switch the display of the images reproduced by the viewer between images captured by the endoscope and those captured by cameras located on the outer surface of the viewer, thus enabling the operator to switch between an endoscopic image display and a display of the surrounding reahty without removing the viewer.
  • a limb recognition system also known as Leap Motion
  • control system should make it possible to operate an endoscope with improved safety.
  • control system should be versatile, easy and economical to implement, install and use, and/or be adaptable to the different types of endoscopic devices that are commercially available.
  • Such a control system for an endoscopy device therefore has first hands-free means for acquiring a control signal on the basis of which movement of the operating segment of the endoscope is controlled on the basis of tracking the movements of the operator's head, and at least second hands-free means for acquiring at least one further control signal that does not require the use of hands, and are capable of acquiring one or more of: a control signal to control blocking or enabling of control of movement of the operating endoscope based on the tracking of head movements, and/or a signal for switching the display of images on the viewer between images from the environmental image acquisition means and images from the endoscopy device, and/or a zoom control signal for an image acquired from the acquisition means of the endoscopy device or an environmental image.
  • This solution therefore makes it possible to release the hands from control of one or more important functions, resulting in improved safety and reliability, as well as versatility for use in surgical applications.
  • the second hands-free acquisition means may include at least one or more pedals and/or a voice command acquisition system.
  • pedals is particularly advantageous because it allows the surgical operator's voice to be freed from control of the endoscope and image display, but use in combination with a voice acquisition system may be advantageous for maximum versatility, for example when switching between images from the endoscopic device and environmental images.
  • At least one or preferably at least two pedals with graduated actuation to acquire a corresponding control signal which is proportional to a pressure force exerted on the pedal.
  • These pedals may advantageously be used to adjust the amount of zoom for images from the endoscope acquisition means and/or environmental image acquisition means.
  • One or more pedals can also be provided for the acquisition of a switching signal and, preferably, an operable part of each switching pedal is shaped with an external surface that differs in shape from that of one or more other control signal acquisition pedals. The operator can therefore easily recognise the different pedals by means of his own foot, so safety is further increased.
  • the switching pedals may preferably acquire a switching signal for the display of images on the viewer between images from the environmental image acquisition means and images from the endoscopy device and/or a control signal to command the disabling or enabling of control of movement based on the tracking of head movements.
  • the processing unit (110) is configured to automatically control disabling or enabling (s-lk) movement control of the operating end of the endoscopy device based on tracking the movements of the operator's head when it receives and acts on a signal for the display of images on the viewer switching from/to the images from the endoscopy device to/from the images from the environmental image acquisition means.
  • This solution results in maximum system safety and reliability for surgical use, in particular if used in combination with pedals for graduated control of the amount of zoom, as it allows the surgical operator to change from images of the operating site (possibly at maximum magnification) acquired from the endoscope to environmental images (or vice versa), with maximum safety, reliability and a single control signal, the operator thus being able to act promptly and in the most suitable surgical manner if, for example, in the course of surgery it is necessary to change the manner of operation or the viewing point for the site being operated on.
  • the processing unit is preferably configured to exclude a control signal coming from one of the second hands-free means for acquiring a control signal when another of said second hands-free acquisition means is acquiring a corresponding control signal and/or until an action controlled by a corresponding control signal acquired from another of said second hands-free acquisition means has been completed.
  • a further object of this invention is an endoscopy system according to claim 13, a method for controlling an endoscopy system according to claim 14 and a computer program according to claim 15.
  • Figure 1 shows a block diagram of an endoscopy system according to the present invention
  • Figure 2a is a front perspective view of a wearable viewer as part of the control system according to the present invention.
  • Figure 2b is a side view of the wearable viewer in Figure 2a;
  • Figure 3 Figure 2a is a front perspective view of one example of a pedal unit with pedals for the hands-free acquisition of control signals, forming part of a control system according to the present invention
  • Figure 4 is a view in vertical cross-section of one of the pedals in the pedal unit of the endoscopy system in Figure 3 during use;
  • Figure 5 is a block diagram of an example of control logic that implements a method of control according to the present invention.
  • an endoscopy system comprises an endoscopy device 10 and a control system 100.
  • Endoscopy device 10 comprises an endoscope 11 with a head operating segment that carries means 11a for acquiring images Ill.
  • Endoscope 11 may be a semi-rigid endoscope of the type currently known to those skilled in the art, which is substantially straight, of length varying from 30 to 50 cm, and a cross-section of variable shape averaging 10 mm.
  • the operating segment at the top of endoscope 11 is flexible, equipped with means 11a for the acquisition of images, preferably of the stereoscopic acquisition type, for example comprising at least one camera, preferably two cameras, positioned at the tip.
  • one or more illumination devices for example of the LED type, may be placed on the operating segment to illuminate the area framed by the acquisition means.
  • Movement of the operating segment can be controlled in at least three axes by means of a drive unit 12 of endoscope 11, connected to control system 100 and capable of operating the operating segment on the basis of drive signals sll generated from drive control signals slO generated by control system 100 and received as input from drive unit 12.
  • a stream of images si la acquired by endoscope acquisition means 11a is sent to the control system via an appropriate connection, e.g. via drive unit 12.
  • Control system 100 to control an endoscope device includes an HMD viewer wearable by an operator O, a processing unit 110, first means 120 for acquisition of a control signal sl20 that does not require the use of hands (hereinafter also “hands free"), in particular including a system hm for tracking movements of the operator's head, and further hands-free means 130, 140 for the acquisition of at least one further control signal that do not require the use of hands.
  • hands hereinafter also "hands free”
  • the movement tracking system is designed to detect movements hm of an operator's head with respect to a given reference or horizon.
  • the tracking system is based on the acquisition of movement information by means of sensor means 120 that operate a spatial location system with at least three axes.
  • Sensor means 120 preferably include position sensor devices such as at least one accelerometer, one gyroscope and one magnetometer, preferably configured to detect, in combination, movements in at least three and preferably at least nine axes.
  • the sensor means may also include detection enhancement systems such as infrared (IR)-based locators or other technology having a similar purpose.
  • IR infrared
  • a combination of movement components detected by sensor means 120 also called “fusion” in the multimodal environment, allows the tracking system to obtain extremely precise identification of the position and movements of the operator's head in a three-dimensional space. This has a number of advantages, including perfect recognition of angles and inclinations in every direction.
  • Position/movement signals sl20 acquired by tracking system 120 are a preferred example of a control signal acquired from the first hands free means for the acquisition of a control signal.
  • processing unit 110 can process a motion control signal slO (position and/or orientation) of the endoscope head operating segment 11.
  • the motion control signal slO can be sent to endoscope drive unit 12, for example comprising an electronic control panel connected to the processing unit and actuating means to operate the operating segment of the endoscope on the basis of drive signals generated by the control panel from the motion control signals slO transmitted from processing unit 110 and received as an input from drive unit 12.
  • the result is a movement control system for the operating segment of the endoscope that does not require the use of hands.
  • the actuators may for example be three or more servomotors, at least one of which is positioned and configured to move the operating segment of endoscope 11 about the three pan, tilt and roll axes respectively, X, Y, Z.
  • motion control signals slO from processing unit 110, these preferably include at least: a signal controlling an angle of rotation about the X axis, a signal controlling an angle of rotation about the Y axis, and a signal controlling an angle of rotation about the Z axis.
  • motion control signal slO may also include a zoom control signal, especially if the zoom is of the optical type.
  • motion control signals slO for each axis X, Y, Z may be obtained as follows: the data sl20 for the spatial coordinates of the position of the head of operator O collected by sensor means 120 are transformed into digital coordinates in a three-dimensional plane; based on a predetermined resting position (e.g. coordinates relating to the frontal position, eyes parallel to the horizon line), the tracking system detects a movement of the operator's head and records it according to the axis in which it has been detected: horizontal (Y), vertical (X) and rotational (Z).
  • a predetermined resting position e.g. coordinates relating to the frontal position, eyes parallel to the horizon line
  • each axis X, Y, Z is digitally calculated and sent as a signal to control the movement of operating segment 10 about the respective X, Y or Z axis.
  • This operation may require a preliminary stage of calibration so that, on the basis of many factors, such as actuator sensitivity and/or endoscope model size, digital values are associated with angular rotation positions for each actuator.
  • a zero shift is equal to 0°
  • a positive shift is equal to + ⁇ °
  • a negative shift (counter clockwise) is equal to - ⁇ ° for each actuator in the system.
  • Transformation of signal slO by control panel 12 and/or the processing of signal slO by processing unit 110 can advantageously apply a proportionahty factor between the detected amplitudes of movement of the operator's head and those desired for the operating segment of the endoscope.
  • the HMD wearable viewer may be of any known type to display stereoscopic images, being in particular at least of high definition, to simulate a 3D reality on one or more display screens.
  • it may be an Oculus Rift CV1, already equipped with Oculus VR tracking sensors that operate a head movement tracking system, such as an HTC Vive or other devices of the same category.
  • the connection to the processing unit which may for example be a computer comprising at least one processor, can be made via cable 5 or wireless or by any known means.
  • the HMD viewer is preferably configured to display 3D images in Side-by-Side (SBS) format, i.e. side-by-side and synchronised according to a specific format.
  • SBS Side-by-Side
  • the image stream smhd displayed on the HMD viewer is processed and transmitted to it by processing unit 110.
  • environmental image acquisition means are located on a front surface of the HMD viewer to acquire images of the reality surrounding the HMD viewer and send a corresponding stream of environmental images sl40 to processing unit 110.
  • Cameras 141,142 are preferably placed side by side in the horizontal plane and preferably capture high definition RGB images.
  • the distance between the objectives will preferably be proportional to the human interpupillary distance according to the microscopy (or macroscopy) desired; as known to persons skilled in the art, the greater the focal distance, the greater will be the distance between the two devices.
  • the images captured by cameras 141, 142 on the HMD viewer will therefore be able to correspond exactly to the operator's POV (Point of View) and, therefore, displaying them in SBS format, provide a realistic view of the surrounding environment, as if the viewer were not there.
  • POV Point of View
  • Video stream sl40 captured by environmental image capture means 140 may be sent to processing unit 110 for processing, for example via the same connection as the HMD viewer or via an independent connection.
  • processing unit 110 can control the flow of images smhd displayed by the HMD viewer to switch the display shown on the HMD viewer from images si la from the endoscope to images sl40 of the operator's surroundings from environmental image acquisition means 140, enabling the operator to view the surrounding environment without necessarily having to remove the HMD viewer.
  • image acquisition means 140 and the processing unit may be configured to act as a hand detection device, making it possible to acquire a control signal to command such switching by means of a hand gesture captured by acquisition means 140 and detected by a hand recognition algorithm operated by the processing unit.
  • the processing unit calls up a hand recognition function which, after recognising a predefined gesture, commands a switch and starts sending RAW images from the two cameras 141, 142 directly to the HMD screen.
  • a control system 100 is characterised in that it includes at least second hands-free means 150, 130 for acquisition of a control signal, which do not require the use of hands, are connected to the processing unit and are capable of acquiring and transmitting a corresponding further control signal to the processing unit itself.
  • the at least second means of acquiring a control signal are in fact configured to acquire at least one further control signal in hands-free mode, including one or more of: a control signal to control exclusion of the control signal for movement of the operating end processed on the basis of the tracking of head movements; and/or a signal switching the display of images on the viewer between the images coming from the means for the acquisition of environmental images and images coming from the endoscopy device (or vice versa); and/or an amount of zoom control signal applied to the image acquired by the acquisition means of the endoscopy device.
  • the second means for hands-free acquisition of a control signal thus make it possible to increase the safety, reliability and/or usability of the endoscopy system, in particular to eliminate or at least reduce the need to use hands and/or unwanted display switching and/or provide additional functionality that can be controlled without the use of hands.
  • a first example of second hands-free acquisition means comprises a voice command acquisition system, comprising at least one sound acquisition device, for example a microphone 130, and at least one logical voice command processing unit, configured to process the audio signal sl30 acquired by sound acquisition device 130 and provide a corresponding control signal to processing unit 110 to which the voice command acquisition system is connected.
  • a voice command acquisition system comprising at least one sound acquisition device, for example a microphone 130, and at least one logical voice command processing unit, configured to process the audio signal sl30 acquired by sound acquisition device 130 and provide a corresponding control signal to processing unit 110 to which the voice command acquisition system is connected.
  • the logical voice command processing unit may be provided within processing unit 110 itself, for example through appropriate programming of the same, so that it processes audio signal sl30 coming from acquisition devices 130 to recognise the speech contained in it, transcribe it into an appropriate file (for example using known Speech-to-text applications) and identify one or more predefined words or phrases (Keywords) in the transcribed text that form corresponding control signals entered by the operator by voice command.
  • the logical unit for processing the audio signal input from acquisition devices 130 may also be an intermediate unit connected between devices 130 and processing unit 110, for example implemented in an audio acquisition card for pre-processing audio signal sl30 emitted by acquisition devices 130. A combination of the two solutions is also possible.
  • a further preferred embodiment of second means for hands-free acquisition of a control signal comprises one or more pedals 151, 152, 153, 154 that can be operated by an operator's foot and are connected to processing unit 110 to which the respective acquired control signal is transmitted.
  • a pedal is any device configured to be operated by an operator's foot.
  • pedal unit 150 which can be connected to the processing unit, for example, by means of a cable 158 of appropriate length; pedal unit 150 may be placed permanently on a floor in a desired position, providing a convenient operating interface by means of foot pedals 151, 152, 153, which can be easily moved as a single unit and placed in the most convenient position, for example, by means of cable 158 or other appropriate wireless connection.
  • the acquisition and transmission of signals from the pedal unit to the processing unit can preferably be coordinated by a controller inside the pedal unit, connected to processing unit 110.
  • second hands-free acquisition means 150 may include at least one, preferably at least two graduated-action pedals 151, 152 capable of acquiring a respective control signal proportional to a pressure force exerted on pedals 151, 152.
  • each graduated pedal 151 may include a pedal lever 151a capable of being actuated by the pressure exerted by an operator's foot against the force of resilient means 151b (e.g. one or more thrust springs).
  • pedal lever 151a Operation of pedal lever 151a may for example be rotation around a respective hinge fulcrum 151b in one direction or another within a certain predefined angle of rotation a.
  • Each pedal 151 may include graduated sensor means 15 Id arranged and configured to detect how much pressure is applied to pedal lever 151a during operation, and output a corresponding proportional control signal sl51that can be transmitted to processing unit 110.
  • proportional control signals sl51, sl52 by means of one or more graduated pedals 151, 152 makes it possible to control with great precision drives or processes that can be adjusted between two ends of a control scale in a linear or proportional way, and is therefore particularly advantageous for some preferred examples of graduated control that will be described below, such as in particular a zoom factor control with speed/magnitude proportional to the pressure exerted on one of said graduated pedals 151, 152.
  • second hands-free acquisition means 150 may include at least one, preferably at least two switching pedals 153, 154, that is individual pedals (or push-buttons) of the, for example, switch or ON/OFF push-button type.
  • Each switching pedal 153, 154 may in particular be operated to acquire a control signal for switching between two or more predefined states of a given actuator or signal controlled by processing unit 110.
  • Switching pedals of this type are particularly advantageous when switching signal acquisition that does not require fine tuning but a fast and reliable response is required.
  • a control signal can be sent to processing unit 110 quickly, conveniently and rehably without using the hands and without having to look at the pedal it is desired to operate.
  • This can be implemented in an even more practical way by conforming an operable part of each pedal 153, 154 with an external surface of a particular shape, preferably different from that of one or more of the other pedals, for example hemispherical like the one shown in Figure 3, so that the operator can recognise which control signal it is configured to acquire, by touch or at least with slight pressure on pedal 153, 154.
  • processing unit 110 is configured to process a control signal slO to move operating segment 11 of endoscopy device 10, control signal slO for movement being processed by the processing unit on the basis of a control signal sl20 transmitted by the first hands free acquisition means, in particular by the system tracking movement of the head of operator O, so as to control a proportional actuator drive (e.g. motors) for operating segment 11 about each of at least three axes (X-Y-Z) of movement by means of a control unit 12 for endoscopy device 10.
  • a proportional actuator drive e.g. motors
  • Processing unit 110 is also configured to control the display of an image stream smhd on the HMD viewer, and in particular to switch the source of this image stream smhd between images from environmental image acquisition means 140 and images si la from acquisition means 11a of endoscopy device 10.
  • the processing unit is also connected to hands-free second means 130, 140 for acquiring an additional control signal that does not require the use of hands and which in the preferred embodiment illustrated include two switching pedals 153, 154, two graduated pedals 151, 152 and a voice command acquisition system 130.
  • a first switching pedal 153 is used to acquire and send a control signal sl53 to unit 110 signal to control switching of the source of said image stream smhd between the images coming from environmental image acquisition means 140 and images si la coming from acquisition means 11a of endoscopy device 10 (and/or vice versa).
  • a similar control signal for switching said flow of displayed images smhd can be acquired by voice command acquisition system 130 by saying a corresponding predetermined word or phrase into the respective microphone.
  • Pressing pedal 153 or the saying of the predetermined word or phrase by operator O will therefore cause processing unit 110 to switch image stream smhd displayed on the HMD viewer without the need to use the hands.
  • First graduated pedal 151 and/or second pedal 152 may be able to acquire a proportional/graduated hands-free control signal. According to a preferred embodiment, one or both graduated pedals 151, 152 will be able to acquire a control signal for an amount of zoom applied to images from the acquisition means of endoscope 11a or possibly from environmental image acquisition means 140.
  • a proportional signal sl51 acquired by first graduated pedal 151 and sent to processing unit 110 may be able to control an amount of close or magnification zoom (Zoom-in) applied to images captured by endoscopy device 10.
  • a proportional signal sl52 acquired by second graduated pedal 152 may instead control an amount of zoom-out applied to images captured by endoscopy device 10.
  • control of the amount of zoom applied via processing unit 110 may be for example: a digital zoom, which will be controlled by processing unit 110 by means of appropriate processing of flow si la of images Ill captured by means 11a of endoscopy device 10, or a zoom of an optical type, which can be controlled by processing unit 110 by sending a suitable signal to control panel 12 of endoscopy device 10, which will include suitable actuators able to vary optical acquisition arrangement of image acquisition means 11a proportionally so that flow of images si la received by processing unit 110 will already be suitably zoomed by a certain amount.
  • a combination of the two zoom methods is also possible, since the amount of digital zoom is limited by the definition of the original flow of images si la captured by means 11a of the endoscope and the quantity of optical zoom is limited by the physical structure of the endoscope itself.
  • a similar control signal for the amount of zoom may be acquired by voice command acquisition system 130 by saying a corresponding predetermined word or phrase into the respective microphone.
  • each predetermined word or phrase e.g. "zoom in” or “zoom out”
  • processor 110 is preferably configured to control blocker or enabler s-lk ( Figure 5) for control of movement of the operating end of endoscopy device 10 based on tracking head movement signal sl20 from tracking system 120. That is, the processing unit can block or disable the processing and/or sending of signal slO for control of movement of the operating end of the endoscope, processed to control movement of the operating segment based on the detected and tracked movements of the operator's head, when this is not desired or appropriate, and enable it again when it is desired or appropriate.
  • the processing unit can be configured to block or disable said control of movement of the operating end of endoscopy device 10 based on tracking movements of the operator's head: automatically when it receives and implements a control signal for switching the source of image stream smhd displayed by the viewer from images si la from acquisition means 11a of endoscopy device 10 to images from environmental image acquisition means 140; and/or when it receives a control signal from second hands-free acquisition means that commands the blocking (disabling) of control of movement of the operating end based on the tracking of head movements.
  • processing unit 110 may be configured to (re-)enable control of movement of the operating end of endoscopy device 10 by tracking the operator's head movements: automatically when it receives and acts on a control signal for switching the source of image stream smhd displayed by the viewer from images from environmental image acquisition means 140 to images si la from acquisition means 11a of endoscopy device 10, and or when it receives a control signal from second hands-free acquisition means that commands the (re-)enabling of control of movement of the operating end based on the tracking of head movements.
  • a second switching pedal 154 may be used to acquire said control signal sl54 and send it to unit 110 to control blocking/enabling control of movement of the operating end based on the tracking of head movements (and/or vice versa).
  • a similar control signal for enabling/disabling control of movement of the operating end based on the tracking of head movements may be acquired by voice command acquisition system 130 through saying a predetermined word or phrase into the respective microphone.
  • Pressure on pedal 154 or saying of the predetermined word or phrase by operator O will therefore bring about the manual enabling or disabling of control of movement of the operating end based on the tracking of head movements, without the need to use the hands.
  • processor 110 is configured so that when in a start-up state and/or default state the control system according to the present invention is in a mode in which control of movement of the operating end based on the tracking of head movement is blocked (disabled).
  • processing unit 110 may also be configured to block/enable a signal controlling the amount of zoom when it commands the corresponding enabling/disabhng of control of movement of the operating end based on the tracking of head movements.
  • a particularly preferred embodiment of the control system provides for processing unit 110 to be configured to exclude a control signal s-esc from one of second hands-free means 150, 130 for acquisition of a control signal, when another of said second hands-free acquisition means 130, 150 is acquiring a corresponding control signal and/or until an action controlled by a corresponding control signal acquired by the other second hands free acquisition means has been completed.
  • processing unit 110 is receiving or has just received a signal sl51, sl52 controlling an amount of zoom from one of graduated pedals 151, 152, it will command the exclusion of any signals controlling the amount of zoom ("Zoom-in” or "Zoom out") spoken by the operator and acquired by voice command acquisition system 130 (for example by ignoring them), until the action of changing the amount of zoom controlled by the respective signal acquired via pedal 151, 152 has been completed.
  • a surgical operator connects the control unit to an endoscopy device 10 comprising a mechanical and semi-rigid endoscope with three or more degrees of freedom and switches the control system on, places footswitch 150 next to the operating table, prepares the instruments and places them inside the patient in accordance with practice.
  • an endoscopy device 10 comprising a mechanical and semi-rigid endoscope with three or more degrees of freedom and switches the control system on, places footswitch 150 next to the operating table, prepares the instruments and places them inside the patient in accordance with practice.
  • Operating switch button 154 with one foot, it sends a signal sl54 to the processing unit to order the enabling of movement control based on the tracking of head movements; at this point the operator starts to explore the environment inside the patient's body by moving the head and, therefore, by directing the operating segment of the endoscope in proportion to movements of the head.
  • a predetermined voice command into microphone 130 e.g. "EXIT”
  • the operator causes control signal sl30 for switching the display by voice command acquisition system 130 to be acquired and sent to the processing unit which controls switching of the source of image flow smhd sent for display to the HMD viewer from images 11a sourced from the endoscope to images sl40 coming from cameras 141, 142 on the front of the HMD viewer; the operator can see the surrounding world and change the instrument.
  • the system automatically blocks movement command based on the tracking of head movements, or the operator orders the block using button 154.
  • the operator decides to use pedal 153 (and possibly 154) because he is talking to one of his assistants at the same time and therefore cannot use voice command acquisition system 130 again; he therefore proceeds to press pedal 153 which sends the respective control signal sl53 to processing unit 110, in response to which unit 110 commands switching to images from the endoscope.
  • control signal si 54 that controls the blocking of movement control based on head movements, extracts the endoscope from the patient's body, takes off the HMD viewer, continues with the final routine tasks and finally switches off the control system.
  • a further object of the present invention is a computer program including instructions that implement a method of control according to the present invention when performed by a computer processing unit, such as the processor of a personal computer.
  • a control system according to the present invention for the control of an endoscopy device offers improved reliability, safety and usability in that it is able to implement the acquisition of control signals for all the functions required without using the hands, and also in different ways depending on the specific needs that may arise during surgery.
  • the system and method of control also make it possible to control auxiliary functions that require proportional control and could not be implemented through control based solely on the tracking of head movements, without the need for the use of hands.
  • the control system may also be configured to further improve safety of use through means for disabling/enabhng movement control based on head movements that may be either automatic or manually operated, again without requiring the use of the hands.
  • the system is also easy and economical to implement as it can be implemented using a personal computer on which runs a computer program that implements the method of control according to the invention that is connected to a limited number of peripheral devices (viewer, head movement tracking system and at least a second means of acquiring hands-free control signals, in particular at least one pedal or voice command acquisition system) and is connected through standard connections to any endoscopy device designed to receive movement control signals in at least three axes at right angles.
  • peripheral devices viewer, head movement tracking system and at least a second means of acquiring hands-free control signals, in particular at least one pedal or voice command acquisition system

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Pathology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Veterinary Medicine (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Signal Processing (AREA)
  • Mechanical Engineering (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Endoscopes (AREA)

Abstract

L'invention concerne un système de commande pour commander un dispositif d'endoscopie (10), comprenant un segment de fonctionnement qui porte des moyens d'acquisition d'image (11a) et dont le mouvement peut être commandé selon au moins trois axes à angle droit. Le système de commande comprend : une unité de traitement (110) apte à être connectée au dispositif d'endoscopie pour le commander ; une visionneuse (HMD) qui peut être portée par un opérateur (O), reliée à l'unité de traitement (110) ; un premier moyen d'acquisition mains libres d'un signal de commande, comprenant un système (120) pour suivre les mouvements de la tête de l'opérateur (O) connecté à l'unité de traitement (110) ; un moyen (140) pour l'acquisition d'images environnementales, agencé pour acquérir des images de la réalité entourant la visionneuse (HMD) et relié à l'unité de traitement. L'unité de traitement (110) est configurée pour traiter un signal (s10) pour commander le mouvement du segment de fonctionnement du dispositif d'endoscopie (10) sur la base du signal de commande (s120) transmis par le premier moyen d'acquisition mains libres (120) de manière à commander le mouvement du segment de fonctionnement sur la base du suivi des mouvements de la tête de l'opérateur (O), et pour commander l'affichage d'un flux (smhd) d'images sur la visionneuse (HMD) et pour commuter cet affichage sur la visionneuse entre des images environnementales provenant du moyen d'acquisition d'images environnementales (140) et des images (I11) provenant du dispositif d'endoscopie. Le système comprend au moins moyen d'acquisition mains libres (150, 130) d'un signal de commande pour acquérir un signal de commande qui ne nécessite pas l'utilisation de mains, connecté à l'unité de traitement (110) et capable d'acquérir et de transmettre à l'unité de traitement au moins un autre signal de commande (s151, s152, s153, s152 ; s130) qui comprend un ou plusieurs des éléments suivants : un signal de commande (s154) pour commander le blocage ou l'activation de la commande du mouvement de l'extrémité de fonctionnement sur la base du suivi des mouvements de la tête ; et/ou un signal (s153) pour commuter l'affichage d'images sur la visionneuse entre les images provenant du moyen d'acquisition d'images environnementales et des images provenant du dispositif d'endoscopie (10) ; et/ou un signal de commande du zoom pour une image acquise par le moyen d'acquisition du dispositif d'endoscopie ou une image environnementale.
PCT/IB2020/061440 2019-12-12 2020-12-03 Système de commande pour un système endoscopique, système et procédé de commande d'un système endoscopique Ceased WO2021116846A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
IT102019000023826 2019-12-12
IT102019000023826A IT201900023826A1 (it) 2019-12-12 2019-12-12 Sistema di controllo per un dispositivo endoscopico, relativi metodo di controllo e dispositivo

Publications (1)

Publication Number Publication Date
WO2021116846A1 true WO2021116846A1 (fr) 2021-06-17

Family

ID=70155016

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2020/061440 Ceased WO2021116846A1 (fr) 2019-12-12 2020-12-03 Système de commande pour un système endoscopique, système et procédé de commande d'un système endoscopique

Country Status (2)

Country Link
IT (1) IT201900023826A1 (fr)
WO (1) WO2021116846A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050033580A1 (en) * 1994-09-22 2005-02-10 Computer Motion, Inc. Speech interface for an automated endoscope system
JP3668269B2 (ja) * 1995-01-10 2005-07-06 オリンパス株式会社 体腔内手術用マニピュレータ装置
US20140296633A1 (en) * 2008-02-07 2014-10-02 The Trustees Of Columbia University In The City Of New York Remote endoscope handle manipulation
EP2979605A1 (fr) * 2013-03-29 2016-02-03 Tokyo Institute of Technology Système de fonctionnement endoscopique et programme de fonctionnement endoscopique
US20190000578A1 (en) * 2017-06-29 2019-01-03 Verb Surgical Inc. Emulation of robotic arms and control thereof in a virtual reality environment

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050033580A1 (en) * 1994-09-22 2005-02-10 Computer Motion, Inc. Speech interface for an automated endoscope system
JP3668269B2 (ja) * 1995-01-10 2005-07-06 オリンパス株式会社 体腔内手術用マニピュレータ装置
US20140296633A1 (en) * 2008-02-07 2014-10-02 The Trustees Of Columbia University In The City Of New York Remote endoscope handle manipulation
EP2979605A1 (fr) * 2013-03-29 2016-02-03 Tokyo Institute of Technology Système de fonctionnement endoscopique et programme de fonctionnement endoscopique
US20190000578A1 (en) * 2017-06-29 2019-01-03 Verb Surgical Inc. Emulation of robotic arms and control thereof in a virtual reality environment

Also Published As

Publication number Publication date
IT201900023826A1 (it) 2021-06-12

Similar Documents

Publication Publication Date Title
US12322164B2 (en) UI for head mounted display system
US12138129B2 (en) Imaging system for surgical robot, and surgical robot
JP2022036255A (ja) ロボット外科手術装置および視聴者適合型の立体視ディスプレイの態様を制御するためのシステム、方法、およびコンピュータ可読記憶媒体
EP2903551B1 (fr) Système numérique pour la capture et l'affichage d'une vidéo chirurgicale
CN113786152B (zh) 一种内窥镜镜头跟踪方法及内窥镜系统
US11278369B2 (en) Control device, control method, and surgical system
US11109916B2 (en) Personalized hand-eye coordinated digital stereo microscopic systems and methods
CN110235047A (zh) 用于增强现实眼科手术显微镜投影的系统和方法
US20170325907A1 (en) Spectacle-style display device for medical use, information processing device, and information processing method
AU2003234910A1 (en) Medical cockpit system
EP3531953A1 (fr) Système d'orientation spatiale à usage médical
WO2018096987A1 (fr) Appareil et procédé de traitement d'informations et programme
JP2015149552A (ja) ウェアラブル型電子機器
WO2017061293A1 (fr) Système d'opération chirurgicale, dispositif de commande d'opération chirurgicale, et procédé de commande d'opération chirurgicale
US11517395B2 (en) Control device, control method, and surgical system
JP2018107603A (ja) センサ情報取得装置、センサ情報取得方法、センサ情報取得プログラム及び医療器具
JP7367041B2 (ja) 頭部装着型ディスプレイシステムのためのｕｉ
JP7414590B2 (ja) 制御装置および医療用観察システム
WO2021116846A1 (fr) Système de commande pour un système endoscopique, système et procédé de commande d'un système endoscopique
CA3117533A1 (fr) Iu destinee a un systeme de visiocasque
US20220142722A1 (en) Method and system for controlling dental machines
RU2785887C1 (ru) Система визуализации для хирургического робота и хирургический робот
WO2023064954A1 (fr) Dispositif de commande mains libres pour microscope chirurgical

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20829351

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 12/09/2022)

122 Ep: pct application non-entry in european phase

Ref document number: 20829351

Country of ref document: EP

Kind code of ref document: A1