KR20230003653A - Teleoperated system with patient health records based instrument control - Google Patents

Abstract

A method for intraoperative use of a surgical patient health record in a teleoperated surgical system comprising surgical instruments and instrument actuators is provided, the method comprising: receiving user input commands to control movement of a robotic surgical instrument; tracking a robotic surgical instrument actuator state in response to user input commands; and transitioning the robotic surgical instrument actuator state to the safe mode in response to the robotic surgical instrument transitioning to the defined actuator state.

Description

Teleoperated surgical system with instrument control based on patient health records {TELEOPERATED SYSTEM WITH PATIENT HEALTH RECORDS BASED INSTRUMENT CONTROL}

priority claim

This application claims the benefit of priority to U.S. Patent Application Serial No. 62/421,064, filed on November 11, 2016, which is hereby incorporated by reference in its entirety.

1. Field of Invention

Aspects of the invention relate to medical devices used during surgery. More specifically, aspects relate to surgical planning tools implemented on a medical device in communication with a video library database.

2. Prior Art.

Surgeons typically undertake extensive study prior to performing a surgical procedure. Traditionally, surgeons have been limited to the study of general anatomical models, such as photographs or drawings. More recently, various preoperative diagnostic procedures (eg, x-ray, CT, MRI, etc.) have made available patient specific anatomical information.

In some cases, it is desirable to make additional anatomical and surgical procedure information available to the surgeon. In one aspect, it is desirable to provide a surgical site video recording of a previous surgical procedure performed on a particular patient to a surgeon planning surgery on a particular patient. In another aspect, it is desirable to provide a surgeon with one or more surgical video recordings of surgical procedures on other patients that are similar to surgical procedures planned for a particular patient. In one aspect, it is desirable to provide such information to a surgeon prior to the surgeon undertaking a particular surgical procedure. And in another aspect, it may be desirable to provide such information to the surgeon intraoperatively.

In one aspect, it is desirable to construct a video database containing intraoperative surgical site video recordings of various procedures undergone by various patients. In one aspect, it is desirable to configure a medical device capable of video recording to further include an input that allows a surgeon using the medical device to highlight and annotate the video recording in real time as the video recording is recorded. . In one aspect, it is desirable to configure a computer-based pattern matching algorithm to search individual records in a video database, identify relevant video records, and provide the surgeon with this relevant information for a particular surgical procedure.

The following summary introduces certain aspects of the inventive subject matter in order to provide a basic understanding. This summary is not a comprehensive overview of the subject matter and is not intended to identify key or critical elements or to delineate the scope of the subject matter. Although this summary contains information relating to various aspects and embodiments of the subject matter, its sole purpose is to present some aspects and embodiments in a general form as a prelude to the more detailed description that follows.

A method for intra-surgical use of surgical patient health records in a teleoperated surgical system that includes surgical instruments and surgical instrument actuators is provided. User input commands are received from a user to control movement of the robotic surgical instrument. Actuator state of a surgical instrument actuator state of the robotic instrument during movement of the robotic instrument in response to user input commands is tracked. The surgical instrument actuator state of the robotic instrument transitions to the safe mode in response to the robotic surgical instrument transitioning to the defined actuator state.

1 is a plan view of a minimally invasive teleoperated surgical system.
2 is a perspective view of a surgeon's console;
3 is a perspective view of an electronics cart;
4 is a schematic illustration of a teleoperated surgical system.
5 is a perspective view of a patient side cart;
6 is an elevational view of a surgical instrument.
7 is a perspective view of an instrument manipulator.
8 is a schematic illustration of a surgical planning tool.
9 is a flow diagram of a method of using the surgical planning tool.
10 is an exemplary diagram illustrating a storage atlas information structure in a computer readable storage device according to some embodiments.
11 is an exemplary diagram illustrating an example of a seventh information structure included in an atlas in a storage device that associates recorded video information from an individual surgery with corresponding surgical instrument actuator state information, in accordance with some embodiments. .
12A-12C are exemplary diagrams illustrating an exemplary surgical instrument and an exemplary actuator assembly in which the surgical instrument is shown in three different exemplary operating states in accordance with some embodiments.
FIG. 13 illustrates an eighth exemplary information structure of an atlas stored in a computer readable storage device corresponding to a specific surgical procedure and associating different exemplary surgical patient health record characteristics with surgical risk management information in accordance with some embodiments. it is a figurative drawing.
14 is an example flow diagram illustrating configuration of a processor for presenting an alert message and transitioning a surgical instrument to a safe state based at least in part on a surgical patient's health record information, in accordance with some embodiments.

This description and accompanying drawings, which illustrate inventive aspects, embodiments, implementations, or applications, are not to be taken as limiting - the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the scope of this description and claims. In some instances, well-known circuits, structures, and techniques have not been shown or described in detail in order not to obscure the invention. Like numbers in two or more of the drawings indicate the same or similar elements.

Elements described in detail with reference to one embodiment, implementation, or application, whenever practicable, may be included in other embodiments, implementations, or applications where they are not specifically shown or described. there is. For example, if an element is described in detail with reference to one embodiment and not with reference to a second embodiment, the element may nevertheless be claimed as being included in the second embodiment. Accordingly, in order to avoid unnecessary repetition in the following description, one or more elements shown and described in connection with an embodiment, implementation, or application, unless specifically described to the contrary, is one or more elements in an embodiment, implementation, or application. may be incorporated into other embodiments, implementations, or aspects as long as they do not render the implementation non-functional or two or more of the elements provide conflicting functions.

Aspects of the present invention are primarily aspects of embodiments using the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, Calif. (specifically, the Model IS4000 marketed as the da Vinci® Xi™ HD™ Surgical System). explained in However, those skilled in the art will understand that the inventive aspects disclosed herein may be embodied and implemented in a variety of ways, including robotic and, where applicable, non-robotic embodiments and implementations. . Embodiments for da Vinci® Surgical Systems (e.g., Model IS4000 da Vinci® Xi™ Surgical System, Model IS3000 da Vinci Si® Surgical System) are illustrative only and are to be considered limiting of the scope of the inventive aspects disclosed herein. It shouldn't be.

According to various aspects, the present disclosure describes a surgical planning tool that includes a medical device configured to video record the performance of surgical procedures. Video recordings may be embedded with various metadata, eg, highlights made by medical personnel. Additionally, video recordings may be tagged with various metadata, such as text annotations describing a particular topic of the video, the identity of the patient corresponding to the video recording, biographical or medical information about the patient, and the like. can In one aspect, tagged metadata is embedded in video recordings.

According to additional aspects, the present disclosure describes a teleoperated medical device that includes surgical instruments used in one or more stages of a surgical procedure. Different stages of a surgical procedure may be associated with different risk levels for different patients. In some embodiments, the level of risk to the patient may be determined manually by a medical practitioner or automatically through an expert system or artificial intelligence, based at least in part on the patient's health history. A surgical instrument is controlled by one or more surgical instrument actuators operable in a number of actuator states. During surgical procedures the actuator state of the actuator controlling the surgical instrument is tracked. In some embodiments, the information structure in the computer-readable storage device may include surgical guidance information and surgical instrument actuators for presentation to the surgeon in response to surgical instruments transitioning to a condition in which the patient is potentially at increased risk during a surgical procedure. relate the state In some embodiments, the surgical guidance information presented to the surgeon is determined based at least in part on the surgical patient's health record. In some embodiments, an information structure in a computer readable storage device is used by a surgical instrument actuator to cause the patient to transition into an actuator safe mode of operation that is less risky and therefore safer, at least with respect to some aspects of a surgical procedure. Associates safety transition information to operate and surgical instrument actuator states. In some embodiments, the surgical instrument actuator safe operating mode is determined based at least in part on the surgical patient's health record.

Video recordings and information structures associating surgical instrument actuator states with surgical guidance or actuator safety mode information can be archived locally or on an electronic medical record database implemented on a cloud data storage service. Video recordings may be made available to interested health care providers. The information structures may be made available for use with the teleoperated medical device to provide surgical guidance and to control surgical instrument actuator state during performance of surgical procedures.

Health care providers can use the metadata tags described above to search a medical device database based on patient health care records for videos of interest and information structure relationships. Additionally, in one aspect, the surgical planning tool includes a computer-based pattern matching and analysis algorithm. In one aspect, a pattern matching algorithm singles out videos stored on an electronic medical record database to identify correlations between visual characteristics in the video recordings and associated metadata tags made by medical practitioners. A surgical planning tool can apply these correlations to newly encountered anatomy, allowing the practitioner performing the procedure to make decisions about patient anatomy, preferred surgical approaches, disease states, potential complications, etc. can help you get down. In another aspect, the pattern matching algorithm identifies correlations between visual characteristics in video recordings and patient health record information to identify anatomical characteristics correlated with health record information in an electronic medical record database (private or public or both). c) Select the videos stored on the image. The surgical planning tool can apply these correlations between anatomy and health care records to the current patient's anatomy and health records, thereby helping medical personnel plan and perform surgical procedures involving the current patient. can give

Minimally invasive teleoperated surgical system

Referring now to the drawings in which like reference numbers indicate like parts throughout the several views, FIG. 1 is typically used to perform a minimally invasive diagnostic or surgical procedure on a patient 12 resting on an operating table 14. It is a plan view of the minimally invasive remotely operated surgical system 10 to be. The system includes a surgeon's console 16 for use by the surgeon 18 during a procedure. One or more assistants 20 may also participate in this procedure. The minimally invasive teleoperated surgical system 10 further includes a patient side cart 22 and an electronics cart 24 . While the surgeon 18 views the surgical site through the surgeon's console 16, the patient side cart 22 passes through a minimally invasive incision in the body of the patient 12 and at least one releasably coupled surgical instrument 26 ) can be manipulated. Images of the surgical site may be acquired by endoscope 28 , such as a stereoscopic endoscope, which may be manipulated by patient side cart 22 to orient endoscope 28 . Computer processors located on electronics cart 24 may be used to process images of the surgical site for subsequent display to surgeon 18 via surgeon console 16 . The number of surgical instruments 26 used at one time will generally depend on, among other factors, the diagnostic or surgical procedure and space constraints within the operating room. If it is necessary to change one or more of the surgical instruments 26 used during the procedure, the assistant 20 removes the surgical instruments 26 from the patient side cart 22 and removes them from the tray 30 in the operating room for another procedure. Mechanism 26 can be substituted.

2 is a perspective view of the surgeon's console 16 . The surgeon's console 16 includes a left eye display 32 and a right eye display 34 for presenting to the surgeon 18 a coordinated stereoscopic view of the surgical site enabling depth perception. Console 16 further comprises one or more control inputs 36 . In response to operation by surgeon 18 of one or more control inputs 36, one or more surgical instruments installed for use on patient side cart 22 (shown in FIG. 1) are moved. To provide the surgeon 18 with a telepresence, or perception that the control inputs 36 are integral with the instruments 26 so that the surgeon has a strong sense of direct control of the instruments 26. The surgical control inputs 36 may provide the same degrees of mechanical freedom as their associated surgical instruments 26 (shown in FIG. 1 ). To this end, position, force and tactile feedback sensors (not shown) may be used to transfer position, force and tactile sensations from surgical instruments 26 via control inputs 36 back to the surgeon's hand. can

The surgeon's console 16 is usually located in the same room as the patient so that the surgeon can directly monitor the procedure, be physically present if necessary, and speak to the patient's assistant directly rather than through a telephone or other communication medium. However, the surgeon may be located in a different room, in a completely different building, or other remote location from the patient enabling remote surgical procedures.

3 is a perspective view of the electronics cart 24. The electronics cart 24 may be coupled with the endoscope 28 and capture captured images for subsequent display, such as to the surgeon at the surgeon's console, or on other suitable displays located locally and/or remotely. It includes a computer processor for processing images. For example, if a stereoscopic endoscope is used, a computer processor on electronics cart 24 may process the captured images to present coordinated stereo images of the surgical site to the surgeon. Such coordination may include alignment between opposing images, and may include adjusting the stereo working distance of a stereoscopic endoscope. As another example, image processing may include use of previously determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations. Optionally, the equipment in the electronics cart may be integrated into the surgeon's console or patient side cart, or may be distributed in various other locations within the operating room.

4 schematically illustrates a telemanipulated surgical system 50 (such as the minimally invasive telemanipulated surgical system 10 of FIG. 1 ). A surgeon console 52 (such as surgeon console 16 of FIG. 1 ) may be used by a surgeon to control a patient side cart 54 (such as patient side cart 22 of FIG. 1 ) during a minimally invasive procedure. there is. Patient side cart 54 is used to capture images of the surgical site and to output the captured images to a computer processor located on electronics cart 56 (such as electronics cart 24 in FIG. 1), the stereoscopic endoscope. An imaging device may be used, such as Computer processors typically include one or more data processing boards for executing computer readable code stored in non-volatile memory devices of the computer processor. In one aspect, a computer processor may process captured images in a variety of ways prior to any subsequent display. For example, the computer processor may overlay the captured images with a virtual control interface prior to displaying the combined images to the surgeon via the surgeon's console 52.

Additionally or alternatively, captured images may be subjected to image processing by a computer processor located external to electronics cart 56 . In one aspect, the teleoperated surgical system 50 includes an optional computer processor 58 (as indicated by the broken line) similar to the computer processor located on the electronics cart 56, and the patient side cart 54 outputs the captured images to computer processor 58 for image processing prior to display on surgeon's console 52. In another aspect, captured images first undergo image processing by a computer processor on electronics cart 56 and then undergo additional image processing by computer processor 58 before being displayed on surgeon's console 52 . The teleoperated surgical system 50 may include an optional display 60, as indicated by dashed lines. The display 60 is coupled with a computer processor located on the electronics cart 56 and with a computer processor 58, and the captured images processed by these computer processors are displayed on the display of the surgeon's console 52. In addition to being displayed on the display 60 may be displayed.

5 is a perspective view of a patient side cart 500 of a minimally invasive teleoperated surgical system according to embodiments of the present invention. The patient side cart 500 includes one or more support assemblies 510 . A surgical instrument manipulator 512 is mounted to the end of each support assembly 510 . Additionally, each support assembly 510 optionally includes one or more unpowered, lockable setup joints used to position the attached surgical instrument manipulator 512 relative to the patient for surgery. can do. As depicted, the patient side cart 500 rests on the floor. In other embodiments, the operative parts of the patient side cart may be mounted to a wall, ceiling, operating table 526 that also supports the patient's body 522, or other operating room equipment. Additionally, although the patient side cart 500 is shown as including four surgical instrument manipulators 512, more or fewer surgical instrument manipulators 512 may be used.

A functionally minimally invasive telemanipulated surgical system will generally include a vision system portion that allows a user of the telemanipulated surgical system to view the surgical site from outside the patient's body 522 . A vision system typically includes a camera mechanism 528 for capturing video images and one or more video displays for displaying the captured video images. In some surgical system configurations, camera instrument 528 includes optics that transmit images from the distal end of camera instrument 528 to one or more imaging sensors (eg, CCD or CMOS sensors) external to patient body 522. . Alternatively, the imaging sensor(s) can be placed at the distal end of the camera mechanism 528, and the signals generated by the sensor(s) are processed and a lead is obtained for display on one or more video displays. It can be transmitted wirelessly or wirelessly. One example of a video display is a stereoscopic display on a surgeon's console in surgical systems commercialized by Intuitive Surgical, Inc. of Sunnyvale, Calif.

Referring to FIG. 5 , a surgical instrument 520 operating at a surgical site in a patient's body 522 is mounted on each surgical instrument manipulator 512 . Each surgical instrument manipulator 512 may be of various configurations that allow movement of the associated surgical instrument in one or more mechanical degrees of freedom (eg, all six Cartesian degrees of freedom, no more than five Cartesian degrees of freedom, etc.) can be provided as Typically, mechanical or control constraints constrain each manipulator 512 to move its associated surgical instrument about a center of motion on the instrument that remains stationary relative to the patient; This center of motion is typically located where the instrument enters the body.

In one aspect, surgical instruments 520 are controlled via computer-assisted telemanipulation. A functionally minimally invasive telemanipulated surgical system includes control inputs that receive inputs from a user of the telemanipulated surgical system (eg, a surgeon or other medical personnel). The control inputs communicate with one or more computer-controlled teleoperated actuators, such as one or more motors coupled to surgical instrument 520 . In this way, the surgical instrument 520 moves in response to the medical personnel moving the control input. In one aspect, one or more control inputs are included in a surgeon's console, such as surgeon's console 16 shown in FIG. 2 . The surgeon can operate the control inputs 36 of the surgeon's console 16 to operate the teleoperated actuators of the patient side cart 500 . The forces generated by the teleoperated actuators are transmitted through drivetrain mechanisms that transmit forces from the teleoperated actuators to surgical instrument 520 .

Referring to FIG. 5 , in one aspect, surgical instrument 520 and cannula 524 are detachably coupled to manipulator 512 and surgical instrument 520 is inserted through cannula 524 . One or more teleoperated actuators of manipulator 512 move surgical instrument 512 as a whole. Manipulator 512 further includes an instrument carriage 530 . Surgical instruments 520 are removably connected to instrument carriage 530 . In one aspect, instrument carriage 530 houses therein one or more teleoperated actuators that provide multiple controller movements, and surgical instrument 520 converts these controller movements into various movements of an end effector on instrument 520. convert to Accordingly, teleoperated actuators within instrument carriage 530 move only one or more components of surgical instrument 520 rather than moving the instrument as a whole. Inputs to control the instrument as a whole or to control components of the instrument are such that the input provided by the surgeon or other medical personnel to the control input ("master" command) is translated into a corresponding action ("slave" response) by the surgical instrument. has been

In an alternative embodiment, instrument carriage 530 does not house teleoperated actuators. The teleoperated actuators that enable the various movements of the end effector of the surgical instrument 520 are housed at a location remote from the instrument carriage 530, eg elsewhere on the patient side cart 500. A cable-based force transfer mechanism or the like is used to transfer the motions of each of the remotely positioned teleoperated actuators to a corresponding instrument-interfacing actuator output located on instrument carriage 530 . In some embodiments, surgical instrument 520 is mechanically coupled to a first actuator that controls a first motion of the surgical instrument, such as longitudinal (z-axis) rotation. Surgical instrument 520 is mechanically coupled to a second actuator that controls a second motion of the surgical instrument, such as a two-dimensional (x, y) motion. Surgical instrument 520 is mechanically coupled to a third actuator that controls a third movement of the surgical instrument, such as opening and closing a jaws end effector.

6 is a side view of a surgical instrument 520 that includes a proximal control mechanism 640 and a distal portion 650 coupled by an elongate tube 610 having an elongate tube centerline axis 611 . Surgical instrument 520 is configured to be inserted into the body of a patient and used to perform surgical or diagnostic procedures. Distal portion 650 of surgical instrument 520 may be forceps as shown, needle drivers, cautery devices, cutting tools, imaging devices (eg endoscopes or ultrasound probes), or the like. Any of a variety of end effectors 654, such as The surgical end effector 654 may include mechanical degrees of freedom in function, such as a jaw that opens or closes, or a knife that translates along a path. In the illustrated embodiment, the end effector 654 is coupled to the elongate tube 610 by a wrist 652 that allows the end effector to be oriented about the elongate tube centerline axis 611 . Surgical instrument 520 may also include stored information (eg, on a semiconductor memory within the instrument), which may be permanent or updatable by a surgical system configured to operate surgical instrument 520 . Accordingly, the surgical system may provide either one-way or two-way communication of information between the surgical instrument 520 and one or more components of the surgical system.

7 is a perspective view of a surgical instrument manipulator 512 . An instrument manipulator 512 is shown with no surgical instruments installed. Instrument manipulator 512 includes an instrument carriage 530 that can be removably connected to a surgical instrument (eg, surgical instrument 520). Instrument carriage 530 houses a plurality of teleoperated actuators. Each teleoperated actuator includes an actuator output 705. When surgical instruments are installed on instrument manipulator 512 , one or more instrument inputs (not shown) of an instrument proximal control mechanism (eg, proximal control mechanism 640 in FIG. 6 ) generate corresponding actuator outputs 705 ) and mechanically coupled. In one aspect, this mechanical coupling is direct, and actuator outputs 705 directly contact corresponding instrument inputs. In another aspect, this mechanical coupling occurs through an intermediate interface, such as a component of a drape configured to provide a sterile barrier between the instrument manipulator 512 and the associated surgical instruments. .

In one aspect, movement of one or more instrument inputs by corresponding teleoperated actuators results in movement of a surgical instrument mechanical degree of freedom. For example, in one aspect, the surgical instrument installed on instrument manipulator 512 is surgical instrument 520 , shown in FIG. 6 . Referring to FIG. 6 , in one aspect, movement of one or more instrument inputs of proximal control mechanism 640 by corresponding teleoperated actuators causes elongate tube 610 (and attached wrist 652) and end effector ( 654)) about the elongate tube centerline axis 611 relative to the proximal control mechanism 640. In another aspect, movement of one or more instrument inputs by corresponding teleoperated actuators results in movement of wrist 652 to orient end effector 654 about elongate tube centerline axis 611 . In another aspect, movement of one or more instrument inputs by corresponding teleoperated actuators results in movement of one or more movable elements of end effector 654 (eg, jaw member, knife member, etc.). Accordingly, the various mechanical degrees of freedom of the surgical instruments installed on the instrument manipulator 512 can be moved by the operation of the teleoperated actuators of the instrument carriage 530 .

Adding Comments to Recorded Videos

8 shows a schematic diagram of an exemplary surgical planning tool 800 . In one aspect, the surgical planning tool 800 includes a teleoperated surgical system 850 in data communication with an electronic medical device record database 830 . The telemanipulated surgical system 850 shown here is similar to the telemanipulated surgical system 850 shown in FIG. 4 . In one aspect, electronic medical records database 830 includes medical records of patients who have been treated at a particular hospital. The database 830 may be implemented on a server located on-site in the hospital. Medical record entries contained in database 830 can be accessed from hospital computers via an intranet network. Alternatively, database 830 may be implemented on a remote server located off-site using, for example, one of a number of cloud data storage services. In this case, the medical record entries of database 830 are stored on a cloud server and can be accessed by any computer having Internet access.

In one aspect, a surgical procedure is performed on a first patient using the teleoperated surgical system 850 . An imaging device associated with teleoperated surgical system 850 captures images of the surgical site and displays the captured images as frames of video on the display of surgeon console 52 . In one aspect, the medical personnel at the surgeon's console 52 use the input device of the surgeon's console 52 to highlight or annotate specific patient anatomy seen in the displayed video. One example of such an input device is the control input 36 shown in FIG. 2, which is coupled to a cursor that operates in conjunction with a graphical user interface overlaid on a displayed video. A graphical user interface includes a QWERTY keyboard, a pointing device such as a mouse and other means for entering data or text, including repurposing any of the existing manipulators as an input device, an interactive screen display, a touch screen display, or the like. can do. Accordingly, the medical personnel may highlight a particular tissue of interest in the displayed image or input text annotations.

In one aspect, the surgical site video is additionally displayed on a display located on the electronics cart 56. In one aspect, the display on the electronics cart is a touch screen user interface usable by the medical practitioner to highlight and annotate specific parts of the patient's anatomy shown on the image displayed for viewing on the display on the electronics cart. A user may highlight parts of the displayed image by touching parts of the patient anatomy displayed on the touch screen user interface. Additionally, a graphical interface including a QWERTY keyboard can be overlaid on the displayed image. Users can use the QWERTY keyboard to enter text annotations.

In one aspect, surgical site video captured by an imaging device associated with teleoperated surgical system 850 is recorded by teleoperated surgical system 850 and displayed to a user in real-time or near real-time, in addition to being displayed in a database. stored on 830. Highlights and/or annotations associated with the recorded video made by the user may also be stored on database 830 . In one aspect, highlights made by the user are embedded in the recorded video before the recorded video is stored on database 830 . Later, the recorded video can be retrieved for viewing. In one aspect, a viewer of recorded video can choose whether highlights are displayed or suppressed from view. Similarly, annotations associated with recorded video may also be stored on database 830 . In one aspect, annotations made by the user are used to tag the recorded video and can be used to provide a means of identifying a subject included in the recorded video. For example, an annotation may describe the conditions of a particular disease state. This annotation is used to tag the recorded video. Later, anyone wishing to view recorded procedures related to these disease states can use keyword search to find the video.

Retrieval of saved videos

In some cases, it is desirable for a medical practitioner to be able to view video recordings of past surgical procedures performed on a given patient. In one aspect, a patient who has previously undergone a first surgical procedure to treat a medical condition is treated to treat a recurrence of the same medical condition or to treat an anatomy located near the surgical site of the first surgical procedure. To do so, a second surgical procedure is subsequently required. In one aspect, surgical site events of a first surgical procedure were captured in a surgical site video recording, and the video recording was stored in database 830 as part of the patient's electronic medical records. Prior to performing a second surgical procedure on a patient, a medical practitioner may perform a search of database 830 to find video recordings of the patient's previous surgical procedure.

In some cases, it is desirable to plan to perform a surgical procedure on a patient so that a medical practitioner can view video recordings of similar surgical procedures performed on people with certain characteristics similar to the patient. In one aspect, surgical site video recordings of surgical procedures are tagged with metadata information such as the patient's age, gender, body mass index, genetic information, type of procedure the patient underwent, etc. before each video recording is stored in the database 830. It can be. In one aspect, the metadata information used to tag the video recording is automatically retrieved from the patient's contemporaneous medical records and then used to tag the video recording before it is archived in database 830. Accordingly, prior to performing a medical procedure on a patient, a medical practitioner may search database 830 for video recordings of similar procedures performed on patients that share certain characteristics in common with the patient. For example, if a medical practitioner is planning to use the teleoperated surgical system 850 to perform a prostatectomy on a 65-year-old male patient with an elevated body mass index, the medical practitioner may have a similarly elevated body mass index. The database 830 may be searched for surgical site video recordings of prostatectomies performed using the teleoperated surgical system 850 on other men of a similar age.

In one aspect, a video recording of the surgical procedure is transferred by database 830 to an optional personal computer 820 (as indicated by the dashed line) and is made available for viewing by medical personnel planning to perform the surgical procedure. . Additionally or alternatively, video recordings of previous surgical procedures are communicated by database 830 to teleoperated surgical system 850 and made available for pre- or intra-operative viewing. In one aspect, the video recording is displayed by the teleoperated surgical system 850 on a display located on the surgeon's console 52 . In another aspect, a video recording of the first surgical procedure is displayed on a display located on the electronics cart 56 .

Cloud-based video database

In one aspect, database 830 is implemented on a remote server using a cloud data storage service and accessible by multiple health care providers. Referring to FIG. 8 , as shown by the dashed lines, the surgical planning tool 800 includes a teleoperated surgical system 850 (as indicated by the dashed lines) and a personal computer 840 (as indicated by the dashed lines). ) is optionally included. In one aspect, except that the teleoperated surgical system 50 and personal computer 820 are located at a first health care provider and the teleoperated surgical system 850 and personal computer 840 are located at a second health care provider. In other words, teleoperated surgical system 850 is similar to teleoperated surgical system 50 and personal computer 840 is similar to personal computer 820 . In one aspect, a first patient is in need of surgical treatment of a medical condition and undergoes a surgical procedure using teleoperated surgical system 50 at a first health care provider. Video recordings of surgical procedures are stored on database 830 . Later, a second patient requires surgical treatment for the same medical condition and plans to receive surgical treatment using teleoperated surgical system 850 at a second health care provider. Prior to performing the surgical procedure on the second patient, the medical personnel accesses the database 830 via a secure internet connection and searches the database 830 for surgical site video recordings of similar procedures. In one aspect, a health care provider treating a second patient can retrieve a video recording of a first patient's surgical procedure from database 830 without acquiring knowledge of the first patient's identity. In this way, the privacy of the first patient is maintained. In one aspect, the video recording of the first patient's surgical procedure includes highlights and/or annotations made by a medical practitioner who treated the first patient.

Computer-based pattern matching and analysis

The surgical planning tool 800 may include a pattern matching and analysis algorithm implemented in the form of computer executable code. In one aspect, the pattern matching and analysis algorithm is stored on the non-volatile memory device of surgical planning tool 800 and is configured to analyze video recordings stored in database 830 . As previously discussed, each of the video recordings archived in database 830 may be tagged and/or embedded with specific metadata information. This metadata information may include patient information such as patient age, gender, and other information describing the patient's health or medical history. Additionally, as previously discussed, metadata information may include highlights or annotations made by a medical practitioner. In one aspect, these highlights and annotations are embedded in the video recording and stored in database 830 along with the video. The metadata may also include either an objective or subjective grading of proficiency in surgical practice, so that such pattern matching algorithms can be designed to select the best matches representative of the highest level of proficiency in the surgical practice.

In one aspect, the pattern matching and analysis algorithm includes an image analysis component that identifies patterns of shapes and colors shared among multiple video recordings stored on database 830 . A pattern matching and analysis algorithm then examines the tagged metadata associated with this subset of video recordings to determine if any words or phrases are frequently associated with videos within this subset of video recordings. These analyzes, performed by pattern matching and analysis algorithms, will be used to help health care providers make decisions about patient anatomy, preferred surgical approaches, co-morbidities, disease states, potential complications, and the like. can

How to use the Surgical Planning Tool

9 shows a method 900 of using a surgical planning tool. In one aspect, the surgical planning tool is similar to the surgical planning tool 800 in FIG. 8 . At 910, facts or characteristics describing the medical patient, such as a medical condition suffered by the patient, are received by the medical device. The medical device may be configured via a user interface located on the telemanipulated surgical system (e.g., telemanipulated surgical system 10 in FIG. 1 or telemanipulated surgical system 50 in FIG. 4) or, alternatively, in FIG. This fact or situation may be received through a personal computer similar to the personal computer 820 of . At 920, the medical device uses the facts or characteristics received at 910 to retrieve at least one relevant video recording of the surgical procedure from the medical device database. At 930, the medical device uses the video recordings to determine surgical planning information. In one aspect, the surgical planning information includes the types of instruments used in the recorded procedure. At 940 , the medical device displays the surgical plan information determined at 930 to the user.

How to use patient health records to guide surgical procedures

10 is an exemplary diagram illustrating a storage atlas in computer-readable storage device 1004 in accordance with some embodiments. The storage atlas 1002 includes first information structures 1006 relating to surgical procedures of the prior art, second information structures 1008 relating to teleoperated medical device operation, and surgical instrument operation during surgical procedures. third information structures (1010) associating patient health records with surgical procedures, fourth information structures (1012) relating patient health records to surgical procedures, and a fifth information structure relating patient health records to surgical instrument operation during medical procedures. s 1014. In some embodiments, storage atlas 1002 includes sixth information structures 1016 that provide correlation between surgical patient risks and surgical patient health records. In some embodiments, storage atlas 1002 includes seventh information structures 1018 that provide correlation between video images of surgical procedure stages and surgical instrument actuator states during surgical procedures.

In some embodiments, information in various information structures 1004-1020 is evaluated to identify correlations between patient health records and surgical procedure outcomes/risks. In some embodiments, information in various information structures 1004-1020 is evaluated to identify correlations between patient safety concerns/risks and stages of a surgical procedure. In some embodiments, teleoperated surgical procedures are evaluated to identify correlations between patient safety concerns/risks and surgical instrument operating state. Such assessments may involve machine learning (ML) techniques, for example. In some embodiments, storage atlas 1002 includes eighth information structures 1020 that provide correlation between surgical outcomes/risks and surgical instrument actuator states.

Storage atlas 1002 contains data relating to patients and surgeries. In some embodiments, storage atlas 1002 includes video images of surgical scenes and corresponding annotations, such as text and telestration tags 1022 . In some embodiments, storage atlas 1002 includes recordings 1024 of surgical instrument actuator states during surgical procedures.

11 shows a seventh information structure 1018 contained within an atlas 1002 in a storage device 1004 that associates recorded video information from an individual surgery with corresponding surgical instrument actuator state information, according to some embodiments. It is an exemplary diagram showing an example of. In one aspect, video recorded images during surgery and surgical instrument actuator states are used to create a chronological record of the times of occurrence of events seen in the video images and correspond to the times of occurrence of surgical instrument actuator states during a surgical procedure. are time stamped to provide a chronological record (t1, t2 ... tn). Accordingly, time stamps recorded during the surgical procedure are used to temporally align video images with surgical instrument actuator states.

During surgery, the user can annotate the video recordings and the mechanical surgical instrument operation status recordings with metadata representing the stage of the surgery. Annotation may include, for example, one or more or a combination of written notes tagged with video information and/or surgical instrument operating state information, coloring or highlighting (eg, telestration) of images in a video recording. can include Annotations may be time stamped for use in temporally correlating them with corresponding video recording information and corresponding recorded machine tool state information.

Surgical procedures can be associated with surgical risks. Some surgical risks are more strongly associated with a particular stage of a medical procedure. In some embodiments, different stages of surgery are demarcated by the use of different surgical instruments during the different stages. Moreover, some patients have medical conditions that put them at greater surgical risk than other patients during certain stages of the surgical procedure. For example, during surgical procedures, patients with chronic hypertension have a higher risk of stroke during stages of the procedure requiring the Trendelenburg position than patients without the condition.

In a teleoperated surgical system, different instruments may be used at different stages of a surgical procedure. Moreover, the same instrument may be used with different actuator states at different stages of a surgical procedure. Thus, in some surgeries a change in instrument means a transition to a different surgical stage corresponding to a different risk level, and in some surgical procedures a change in surgical instrument actuator state means a transition to a different surgical stage corresponding to a different risk level. means the transition of As used herein, the term actuator state is the mechanical disposition of a surgical instrument as determined by an actuator, such as a motor, in response to input commands received from a surgeon or other surgical team member. refers to

12A-12C are exemplary diagrams illustrating an exemplary surgical instrument 1202 and an actuator assembly 1203 in which the surgical instrument is shown in three different exemplary operating states in accordance with some embodiments. The exemplary instrument 1202 includes a jaw end effector 1204 capable of transitioning between open and closed states and a continuum of partially open/partial closed states in between. The example instrument 1202 also includes a two-degree-of-freedom (2-dof) list 1206 that can move between different two-dimensional (x, y) positional states. The exemplary actuator assembly 1203 includes a first actuator 1208, which in some embodiments includes a jaw motor (JM) used to actuate the jaw end effector 1204. The exemplary actuator assembly 1203 includes a second actuator 1210, which in some embodiments includes a wrist motor (WM) used to actuate the wrist 1206. During surgery, surgical instrument 1202 may transition through a number of operating states corresponding to different stages of a surgical procedure. As shown in FIG. 12A , for example, the surgical procedure is such that first actuator 1208, JM, places jaw end effector 1204 in a fully open position and second actuator 1210, WM, moves list 1206. It may involve a first stage that places it in a first location state (x1, y1). As shown in FIG. 12B , for example, a surgical procedure may involve first actuator 1208 transitioning jaw end effector 1204 to a fully closed state and second actuator 1210 moving wrist 1206 to a second position state. It may entail a second stage transitioning to (x2, y2). As shown in FIG. 12C , for example, a surgical procedure may include first actuator 1208 placing jaw end effector 1104 in a partially open/partial closed state and second actuator 1210 removing wrist 1206. It may involve a third stage transitioning to the three-position state (x3, y3).

The surgeon may tailor the surgical procedure based on the surgical patient's medical condition as indicated by the patient's health records. The surgeon may take precautions during the surgical procedure to reduce the risk to the patient based on the patient's health records. For example, a patient's health record may indicate a condition that correlates with surgical risk during a stage of a teleoperated surgical procedure. Precautions that can be taken to reduce the risk to such a patient during stages of a teleoperated surgical procedure include, for example, controlling the operation of instruments.

13 is an exemplary diagram of an atlas 1002 stored on computer readable storage device 1004 corresponding to a specific surgical procedure and associating different exemplary surgical patient health record characteristics with surgical risk management information in accordance with some embodiments. It is an exemplary diagram showing an 8-information structure 1020. The first column of information structure 1020 includes patient health record (HR) characteristics; Lists 1, 2, 3, 4, and 5. The second column of information structure 1020 represents the surgical risks associated with the HR characteristics at the stage of the surgical procedure. Specifically, for example, HR characteristic 1 is associated with a risk of 1 at stage 1 of a surgical procedure; HR characteristic 3 is associated with risk 2 at stage 3 of the surgical procedure; HR characteristic 5 is associated with risk 3 at stage 5 of the surgical procedure. The third column of information structure 1020 indicates surgical instrument operating status during surgical stages for which the HR characteristic is associated with increased risk. For example, HR characteristic 1 is associated with surgical instrument actuator state X; HR characteristic 3 is associated with surgical instrument actuator state Y; HR characteristic 5 is associated with surgical instrument actuator state Z. The fourth column of information structure 1020 represents messages to be presented to the surgical team at different stages of the surgical procedure, based on HR characteristics. For example, HR characteristic 1 is associated with message A, which is associated with stage 1 and surgical operating state X; HR characteristic 3 is associated with message B, which is associated with stage 3 and surgical operating state Y; HR characteristic 5 is associated with message B, which is associated with stage 5 and surgical operating state Z. A fifth column of information structure 1020 indicates the surgical instrument actuator safety modes to be used in different stages of the surgical procedure, based on the HR characteristics. For example, HR characteristic 1 is associated with safe mode L, which is associated with stage 1 and surgical operating state X; HR characteristic 3 is associated with safety mode M, which is associated with stage 3 and surgical operating state Y; HR characteristic 5 is associated with safe mode N, which is associated with stage 5 and surgical operating state Z.

Referring to row 1 of the exemplary eighth information 1020 structure of FIG. 13 , stage 1 of surgery may involve dissection, for example during a colonectomy surgical procedure. Risk 1 may involve damage to the ureters or bladder due to adhesions from previous surgeries, as indicated by the EHR. Message A may indicate that mobilization of the bowl should be limited until full exposure is established. The surgical instrument safe operating state L may involve limiting possible mobilization damage by reducing the force available to the instrument arms to a level lower than the level available during normal non-safety mode operation.

Referring to the third row of the structure of the exemplary eighth information 1020 of FIG. 13 , for example, during a prostatectomy surgical procedure, stage 3 of the surgery is near the nerves responsible for potency and continency. may involve exfoliation in Risk 2 may involve nerve damage due to application of cauterizing energy in the vicinity. Message B may indicate the presence of neural tissue within a predefined range of the cautery instrument. A surgical instrument safe operating state M may involve preventing activation of a cautery energization function.

Referring to row 5 of the exemplary eighth information 1020 structure of FIG. 13 , for example, during a surgical procedure involving liver tumor resection, stage 5 of the surgery may involve dissection of a tumor mass. can Risk 3 may involve damage to large blood vessels within the liver near the mass. Message C may indicate that the vessel is predicted to be near dissection progress as determined by preoperative imaging mapped onto the current surgical image. The surgical instrument safe operating state N may involve restricting the motion of the surgical instruments to a range less than the range available during normal non-safe mode operation, such that the instruments do not physically contact the artery.

14 shows configuration of processor 58 for performing a process 1402 of presenting an alert message and transitioning a surgical instrument to a safe mode based at least in part on the surgical patient's health record information, in accordance with some embodiments. This is an exemplary flow diagram. Computer program code is used to configure one or more CPUs of processor 58 to perform process 1402 in some embodiments. At block 1404 , the surgical patient's health record (HR) information is received at an input to a computer processing system associated with the electronics cart 56 . At block 1406 , surgical procedure information representative of one or more surgical stages of a surgical procedure is received at an input of a computer processing system associated with the electronics cart 56 . For some instruments used during a surgical procedure, the transition to use of the instrument represents a transition to a different surgical stage. For some instruments used during a surgical procedure, the transition of the instrument to a different actuator state represents a transition to a different surgical stage. At block 1408, the exemplary eighth information structure 1020 of FIG. 13 combines the HR characteristics of the received patient HR information with one or more surgical procedure stages to identify one or more surgical procedure stages at which the patient is at increased risk. used for matching. At block 1410, example eighth information structure 1020 is used to match one or more identified surgical procedure stages at which the patient is at increased risk with surgical instrument actuator state information indicative of the one or more identified stages. At block 1412, the exemplary eighth information structure 1020 is used to match one or more identified surgical procedure stages at which the patient is at increased risk with one or more corresponding warning messages for presentation to the surgical team. At block 1414, the exemplary eighth information structure 1020 is used to match the one or more identified surgical procedure stages at which the patient is at increased risk with information about corresponding surgical instrument safe operating conditions.

During performance of the surgical procedure, block 1416 determines, based on the surgical instrument actuator state information determined in block 1410, the patient with the HR characteristics for which the surgical procedure was received is identified in block 1408 as being at increased risk. Track the operating state of the surgical instrument actuators to determine when to transition to stage. At decision block 1418, a determination is made as to whether the current instrument actuator state matches the identified actuator state identified at block 1410. In response to no match, control loops back to block 1416 and tracking continues. In response to a match, block 1420 transmits the identified actuator safe mode control information identified in block 1312 to transition the surgical instrument actuator to safe mode operation to reduce risk to the patient with the received health care record. use. Block 1422 configures display devices 32, 34 and/or 60 to present a message to the surgical team regarding a possible increased risk to the patient having health care records received during the identified stage of the surgical procedure. In some embodiments, control may then proceed back to block 1416, where the surgical instrument actuator state may continue to be tracked, eg, based on other identified actuator state transition information.

While exemplary embodiments have been shown and described, wide modifications, variations, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be utilized without corresponding use of other features. For example, in some embodiments, processor 58 is coupled to a memory device, such as storage device 1004, containing instructions for implementing a virtual surgical system that includes virtual surgical instruments and virtual surgical instrument actuators. . Memory device 1004 includes a set of instructions executable on processor 58 to cause processor 58 to perform operations. In some embodiments, these actions include receiving user input commands from the user to control movement of the first virtual robotic surgical instrument. These operations further include tracking a virtual surgical instrument actuator state of the first virtual robotic instrument during movement of the first virtual robotic instrument in response to user input commands. These operations further include transitioning a virtual surgical instrument actuator state of the first virtual robotic instrument to a first safe mode in response to the first virtual robotic surgical instrument transitioning to the first actuator state.

Moreover, in some embodiments, the operations include receiving, at an electronic user interface, health record information of the surgical patient including the first health characteristic. These operations may include matching, within the computer-readable storage device, a received first health characteristic with a first virtual actuator state of a virtual surgical instrument and, within the storage device 1004, matching the received first health characteristic. 1 further includes matching with the message. These operations further include receiving user input commands from the user to control movement of the first virtual robotic surgical instrument. These operations further include tracking a virtual surgical instrument actuator state of the first virtual robotic instrument during movement of the first virtual robotic instrument in response to user input commands. These operations further include transitioning a virtual surgical instrument actuator state of the first virtual robotic instrument to a first safe mode in response to the first virtual robotic surgical instrument transitioning to the first virtual actuator state. These operations further include displaying a first message on a user device display in response to the first virtual robotic surgical instrument transitioning to the first virtual actuator state.

One skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the disclosure should be limited only by the following claims, and it is appropriate that the claims be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims (1)

A method of controlling a surgical system comprising surgical instruments and surgical instrument actuators using surgical health records, comprising:
receiving user input commands from a user to control movement of a first robotic surgical instrument;
storing on the storage device an information structure identifying the two or more different actuator states and the two or more safety modes and associating each of the two or more different actuator states with a respective different safety mode of the two or more different actuator states;
tracking a surgical instrument actuator state of the first robotic surgical instrument during movement of the first robotic surgical instrument in response to the user input commands;
the surgical instrument actuator state of the first robotic surgical instrument to the first safe mode included in the information structure in response to the transition of the first robotic surgical instrument to the first actuator state associated with the first safe mode in the information structure; transitioning; and
in response to a transition of the first robotic surgical instrument to a second actuator state associated with the second safe mode within the information structure, the surgical instrument actuator state of the first robotic surgical instrument to a second safe mode included in the information structure; stage of transition
Including, the control method.

Images