WO2021173044A1 - Method for controlling a camera in a robotic surgical system - Google Patents

Abstract

The invention relates to a method for controlling the movement of a camera fastened to a manipulator in a robotic surgical system. The system comprises two master controllers, each of which is capable of digitizing the movements of a surgeon's hands and transmitting movement vectors. Each master controller is capable of being switched into a camera control mode. The system includes a manipulator with a camera fastened thereto, providing for movement of the camera, and an automatic control system. Movement vector data from the right and left master controllers is transmitted to the automatic control system. The master controllers are switched into camera control mode. The movement of the manipulator is controlled by simultaneous movement of the master controllers and the implementation of the following steps: saving the received master controller movement vectors in the automatic control system; processing and combining the movement vectors of the right and left master controllers to find the total movement; scaling the resulting movement vector; compensating for motion; compensating for divergence of the horizon line from the centre line of the frame; simultaneously transmitting the resulting data to the operating mechanisms of the manipulator. The result is a reduction in the number of compensatory movements performed by the surgeon when controlling the camera.

Description

METHOD FOR CAMERA CONTROL IN A ROBOTSURGICAL COMPLEX

The technical field to which the invention relates

The invention relates to assisting surgical systems for minimally invasive surgical operations. More specifically, the invention relates to a method for controlling a camera in an operator controller compensation system for controlling a robotic surgical system. The invention can be used in various fields where it is necessary to visually interact with the working field located in a place that is difficult for the operator to access: medical robotics, industrial manipulators, interaction with objects in environments that are dangerous for the operator's life, and more.

Background of the invention

The robotic surgical system is a complex consisting of three main units, shown in figure 1. The complex consists of a master device 100, a control system 200 and an actuator 300. Controllers 100 act as a master device, with which the surgeon directly interacts - mechanisms, acting with his hands on which, the surgeon generates at least three translational and three rotational movements sufficient for carrying out a robotic surgery. Actuators 300 are manipulators 310, 320, on which surgical instruments 400 and an endoscopic camera 500 are attached, which makes it possible to transmit an image of the operating field. To display the working field, a three-dimensional imaging system is used, which transmits the image from a stereoscopic camera to a 3D screen located in front of the surgeon, with the effect of real depth. The interaction of all devices of the complex is carried out using the control system 200, which is the central node of the system.

Endoscopic cameras as applied to robotic surgery provide stereoscopic visualization and are one of the important units of the robot-assisting complex, since they provide the transmission of an image of the operating field to the operating surgeon. In the field of minimally invasive surgery, special attention is paid to the convenience of camera control, in particular, the intuitive interaction of the surgeon with the actuator.

Robotic surgery compares favorably with laparoscopy in that the camera is controlled directly by the operating surgeon (in laparoscopy, an assistant is often involved for this function).

The controller can, at the command of the surgeon, control either a manipulator with an instrument or a manipulator with a camera. In this case, the surgeon only needs to switch the controller from instrument control to camera control using the foot pedal. The main problem in camera control is that it operates in a spherical coordinate system, and the controllers that control it operate in a Cartesian coordinate system. If during the operation it is necessary to maintain a constant size and maintain the selected scale of the operating field, then when the camera is rotated, it is necessary to perform its additional compensating movement along the axis of the direction of vision: approaching or moving away, depending on which direction the camera is turned. Traditionally, this procedure has to be performed by a surgeon. But for its high-quality implementation, both considerable experience of the surgeon and the surgeon's ability to switch "in the mind" the Cartesian coordinate system to the spherical one and vice versa are required. Such conditions lead to a significant loss of time, an increase in the likelihood of a mistake by the surgeon and completely block the possibility of automating individual transitions of a surgical operation.

The possibility of automatic compensation of the camera position when it is controlled by the controller is an important and not yet solved problem in robotic surgery. The solution to this problem will allow automatically combining the Cartesian and spherical coordinate systems and automatically using the controller and / or manipulator to work out the necessary compensatory movements, thereby providing the surgeon with a convenient, intuitive control procedure.

In this regard, the selected methods and methods of controlling the camera transmitting the stereo image to the operator are important and sometimes decisive for the operation of the robot-assitant complex as a whole. This is primarily due to the need to implement a compensation system for the operator's controller, which will programmatically implement the method of an intuitive interface for the interaction of the surgeon with the manipulator on which the endoscopic camera is attached.

The interaction of the surgeon with the camera is different from the control of surgical instruments. This is due to the number of degrees of freedom: to control the camera, it is sufficient to control three translational degrees of freedom, and to control the instrument, three rotational degrees of the jaws (tip) of the instrument are added to them.

Differences in the local coordinate systems of the master (controller) and actuator (manipulator) devices, which are explained by the mechanical features of the structures, are insignificant when controlling a surgical instrument, but they are also a source of disagreement that does not allow the surgeon to intuitively, without cognitive correction, interact with the camera. The surgeon has to understand and remember during his work that when controlling the camera, the change in its position will not follow exactly the trajectories that it sets. This leads to the fact that in order to effectively control the camera, the surgeon needs to additionally master professional skills, the acquisition and consolidation of which will take a significant period of time. Intuitive controls are camera controls based on the same motion order as surgical instruments. Discrepancies in the control of the instrument and camera movements by the controller are insignificant during surgical procedures when approaching the operated area very close, but in the case of the procedure for examining the operating field at a certain distance, the discrepancies become already significant, which forces the surgeon to make additional movements of the camera to compensate for them. Insufficient qualification of the surgeon or lack of experience in compensating for the position of the camera can create problems associated with the loss of the operating field during the operation or inadequate perception of its location and size. Compensating for camera position becomes a key challenge when automating individual surgical transitions.

There are camera manipulator control methods that address the problems described above. For example, such methods are disclosed in US patents US 6574355 B2, US 9188973 B2, US9949798B2.

In the method described in US Pat. No. 6,574,355 B2. software and hardware compensation is performed when the camera moves. Changing the visible position of the manipulator tips improves operator comfort when working with an object in the work area. But this compensation is aimed at correcting the camera's field of view and is responsible for the synchronous positioning of surgical instruments on the camera image. The translational movement of the camera itself during the transition from the coordinate system of the control controller to the coordinate system of the manipulator is not compensated in any way.

The method described in US 9188973 B2 uses an algorithm for determining the coordinates of the robot arm based on its location in the camera coordinate system. Coordinate transformation data facilitate the calibration of camera systems. Using the proposed methods allows you to more accurately position the entire system, as well as control the camera, depending on the problem being solved. However, there is no compensation for camera movement when changing position.

In US 9949798 B2, a method for controlling a camera is presented, which uses various local coordinate systems for a control controller as a master device and a camera manipulator as an actuator, which makes it possible to move from one coordinate system to another during control and implement separate methods for controlling a camera within the operating field. ... At the same time, this technical solution does not pay special attention to the intricacies of control and interaction, it is assumed a straight-line transition from one coordinate system to another without introducing any methods of mechanical or software compensation for special cases and camera control modes. For the proposed method of interaction and control of the camera, the surgeon needs additional training and getting used to, since these movements are not intuitive and require certain control skills.

Review of modern solutions, as well as available scientific research in the field of robotic surgical systems, in particular software or hardware methods compensation of movements, leads to the conclusion that there is a need to create an improved system of interaction between the surgeon and the robot-assisting complex. The existing developments of methods and systems for controlling camera movement as part of robotic surgical systems do not allow solving the following set of problems:

1. Lack of intuitive interaction between the surgeon and the camera during its movement, which ensures the retention of the image of the operating field within the specified field of view. Such a property of the system as intuitive control of the system relieves the surgeon of the obligation to constantly monitor, calculate in his head and, on the basis of calculations, make additional compensatory movements by the controller when moving the camera.

2. Lack of software or mechanical methods of compensation for camera control during robotic surgery, which relieve the surgeon of additional control commands to move the camera in order to keep the image at the target point of the surgical field.

3. Lack of software methods for compensating for discrepancies between the middle line of the frame and the horizon line, given the technical capability of the manipulator design.

4. Lack of scaling of movements by the controller at different distances of the end of the stereoscopic camera located in the patient's body from the "zero point" (the point of entry of the camera into the patient's body).

5. The need to acquire certain skills in camera control to understand the operation of existing imaging systems in robotic surgical complexes.

Thus, there is a need to improve the method of compensation of translational movements in the camera control mode in order to provide the surgeon with the possibility of more intuitive and comfortable control of the robotic surgical complex, and in order to provide a more perfect interaction with the complex, which in turn increases the quality and the effectiveness of robotic surgery. This application is dedicated to the solution of the listed problems.

The essence of the invention

The technical problem to be solved by the present invention is the intuitive control of the camera movement for observing the operating field without the need for mastering additional compensation skills during control, improving the quality and efficiency of robotic surgery.

The technical result of the present invention is to create a method of interaction of a control controller with a mechanical manipulator, which is part of a robotic surgical complex, with a camera attached to it, which makes it possible to reduce or completely remove the number of compensatory movements performed by the surgeon during control camera, thereby reducing the duration of the operation, reducing the risk of surgeon errors, as well as reducing the surgeon's fatigue based on the most intuitive camera control procedure.

Convenient and intuitive control of the camera is achieved due to the investigated, simulated and software-implemented system that fixes the control commands generated in the controller's coordinate system and converts them into control commands, but already in the camera manipulator coordinate system, which is a mechatronic-software compensation camera movement.

A method for compensating for camera movements includes: providing compensation for the plane of movement of the camera manipulator; ensuring scaling of movements of the control controller; ensuring the leveling of the discrepancy between the center line of the frame and the horizon line.

The proposed method for controlling the movement of a manipulator with a camera attached to it must meet the following requirements:

1. Implementation of the method of compensation of camera movement should increase the intuitiveness of control for the surgeon.

2. The compensated manipulator movement should accurately reproduce the position of the camera in the operating area in accordance with the control commands of the surgeon's control controller.

3. The use of compensation methods should not reduce the surgeon's working area within the operating field and should allow the surgeon to change the camera's field of view in the operating field with a minimum number of control movements of the control controller.

The proposed method for compensating the camera movement will allow changing the position of the end of the camera in one plane depending only on the depth of immersion of the camera relative to the "zero point". The introduction of additional methods to compensate for the discrepancy between the center line of the frame and the horizon line will reduce the drawbacks that arise when the field of view of the camera changes and the surgeon adapts to the result. The method described below for controlling the movement of the camera manipulator by the control controller is configured to facilitate the movement of the stereo camera that visualizes the surgical area in the operating field due to the compensatory scaling of the controller movements. The method involves controlling the camera in a coordinate system that is similar to the Cartesian coordinate system used to move tools.

The fidelity of the reproduction of the control commands is guaranteed by a mechanical manipulator to which the camera is attached, the design features of which are not considered in this application, but which guarantees at least three translational degrees of freedom for the camera.

The accuracy of determining the change in position, as well as the generation of control commands, is guaranteed by the control controller, which ensures the transmission of the change in position in at least three translational degrees of freedom. The technical result is achieved by creating a method for controlling the movement of the camera fixed on the manipulator of the robotic surgical complex, which includes two controllers for controlling the robotic surgical complex, each of which is designed to digitize the movements of the surgeon's hands and provides the transmission of motion vectors along three translational and three rotational degrees of freedom, representing the difference between the coordinates of the control controller in the initial position and the coordinates of the controller when the position of the surgeon's hand changes, while each control controller is configured to switch to the camera control mode using the control pedal, at least one manipulator with a camera attached to it for viewing the operating field, providing its movement along three translational degrees of freedom, an automatic control system, which is connected with both controllers controlled by the surgeon, and at least one manipulator, while cn is characterized by the following steps: transfer of motion vector data along three translational and three rotational degrees of freedom of the right and left control controller to the automatic control system with a constant frequency, switching controllers to the control mode of the camera attached to the manipulator by pressing the control pedal, when The signal from the control pedal is transmitted to the automatic control system and to the control controllers to block the motion vectors of the rotational degrees of freedom at the latter, and after blocking the rotational degrees of freedom of the control controller, the vector of rotational coordinates in the automatic control system is reset to zero; control of the motion of the manipulator with the camera attached to it by simultaneously moving the controllers and then performing the following stages: the stage of storing the obtained motion vectors of the controllers in the automatic control system, the stage of processing and combining the motion vectors of the right and left controllers characterizing the translational movements to find total displacement, the stage of scaling the displacement vector obtained at the previous stage, the stage of motion compensation, containing the summation of the scaled displacement vector and the vector of the current position of the specified manipulator in its local Cartesian coordinate system, and the translation of the obtained vector of the position of the manipulator from Cartesian to spherical coordinates, the stage of deflection compensation the horizon line from the center line of the frame by calculating the value of the divergence of the angle depending on the tilt angles of the camera in the spherical coordinate system T, the stage of simultaneous transmission of the data obtained at the previous steps to the actuators of the manipulator for its translational movement and rotation around the longitudinal axis of the camera fixed on it.

In this case, the scaling of the displacement vector is carried out taking into account the distance of the camera from the zero point of the camera in the local coordinate system of the camera, relative to which the position vector of the camera changes with the beginning at the point where the camera enters the hole in the patient's body and with the end that coincides with the actual position of the end of the camera.

Brief Description of Drawings

The accompanying drawings, which are incorporated and form part of the present specification, illustrate an embodiment of the invention and together with the foregoing general description of the invention and the following detailed description of exemplary embodiments of the invention serve to explain the principles of the present invention. For clarity and simplicity, the drawings are not necessarily drawn to scale.

Figure 1 shows a block diagram of a robot-assisting complex.

Figure 2 depicts a structural model of a robotic surgical system that is used for the present invention.

Figure 3 is a perspective view of a surgeon control controller used in the present invention.

Figure 4 depicts a general schematic block diagram of the operation of the surgeon control controller.

Figure 5 depicts the location of the local spherical coordinate system of the manipulator, originating at the "zero point".

Figure 6 depicts the location of the Cartesian orthogonal coordinate system of the manipulator with the origin at the "zero point".

Figure 7 depicts a simulation of the change in the position of the end of the camera when changing each coordinate separately. Changing the length of the camera relative to the "zero point".

Figure 8 depicts a simulation of the change in the position of the end of the camera when changing each coordinate separately. Changing the tilt of the camera relative to the longitudinal axis.

Figure 9 depicts a simulation of the change in the position of the end of the camera when changing each coordinate separately. Changing the tilt of the camera relative to the transverse axis.

Figure 10 shows the trajectory of movement of the camera manipulator when changing the angle Q along the longitudinal axis at constant f and R. The movement is carried out in a circle with a radius R.

Figure 11 schematically shows the relationship between movements of a surgeon control controller and movement of a camera arm in a Cartesian coordinate system. Figure 11A schematically shows the movements of the surgeon's hand in the Cartesian coordinate system control controller. The movement of the camera in the Cartesian coordinate system of the camera manipulator is shown in figure 11B.

Figure 12 shows the trajectories of the camera tip in the Cartesian coordinate system of the camera and the compensated view of this trajectory after applying the proposed camera motion compensation algorithm. Figures 12A, 12B and 12B show the trajectories of the camera tip during its various movements.

Figure 13 depicts the movement made by the surgeon's control controller and the deviation of the frame centerline from the horizon. Figure 13A shows an in-frame image at zero tilt angle of the camera arm. Figure 13B shows an in-frame image at a positive tilt angle of the camera arm. Figure 13B depicts the required hand movement of the operator to reconstruct the horizon line in the controller's Cartesian coordinate system.

Figure 14 is a schematic representation of a camera manipulator zooming algorithm.

Figure 15 is a schematic representation of a detailed functional diagram of a camera control.

Figure 16 depicts a block diagram of the operation of the automatic control system of the camera manipulator.

Figure 17 depicts the camera manipulator in spherical coordinates f = 0 °, Q = 0 °, R = 110mm.

Figure 18 depicts the camera manipulator in spherical coordinates f = 0 °, 0 = 30 °, R = 110mm.

Figure 19 shows the sequence of movement of the camera manipulator without using the proposed method of compensating for camera movement (without the applied plane compensation algorithm).

Figure 20 depicts the camera manipulator in spherical coordinates f = 0 °, Q = 30 °, R = 127mm.

Figure 21 shows the sequence of movement of the camera manipulator using the proposed method of compensating for camera movement (using the applied plane compensation algorithm).

Figure 22 shows a sequence of images along a projected trajectory without horizon compensation.

Figure 23 shows a sequence of images along a projected horizon compensated trajectory.

Figure 24 depicts an object of the operating field with applied control points (left), the image of the first control point in the frame (right).

Figure 25 shows the results of a series of experiments after statistical processing of data reflecting the average time for positioning the camera with and without the proposed method of controlling camera movement.

Figure 26 shows the results of a series of experiments after statistical processing of data reflecting the average distance of movement of the surgeon's arms during positioning. manipulator of the camera with the use of the proposed method of controlling the movement of the camera and without it.

Terms and Definitions

For a better understanding of the present invention, below are some of the terms used in the present description of the invention. Unless otherwise specified, technical and scientific terms in this application have the standard meanings generally accepted in the scientific and technical literature.

In the present description and in the claims, the terms "includes", "including" and "includes", "having", "equipped", "containing" and their other grammatical forms are not intended to be construed in an exclusive sense, but, on the contrary, are used in a non-exclusive sense (ie, in the sense of "including"). Only expressions of the type “consisting of” should be considered as an exhaustive list.

In these application materials, the terms “robotic technological complex”, “robotic system”, “robotic complex”, “robotic surgical complex”, “robotic surgical system” mean complex systems or complexes in surgery using a robot assistant during an operation. "Robot assistive systems" or "robotic assisted surgical systems" are robotic systems designed to perform medical operations. These are not autonomous devices, robotic assistive systems are controlled by surgeons during the operation.

In these application materials, the term "mechatronic complex" or "mechatronic system" means a complex or system with computer control of motion, which is based on knowledge in the field of mechanics, electronics and microprocessor technology, computer science and computer control of the movement of machines and assemblies.

In this application, the term "operator" refers to the performing the operation of the surgeon. The terms "operator" and "surgeon" in the present description of the invention are synonymous.

The term "connected" means functionally connected, and any number or combination of intermediate elements between the connected components (including the absence of intermediate elements) can be used.

In addition, the terms "first", "second", "third", etc. are used simply as conditional markers, without imposing any numerical or other restrictions on the enumerated objects.

Detailed description of the invention

The description of exemplary embodiments of the present invention below is provided by way of example only and is intended for illustrative purposes and is not intended to limit the scope of the disclosed invention. The present solution relates to a method for controlling the movement of a camera fixed on a mechanical manipulator of a robotic surgical complex, which allows obtaining an image of the operating field during a surgical operation.

An example of the structure of a robotic surgical complex is shown in figure 2. The robotic surgical complex consists of three interconnected main units: a patient trolley (actuator) 300, an automatic control system 200 and a control console (driver) 100, which receives commands from a surgeon 101 to further transform them into movement of surgical instruments fixed in the manipulator or to ensure the generation of control commands from the surgeon for other units of the robotic surgical complex.

Also, one of the important units in the robotic surgical complex is the stereo imaging system (imaging system), which includes a camera 500 for obtaining an image of the surgical field 330 and a monitor of the imaging system 130 for displaying a three-dimensional image of the surgical field obtained from the camera. The efficiency of the operation depends on the quality of the image obtained by the surgeon. The modern development of mechatronic systems and the constant improvement of robotic surgery imposes new requirements on the developed devices of the imaging system.

The control console 100 is located outside the sterile zone of the surgical unit and is configured to control: manipulators 301, 302, 303 with surgical instruments attached to them; a manipulator 320 with a camera 500 attached to it; directly by the surgical instruments themselves. The control involves a controller 120 operated by the surgeon's hands and pedals operated by the surgeon's feet.

Typically, manipulators with surgical instruments and a camera are mounted on a patient trolley, which is designed to support and position them relative to the patient. It should be understood that the robotic surgical system can have any number of manipulators, such as one or more manipulators. Figure 2 shows three manipulators 301, 302, 303, made with the ability to move in three planes and rotate in three planes, as well as the manipulator 320 of the camera 500. All manipulators indicated in the structural diagram and being part of the patient cart have general mechanical characteristics and design features. Each manipulator has a casing and a manipulator connection unit, to which a surgical instrument or camera can be detachably attached, the movement and position of which the surgeon can change by manipulating / controlling using a control controller that digitizes the movement of the surgeon's hand.

The surgeon controller 120 allows control of surgical instruments and a camera located within a patient during surgery. The control controller converts the mechanical movements of the surgeon's hand over the entire natural range of motion in six degrees of freedom to generate control commands for robotic surgical complex.

The surgeon's controller generates a command to move the surgical instrument. Additionally, the controller controls the turning and opening / closing of the jaw on the surgical instrument. The surgeon has the ability to generate at least three translational and three rotational degrees of freedom and additionally at least one degree of freedom when closing / opening jaws, which is sufficient to control a surgical instrument during a surgical operation.

Pedal 110 in this application is understood as a contact switching device (mechanical or electronic) capable of switching on / off the flow of current in a circuit. A pedal, button, switch, switch and the like can act as such an apparatus. Hereinafter, in the framework of this application, the pedal 110 is a footswitch that closes an electrical circuit when the surgeon depresses the pedal with his foot. There can be several pedals in the control console. The pedals are designed to change the operating modes of the control controller or to switch additional functionality, for example, coagulation, laser, and the like. Footswitches allow the surgeon to control the camera, instruments, electrosurgical instruments.

The automatic control system 200, based on the data received from the controllers 120 and the signal from the pedals 110, generates control commands that can be directed both to the manipulators 301, 302, 303 with surgical instruments and to the camera manipulator 320. Pressing and holding footswitch 110 disables controllers 120 from operating surgical instruments (from operating manipulators 301, 302, 303 with surgical instruments) and enables and allows movement of manipulator 320 with camera 500. In this mode, both controllers 120, right and left work at the same time. When the pedal 110 is released, the controllers 120 will again operate the surgical instruments. In this case, the automatic control system 200 receives data on three translational and three rotational degrees of freedom from the control controller 120 and generates on their basis three translational and three rotational movements of the manipulator with a surgical instrument. When the camera pedal 110 is depressed, it is sufficient to generate three translational degrees of freedom. Thus, the rotational degrees of freedom of the controllers 120 are locked.

The image of the area to be operated, broadcast by the stereoscopic camera 500, is available to the surgeon on the monitor 130 of the imaging system.

As a control controller, it is proposed to use a mechanical structure 600 with a parallel structure, the view of which is shown in figure 3. This structure 600 allows you to provide at least three translational degrees of freedom by reciprocating movement of the mechanism along three mutually orthogonal axes indicated in figure 3, which are sufficient to control the camera arm in the composition of the robotic surgical complex. The depicted controller 600 consists of at least two platforms: a fixed support 610 and a movable 620, and a parallel structure, preferably made on the basis of a delta-type mechanism (delta robot or deltapod), as well as a drive mechanism that drives into the movement of the delta robot, while ensuring minimal backlash.

The delta robot consists of three arms 630, located at an angle of 120 ° relative to each other and attached to the support platform 610. The advantage of the delta robot design is the use of parallelograms containing rods 640 of constant length, arranged in parallel in pairs and connected to each other by means of cardan joints ... The parallelograms are connected at one end by the corresponding levers 630, and at the other end are connected to the movable platform 620. This design allows you to maintain the spatial orientation of the robot mechanisms. In this case, the movable platform 620 is always parallel to the support platform 610. The connection of the arms 630 to the support platform 610 is made through the upper bearing assemblies 650 to provide the necessary angles for the initial state of the delta robot. The upper bearing assemblies 650 are fixed to a support platform 610. Mounted on the upper bearing assemblies 650, the levers 630 form an equilateral triangle at the centers of the connection, the angles of which affect the size of the useful working area of the delta robot. Increasing the length of the arm increases the Z travel. The dimensions for X and Y travel are given by parallelograms.

To control the three rotational degrees of freedom, a handle is attached to the movable platform 620 of the controller 600 using any of the known methods. The design of such a handle is beyond the scope of this application.

The use of a parallel structure provides high positioning accuracy, reduces stress on the operator's arms, increases the range of movement of the arms, and minimizes restrictions on the surgeon's ability to manipulate surgical instruments, including the camera.

The reproduction of commands from the control controller to the direct movement of the camera is carried out by a mechanical manipulator, which is capable of moving in at least three translational degrees of freedom and providing rotation around the longitudinal axis of the device on which the camera (endoscope) is attached.

A block diagram 700 of the control controller operation algorithm is shown in FIG. 4.

At the initial moment of time, at the first stage of work 710, the surgeon sits down at the control console and begins to interact with the controllers (left and right),

At the second stage of operation 720, the current value of the control pedal P. is taken.

In a third step of operation 730, the pedal control signal P is compared with zero to determine the operating mode of the control controller.

In the mode of operation with surgical instruments, when the pedal is released (the control signal P from the pedal, equal to zero, is sent to the automatic control system), each the surgeon's control controller at a constant rate transmits the three translational and three rotational degrees of freedom data to the automatic control system (step 750). Data can be represented as two different vectors:

D = [D _X , D _Y , D _Z ] ^T - translational coordinates R = [R _x , R _Y , R _Z ] ^T - rotational coordinates

In the control mode of manipulators with surgical instruments, packets with six coordinates are sent from the controllers to the automatic control system.

When you press the pedal responsible for switching to the camera manipulator control mode, if at step 730 the control signal from the pedal is different from zero, then the rotational degrees of freedom on the right and left controller are blocked (step 740), leaving the possibility only for translational movements ... In addition, the pedal signal and the three translational degrees of freedom data from the controllers are passed to the automatic control system (step 750), thereby notifying the system that the camera manipulator has begun to operate.

A camera is broadly defined in this application as any device structurally configured to generate a stereo image when inserted into a patient's body. An image of the operating field can be obtained optically using fiber optics, objectives and miniaturized imaging systems, for example, using a video endoscope (hereinafter referred to as an endoscope).

The camera is inserted into the body through an opening in the patient's body. The entry point / plane is defined as the "zero point" since all movements of the instrument or camera in the patient's body are provided by two rotational degrees of freedom about this point and one linear degree of freedom. A trocar is located at the “zero point” for additional fixation of instruments and a camera. There is also a "zero point of the camera" in the local coordinate system of the camera, relative to which the position vector of the camera changes its position with the beginning at the "zero point" and with the end coinciding with the actual position of the end of the camera. The camera has a field of view that depends on the characteristics of the camera lens. The camera transmits an image of only that part of the surgical field that fell into the field of view.

The camera's field of view can have an area configured to be visible to the camera at any given time.

As the camera moves, the field of view moves with the camera, allowing you to visually inspect the operating area. The change in the field of view also depends on the peculiarities of camera movement. In this case, the surgeon must be guaranteed unambiguity in understanding the movement of the camera and the associated movement / change of the field of view within the operating field.

Particular attention is paid to the manipulator, the work of which should be guided by some features that arise due to the specifics of robotic surgery. Since surgical instruments, as well as a stereo camera are introduced into the patient's body through the holes, the trocar is inserted, the position changes are made relative to the "zero point".

In order to simplify the mathematical apparatus and structure of the automatic control system, the local coordinate system of the manipulator is described by a spherical coordinate system, in which movements are carried out by angles f, q relative to the "zero point" around the longitudinal and transverse axes, as well as the radius of the camera extension R relative to the "zero point" ... Coordinates (cp.O.R) form the manipulator's local spherical coordinate system.

The local coordinate system of a mechanical manipulator can be considered as a change in the camera length R, as well as tilts f, q in both directions relative to the "zero point" around the longitudinal and transverse axes. This is a sufficient set of coordinates with which you can reproduce the control commands generated for the camera manipulator. The location of the local spherical coordinate system with the origin at the "zero point" is shown in figure 5. Figure 5 shows the manipulator assembly to which the camera is attached by means of fastening with the possibility of changing its orientation and position.

In addition to the spherical coordinate system, the movement of the manipulator can be described in the Cartesian orthogonal coordinate system with the origin at the "zero point" (figure 6). The CU plane in this case will be parallel to the plane of the lower platform of the manipulator (not shown in the drawing), and the OZ axis is perpendicular to this plane and is positively directed from the "zero point" towards the end of the stereoscopic camera.

Figures 7-9 show changes in the position of the end of the camera when changing each coordinate separately. Figure 7 depicts the change in the length of the chamber relative to the "zero point". This coordinate in the local coordinate system is responsible for approaching or moving away from the operated work area. It is important to note that within the framework of modeling a change in position, the mechanical features of the manipulator are not considered, but attention is paid to how the position of the camera changes when a certain coordinate is changed. Figure 8 reflects the change in the tilt of the camera relative to the longitudinal axis, which allows the surgeon to change the position in the operating field along the longitudinal axis in the field of view. Changing the tilt of the camera with respect to the transverse axis is shown in figure 9. This change allows the surgeon to change the position in the operating field along the transverse axis in the field of view. Thus, any position of the camera in the operating field can be unambiguously determined using these three local coordinates for the mechanical manipulator on which the camera is attached.

Figure 10 shows as an example the trajectory of movement of the camera manipulator (trajectory of the end of the camera) only when the angle of inclination along the longitudinal axis Q is changed, at constant values of the angle of inclination along the transverse axis and the coordinates of the camera length f and R. In this case, the camera will move along the inner the surface of a sphere with a radius equal to R.

However, the controller moves in its own orthogonal Cartesian coordinate system. The difference in local coordinate systems between the controller and the manipulator leads to the fact that the surgeon has to make additional movements with the controller, which, during a long operation, can lead to premature fatigue and errors when holding a given operating field. Reducing the number of movements performed by the surgeon is one of the criteria for improving the quality of the operation.

Let us consider the case in which it is necessary to move the field of view of the camera relative to the object under study at a given height (Figure 11). Figure 11A schematically shows the movements of the surgeon's hand in the Cartesian coordinate system of the control controller, which provide the necessary movement of the chamber manipulator. To move the field of view of the camera relative to the object under study, the surgeon must first move the controller in the same direction. It should be borne in mind that the camera manipulator moves along the inner surface of the sphere with a radius equal to the length of the stereoscopic camera from the "zero point". The movement of the camera in the Cartesian coordinate system of the camera manipulator is shown in figure 11B.

Legend shown in figure 11: 10 - initial and intermediate position of the operator's hand; 20 - the final position of the operator's hand; 30 - initial and intermediate position of the camera, 40 - final position of the camera, 50 - direction of movement, 60 - object under study, 70 - "zero point" of the camera.

Further, the surgeon must manually compensate for the distance by which the position of the camera has changed relative to the initial distance from the object under study. Thus, as follows from figure 11, the surgeon brings the camera to the desired position in two blocks of movements. For example, to move the camera manipulator along one axis OX along the radius, the surgeon had to perform two movements - along the OX axis in the same direction as the manipulator, and along the OZ axis, towards the object under study in order to compensate for the deviation caused by movement along the radius , and thus provide linear movement.

In addition to errors in additional movements by the surgeon, in order to move the camera from one coordinate to another, the surgeon's perception of images can also introduce distortions that arise due to the difference in the coordinate system of the control controller and the camera. With such control, there is no intuitive control of the camera, since the rectilinear movement of the control controller is forced to oblique movement of the actuator (manipulator with the camera), which leads to discrepancies in the perception of the surgeon. In this regard, to control the camera, the surgeon needs additional training and getting used to.

The method of compensation of camera movement will allow changing the position of the end of the camera in one plane, equidistant from the operated area and depending only on the depth of immersion of the camera relative to the "zero point". This compensated movement will facilitate camera control in the operating field by controlling and reproducing camera movement in similar Cartesian coordinate systems. Ultimately, this leads to a leveling of differences in the local coordinate systems of the master and actuator, which increases the intuitiveness of camera control for the surgeon, eliminates unnecessary operator actions and reduces the number of movements of the surgeon control controller, which leads to an increase in the quality and efficiency of operations.

Figures 12A, 12B and 12C show the trajectories of the camera tip in the Cartesian coordinate system (movement along the inner surface of a sphere with a radius equal to R) and the compensated view of this trajectory (straight-line movement) after applying the proposed method of compensating for camera movement when changing the angle of inclination along the longitudinal axes at fixed coordinates of the camera length and tilt angle along the transverse axis on the control controller, when changing the tilt angle along the transverse axis at fixed coordinates of the camera length and tilt angle along the longitudinal axis on the control controller and when changing the camera tilt angles along the longitudinal and transverse axes with fixed camera length coordinate.

In addition, when the camera manipulator is tilted relative to the "zero point" (figure 8, figure 9), there is a deviation of the center line of the frame from the horizon line. This distortion occurs due to the motion described above in a spherical coordinate system.

In more detail, the movement made by the controller and the deviation of the center line of the frame from the horizon line is shown in figure 13. Figure 13A shows the image in the frame at zero tilt angle a of the camera manipulator. Figure 13B shows the image in the frame with a positive tilt angle a of the camera manipulator a - the angle between the middle line of the frame and the horizon line. Figure 13B depicts the required hand movement of the operator to reconstruct the horizon line in the controller's Cartesian coordinate system.

Legend shown in figure 13: 11 - horizon line, 12 - middle line of the frame, 13 - image of the investigated object, 10 - initial position of the operator's hand, 20 - final position of the operator's hand, 50 - direction of movement.

The discrepancy between the center line of the frame and the horizon line is eliminated by rotating the endoscope around its own longitudinal axis in the direction opposite to the inclination (Figure 13B). Usually, the operator performs this procedure manually, if there is such a technical capability incorporated into the design of the manipulator.

To switch to the endoscope rotation mode, the operator presses the pedal, which transmits a signal to the automatic control system and controllers. Rotation is carried out by simultaneously moving both controllers along a circle clockwise or counterclockwise in a given plane. The diameter of the circle is the distance between the controllers.

Thus, the process of aligning the horizon line in the frame is a periodic procedure for the surgeon, which leads to additional hand movements.

Automation of the processes of leveling the horizon line and moving the field of view the camera relative to the investigated object will eliminate unnecessary actions on the part of the operator and reduce the number of movements by the controller.

An additional functionality that needs to be implemented when creating a control system for a camera manipulator is the need to scale the movements of the manipulator with the camera in the XY plane depending on the z coordinate. This need is due to the fact that the higher the stereo camera is above the object or space under study, the less movements must be made by the manipulator to view the working area. A schematic of the algorithm for changing the scale of the camera manipulator is shown in figure 14. Figure 14A shows the increment of the coordinate along the X axis for the surgeon's control controller in the Cartesian coordinate system of the control controller. Figure 14B shows the change in the position of the camera when the control controller moves in the Cartesian coordinate system of the camera manipulator.

Legend shown in figure 14: 10 - initial and intermediate position of the operator's hand, 20 - final position of the operator's hand, 30 - initial and intermediate position of the camera, 40 - final position of the camera, 50 - direction of movement, 70 - zero point of the camera.

Consider the movement of a camera manipulator using XZ motion scaling (Figure 14). The defining motion of the DC controller is the same for both cases, the z coordinate in the local Cartesian coordinate system is different - the projection of the endoscopic camera length from the "zero point" to the end onto the local OZ axis - DZi> DZ. In the first case, the camera is at a greater distance from the "zero point" than in the second. At the same time, for DC, which specifies the same movement, the displacement DCi> DC are not the same, which ensures the correct scale of movements.

Currently, existing approaches to camera control do not use any software or hardware methods to compensate for camera movement in a plane, but reproduce it in a straight line, responding to a control command to move the controller along one axis as a change in tilt relative to the same axis of the manipulator.

This approach is easily implemented in practice and guarantees a sufficient set of data for camera control, but does not provide high quality control, since differences in the master and execution coordinate systems do not allow the surgeon to intuitively interact with the complex, which leads to additional movements of the controller by the surgeon, increases the operating time. interference can lead to premature fatigue and errors.

Due to the fact that in response to the rectilinear movement of the control controller, an oblique movement of the actuator is carried out, the surgeon must constantly take this fact into account when controlling the camera. Thus, for the existing methods of controlling the camera, the surgeon needs additional training and getting used to, since these movements are not intuitive and require certain skills in comprehending and assessing what he saw. .1.1 i follow-up management.

The proposed method for compensating the plane of camera movement implies the introduction of additional algorithms into the automatic control system. Automating the horizontal alignment and zooming of camera movements will increase intuitive control and reduce surgeon fatigue.

The described method for controlling the movement of the camera manipulator included in the robotic surgical system according to the present invention is adapted to reduce the number of movements of the stereo camera that visualizes the surgical area in the operating field.

A detailed functional diagram of the camera control is shown in Figure 15.

The diagram shows the main stages of the implementation of the claimed method, namely, data transfer from controllers, analysis and processing of the data obtained by applying sequential algorithms for transforming the coordinates of the camera manipulator from the moment the automatic control system receives the absolute coordinates of the position of the surgeon's hand and the state of the pedal from the controllers.

The automatic control system performs four sequential algorithms, the first two of which are designed to perform the main function - manipulator control, and two additional ones allow to compensate for the above disadvantages.

The detailed steps of the proposed method are presented below.

The local coordinate system associated with the control controller contains information about the position of the controller in the form of three translational Cartesian coordinates D =

OcL \ Rx

D _Y and three rotational coordinates R = Ry

D _Z \ W

_{The vectors R! T} , D _r are transmitted from the right controller of the surgeon to the automatic control system. Vectors are transmitted from the left controller of the surgeon to the automatic control system

RL 'DL-

When the camera manipulator control pedal is pressed, the camera control mode is activated, the signal from the pedal is transmitted not only to the automatic control system, but also to the controllers that block the rotational degrees of freedom, and the R _R and R _L vectors are reset to zero:

R _x - R _Y - R _z 0, it follows that

Ό

RR = R L 0 0

To control the camera manipulator, three translational degrees of freedom are sufficient, described by the vector D. The vectors obtained by the automatic control system are processed using the algorithm for differentiating and combining motion. Control is carried out using the relative movement of the controller from the moment the camera starts control, initialized by pressing the pedal. Thus, the flexibility of the control system is increased.

The camera is controlled by both controllers at the same time. Thus, the following summarizes the relative movement of each of the controllers:

The resulting AD vector can be scaled, inverted, rotated around orthogonal axes, and composited with turns. Any of the listed mathematical operations can be performed using a cross product.

Further scaling is carried out. Scaling is a cross product of AD and the line S = [S ^, S _Y , S _Z \, containing the scaling factors for each coordinate.

The scaling operation takes place in the scaling and integration unit. The scaled increment of the surgeon's controllers obtained at the previous stage is added to the current position of the chamber manipulator in its local Cartesian coordinate system.

Thus, the in-plane motion compensation unit receives the calculated position of the camera manipulator in the local Cartesian coordinate system.

To control the manipulator, it is necessary to convert the calculated Cartesian coordinates into spherical ones. In this regard, the operations of transferring vectors from one coordinate system to another are important. Conversion from Cartesian to spherical coordinate system is carried out by the following algebraic transformations:

In addition to the algorithm for compensating for the manipulator's movement in the plane, it is also necessary to implement an algorithm for comparing the horizon line with the middle line of the frame, which consists in calculating the value of the angle of divergence between the horizon line and the center line of the frame, depending on the tilt angles f and Q. The divergence of the center line of the frame with the horizon line is calculated in the horizon compensation block according to the formula:

At the final stage of the method according to the present invention, the obtained data is simultaneously transmitted to the manipulator movement control unit with the camera attached to it for its movement inside the patient in order to display the surgical field. In this case, the manipulator with the camera carries out both translational movements to compensate for the movement of the camera and controllers, and rotation around its longitudinal axis to level the horizon line and the center line of the frame of the image obtained from the camera. The above steps of the proposed method, carried out using the automatic control system of the camera manipulator, are described in accordance with the block diagram 800 shown in FIG. 16.

At step (810), movement data of the surgeon's right and left controllers is obtained. In the next step (820), it is determined whether the signal is from the pedal. In the event that a signal is sent to the automatic control system from the pedal that switches the controllers to the camera control mode, a signal is initiated to start the operation of the automatic control system of the camera manipulator.

At step (830), the actual data received from the controller is sequentially sent to the calculating unit of derivatives for the right and left controllers, the increment of which is the total movement.

At the next stage (840), this movement is necessarily scaled taking into account the distance of the endoscope from the zero point.

At step (850), the resulting increment is integrated with the position of the manipulator in its Cartesian coordinate system. This compensated plane of motion is translated into the spherical coordinates of the manipulator f, q, IΪ, on the basis of which, at step (860), the compensation value of the horizon line R _z ° ^{mp is} calculated. The resulting data set (f, q, R, R _z ° ^mp ) is transmitted to the camera manipulator (step 870). Then the steps are repeated.

The proposed method for controlling camera movement adds an intuitive interaction with the robotic surgical complex and increases the efficiency of the robotic surgery.

The principle of operation of the compensation of camera movements is based on the transition from one coordinate system to another, which will allow the surgeon to work in similar coordinate systems for both the master and the actuator. In turn, such work will allow the surgeon to control the camera almost immediately, since the control becomes intuitive. There is no need for the surgeon to acquire additional camera control skills, since the compensated camera movement accurately reproduces the position of the camera in accordance with the operator's control commands. The applied algorithms for the compensation of the horizon line and the scaling algorithms make it possible to reduce the average time of positioning the manipulator with the camera in the patient's body to observe the operating field and to reduce the number of movements of the surgeon's hands.

Information obtained as a result of experiments and confirming a decrease in the time of positioning the manipulator with the camera in the patient's body, which leads to a decrease in the time of surgical intervention and a decrease in the number of movements of the surgeon's hands, which increases the intuitiveness and efficiency of control of the camera manipulator

Laboratory tests were carried out using the camera manipulator shown in Figure 17. The use of the method for controlling camera movement during surgery according to the present invention reduces the number of movements for positioning the camera manipulator to display the operating area and reduce the time of surgery.

This result is achieved due to the complex application of the plane compensation algorithm for linear movements of the camera manipulator and the horizon compensation algorithm.

The operability of the plane compensation algorithm was tested by performing linear movements of the camera manipulator along the Cartesian X and Y axes in the coordinate system of the camera manipulator using the surgeon's controller. At the same time, the projection on the Z-axis of the distance from an arbitrary point of the object under study to the zero point of the manipulator was observed.

Figure 17 shows the initial position of the camera manipulator in spherical coordinates f = 0 °, Q = 0 °, R = 110mm, which corresponds to coordinates in the Cartesian coordinate system of the manipulator x = 0mm, y = 0mm, z = 110mm.

Figure 18 shows the position of the camera manipulator in the position achieved by moving the surgeon's controller in the Cartesian coordinate system from the point x = 0 mm, y = 0 MM, Z = 100mm To the point x = 50mm, y = 0 mm, z = 100mm. The plane compensation algorithm is forcibly disabled, which leads to the following coordinates in the spherical coordinate system of the camera manipulator f = 0 °, Q = 30 °, R = 110mm, in the Cartesian coordinate system of the manipulator x = 0mm, y = 55mm, z = 95mm. It can be seen that during the described movement there is a change in the z coordinate of the camera manipulator in its Cartesian coordinate system, which corresponds to a change in the projection of the distance onto the Z axis from the zero point of the manipulator to the object under study. Figure 19 shows frames of the sequential movement of the camera manipulator when the plane compensation algorithm is forcibly disabled. On all frames, the zero point and the trajectory of the end of the camera are indicated.

Figure 20 shows the position of the camera manipulator in the position achieved by moving the surgeon's controller in the Cartesian coordinate system from thin x = 0mm, y = 0 MM, Z = 100mm To point x = 50mm, = 0mm, z = 100mm. In this case, the plane compensation algorithm is turned on, which leads to the following coordinates in the spherical coordinate system of the camera manipulator f = 0 °, Q = 30 °, R = 127mm, in the Cartesian coordinate system of the manipulator x = 0mm, y = 65mm, z = 110mm.

It can be seen that the described movement does not change the z coordinate of the camera manipulator in its Cartesian coordinate system, which corresponds to the preservation of the projection length of the distance on the Z axis from the zero point of the manipulator to the object under study. Figure 21 shows frames of the sequential movement of the camera manipulator when the plane compensation algorithm is enabled. On all frames, the zero point and the trajectory of the end of the camera are indicated.

The operation of the horizon compensation algorithm was checked by transmitting the control action along the Cartesian X axis in the controller's coordinate system, while the initial position of the camera manipulator in its spherical coordinate system was nonzero and was equal to f = 0 °, Q = 20 °, R = 110mm. Figure 22 shows a sequence of images of the object under study, plotted on the projection of the trajectory of the camera manipulator. You can see that the line drawn on the object under study, parallel to the horizon line, in the initial position is parallel to the center line of the frame. When moving without using the horizon compensation algorithm, the line rotates around the R axis of the camera manipulator in its spherical coordinate system, depending on the angle f.

The use of the horizon compensation algorithm allows maintaining the parallelism of the horizon line and the center line of the frame due to the automatic rotation of the camera around the R axis in the spherical coordinate system of the camera manipulator. Figure 23 shows a sequence of frames plotted on the trajectory of the camera manipulator. The initial position of the camera manipulator in the spherical coordinate system was f = 0 °, Q = 20 °, R = 110mm, the control action was carried out along the Cartesian X-axis in the controller's coordinate system. It is shown that the line drawn on the object under study, parallel to the horizon line, remains parallel to the center line of the frame at any part of the trajectory, which is achieved by calculating the required rotation angle by the compensation algorithm.

The study of efficiency was carried out by measuring the time, the number of movements with multiple changes in the field of view in the camera positioning mode. The study involved surgeons with experience working on a robotic-assisted complex.

Several control points were plotted on the object of the operating field; the operator-surgeon was tasked with moving the camera between them in a predetermined sequence, which simulates the process of moving the camera during the robotic-assisted operation. The image of the object of the operating field and the image of the frame of the first control point are shown in figure 24.

Two series of measurements were carried out: without using the algorithms for compensation of the plane and horizon, and with their use. The results of the series of experiments after statistical processing are shown in Figures 25 and 26.

It is shown that the use of the developed methods and compensation algorithms allows one to reduce the average positioning time by a factor of 1.5 and reduce the number of hand movements by a factor of 1.3, confirming the previously presented theses about an increase in the intuitiveness and efficiency of control of the camera manipulator.

Although the invention has been described with reference to the disclosed embodiments, it should be apparent to those skilled in the art that the specific experiments described in detail are provided for the purpose of illustrating the present invention only and should not be construed as in any way limiting the scope of the invention. It should be understood that various modifications are possible without departing from the spirit of the present invention.

In addition, while the present patent application relates to the subject matter defined in the appended claims below, it is important to note that this patent application contains the basis for the formulation of other inventions, which may, for example, be claimed as the subject of the amended claims of the present application or as the subject of the claims in a divisional and / or continuing application. Such an object can be characterized by any feature or combination of features described herein.

Claims

CLAIM 1. A method for controlling the movement of a camera mounted on a manipulator of a robotic surgical complex, which includes two controllers for controlling the robotic surgical complex, each of which is designed to digitize the movements of the surgeon's hands and provides the transmission of motion vectors along three translational and three rotational degrees of freedom, representing the difference between the coordinates of the control controller in the initial position and coordinates of the control controller when changing the position of the surgeon's hand, with each control controller being configured to switch to the camera control mode using the control pedal; at least one manipulator with a camera attached to it for viewing the operating field, ensuring its movement in three translational degrees of freedom; an automatic control system that is associated with both controllers and at least one manipulator; the method is characterized by the following steps: transfer of motion vector data over three translational and three rotational degrees of freedom of the right and left control controller to the automatic control system with a constant frequency, switching controllers to the control mode of the camera attached to the manipulator by pressing the control pedal in this case, the signal from the control pedal is transmitted to the automatic control system and to the control controllers to block the motion vectors of the rotational degrees of freedom at the latter, and after blocking the rotational degrees of freedom of the control controller, the vector of rotational coordinates in the automatic control system is reset to zero; control of the movement of the manipulator with the camera attached to it by means of the simultaneous movement of the controllers and the subsequent execution of the following steps: the stage of storing the obtained vectors of motion of the controllers in the automatic control system; the stage of processing and combining the motion vectors of the right and left controllers, characterizing the translational movements, to find the total displacement; the stage of scaling the displacement vector obtained at the previous stage; a motion compensation stage, comprising summing the scaled displacement vector and the vector of the current position of the specified manipulator in its local Cartesian coordinate system, and translating the obtained vector of the manipulator's position from Cartesian to spherical coordinate system; the step of compensating for the deviation of the horizon line from the center line of the frame by calculating the value of the divergence of the angle depending on the angles of the camera in the spherical coordinate system; the stage of simultaneous transmission of the data obtained at the previous steps to the actuators of the manipulator for its translational movement and rotation around the longitudinal axis of the camera fixed on it. 2. The method according to and. 1, characterized in that the scaling of the displacement vector is carried out taking into account the distance of the camera from the zero point of the camera in the local coordinate system of the camera, relative to which the position vector of the camera changes with the origin at the point of entry of the camera into the hole in the patient's body and with the end coinciding with the actual end position of the camera.