TWI903639B - Medical system and medical method thereof - Google Patents

Abstract

A medical system and medical method thereof. The medical method includes following steps: obtaining, by a sensing data acquisition sub-module of a patient end device, sensing data between a robotic arm of the patient end device and a patient body surface through a sensor; using, by a robotic arm adjustment sub-module of the patient end device, the sensing data to adjust the robotic arm; and using, by a simulated force feedback calculation sub-module of the patient end device, the sensing data to obtain a force feedback value, and using, by the simulated force feedback calculation sub-module, the force feedback value to trigger a force feedback manual controller of a physician device.

Description

Medical system and its medical methods

This invention relates to a medical system and its medical methods.

Currently, when doctors perform remotely controlled ultrasound examinations, they must rely on a patient-end camera to provide real-time images to confirm the position of the ultrasound probe on the patient's body surface. Doctors also need to adjust the ultrasound probe using a force-feedback manual controller. Since the patient-end camera typically only provides images of the patient from a specific angle, doctors must switch between the patient image display device and the patient ultrasound image display device to confirm the ultrasound probe's position on the patient's body surface.

In addition, since the doctor does not personally hold the ultrasound probe, the doctor lacks the feeling of pressing it on the patient's body surface in person.

The medical system of this invention includes a patient-side device and a physician-side device. The patient-side device includes a storage medium, a processor, a probe holder, and a robotic arm. The storage medium stores multiple sub-modules, including a sensor data acquisition sub-module, a robotic arm adjustment sub-module, and a simulated force feedback calculation sub-module. The processor is coupled to the storage medium, the probe holder, and the robotic arm, and accesses and executes the multiple sub-modules. The probe holder includes sensors. The physician-side device is communicatively connected to the patient-side device and includes a force feedback manual controller. The sensing data acquisition sub-module obtains sensing data between the robotic arm and the patient's body surface through sensors; the robotic arm adjustment sub-module uses the sensing data to adjust the robotic arm; the simulated force feedback calculation sub-module uses the sensing data to obtain the force feedback value, and the simulated force feedback calculation sub-module uses the force feedback value to trigger the force feedback manual controller.

The medical method of this invention includes the following steps: a sensing data acquisition sub-module obtains sensing data between the robotic arm and the patient's body surface through sensors; a robotic arm adjustment sub-module adjusts the robotic arm using the sensing data; and a simulated force feedback calculation sub-module obtains a force feedback value using the sensing data, and the simulated force feedback calculation sub-module triggers a force feedback manual controller using the force feedback value.

Based on the above, the medical system and method of this invention can adjust the robotic arm using the sensory data obtained between the robotic arm and the patient's body surface. This eliminates the need for physicians to repeatedly check between the patient imaging display and the patient ultrasound imaging display to confirm the position of the ultrasound probe on the patient's body surface. Furthermore, the medical system and method of this invention can also use the sensory data to obtain force feedback values and use these values to trigger a manual force feedback controller on the physician's end device. Based on this, the physician can obtain a realistic sensation of pressing on the patient's body surface.

Figure 1 is a schematic diagram of a medical system 1 according to an embodiment of the present invention. Referring to Figure 1, the medical system 1 may include a patient-side device 10 and a physician-side device 30. The patient-side device 10 may include a storage medium 11, a processor 12, a probe holder 13, and a robotic arm 14. The physician-side device 30 can be communicatively connected to the patient-side device 10 (via a network 20). In one embodiment, the patient-side device 10 and the physician-side device 30 may be located in different locations. In other words, the physician cannot directly contact the patient.

Storage medium 11 may be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or similar components or combinations thereof, used to store multiple modules or various applications that can be executed by processor 12. In this embodiment, storage medium 11 may store multiple sub-modules, and the multiple sub-modules may include a sensing data acquisition sub-module 111, a robotic arm adjustment sub-module 112, and a simulation power feedback calculation sub-module 113.

Processor 12 may be, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microcontroller (MCU), microprocessor, digital signal processor (DSP), programmable controller, application-specific integrated circuit (ASIC), graphics processing unit (GPU), image signal processor (ISP), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field programmable gate array (FPGA), or other similar components or combinations thereof. The processor 12 can be coupled to the storage medium 11, the probe holder 13, and the robotic arm 14 and access and execute the multiple sub-modules.

The probe holder 13 may include a sensor 131. In one embodiment, the probe holder 13 may be used to secure an ultrasound probe. In one embodiment, the sensor 131 may be an IoT (Internet of Things) sensing element such as a Time of Flight (ToF), a Load Cell, and/or a Limit Switch; however, the invention is not limited thereto.

The physician-side device 30 may include a force-feedback manual controller 31. The force-feedback manual controller 31 may be, for example, a 3D joystick. In one embodiment, the physician-side device 30 may include an input device 32. Furthermore, the physician-side device 30 may also include a patient image display device 33 and a patient ultrasound image display device 34.

The patient-side camera 40 can be connected to the doctor-side device 30 via network 20.

Figure 2 is a flowchart illustrating a medical method according to an embodiment of the present invention, wherein the medical method may be implemented by the medical system 1 shown in Figure 1. Please refer to both Figure 1 and Figure 2.

In step S210, the sensing data acquisition sub-module 111 can obtain sensing data between the robotic arm 14 and the patient's body surface through the sensor 131.

In step S230, the robotic arm adjustment sub-module 112 can use sensor data to adjust the robotic arm 14.

In step S250, the simulated force feedback operator module 113 can obtain the force feedback value using the sensing data, and the simulated force feedback operator module 113 can use the force feedback value to trigger the force feedback manual controller 31.

It is worth noting that although the patient-side camera 40 typically only provides patient images from a specific angle, this invention eliminates the need for physicians to repeatedly check between the patient image display device 33 and the patient ultrasound image display device 34 to confirm the position of the ultrasound probe on the patient's body surface, and also eliminates the need for physicians to frequently adjust the force feedback manual controller 31 to move the probe holder 13. In particular, this invention allows physicians to focus on interpreting the ultrasound images displayed on the patient ultrasound image display device 34, and provides a realistic sensation of pressing against the patient's body surface. The following will further illustrate implementation examples of steps S210, S230, and S250.

Figure 3 is an embodiment of steps S210 and S230 shown in Figure 2. Please refer to Figures 1, 2, and 3 simultaneously. As shown in Figure 3, sensor 131 may include a first sensor 131-1, a second sensor 131-2, a third sensor 131-3, and a fourth sensor 131-4. In this embodiment, the first sensor 131-1, the second sensor 131-2, the third sensor 131-3, and the fourth sensor 131-4 may be respectively disposed at the front, rear, left, and right corners/endpoints of the probe holder 13. In one embodiment, the first sensor 131-1, the second sensor 131-2, the third sensor 131-3, and the fourth sensor 131-4 may be, for example, distance sensing (Time of Flight) and/or angle sensing (Time of Flight). It should be noted that the present invention does not limit the number of sensors 131.

Please continue to refer to Figure 3. In this embodiment, the sensing data may include the difference in sensing distance between the robotic arm 14 and the patient's body surface, and the sensing data may include the sensing horizontal distance between the robotic arm 14 and the patient's body surface. In other words, in step S210 shown in Figure 3, the sensing data acquisition submodule 111 can obtain the sensing distance difference between the robotic arm 14 and the patient's body surface through the first sensor 131-1, the second sensor 131-2, the third sensor 131-3, and the fourth sensor 131-4, and the sensing data acquisition submodule 111 can obtain the sensing horizontal distance between the robotic arm 14 and the patient's body surface through the first sensor 131-1, the second sensor 131-2, the third sensor 131-3, and the fourth sensor 131-4. Next, in step S231, the robotic arm adjustment submodule 112 can calculate the vertical angle using the sensing distance difference and the sensing horizontal distance. Then, the robotic arm adjustment submodule 112 can adjust the robotic arm 14 using the vertical angle. Specifically, in step S232, the robotic arm adjustment submodule 112 can adjust the robotic arm 14 so that the vertical angle is 0. This allows the robotic arm 14 to be as perpendicular as possible to the patient's body surface, so that the physician can obtain a clearer ultrasound image through the patient ultrasound image display device 34.

Figure 4 is an embodiment of step S230 shown in Figure 2. Please refer to Figures 1, 2, and 4 simultaneously. In this embodiment, the robotic arm 14 can correspond to a robotic arm posture (i.e., Rx, Ry, Rz). The robotic arm adjustment submodule 112 can perform a translation operation to translate the movement reference point from the robotic arm 14 to the probe holder 13. Then, the robotic arm adjustment submodule 112 can use the robotic arm posture, 3D rotation coordinate transformation operation, and vector mapping operation to calculate the movement distance and movement components corresponding to the probe holder 13. In other words, the robotic arm adjustment submodule 112 can convert the sensing data (e.g., the aforementioned sensing distance difference) into coordinate transformation to calculate the amount of movement/rotation required for the robotic arm 14. In particular, since the direction and coordinate system of the force feedback manual controller 31 are different from those of the robotic arm 14, in order to make the doctor feel that the movements of the robotic arm 14 are consistent with those of the force feedback manual controller 31, the medical system 1 of this invention can perform 3D rotation coordinate transformation operations and vector mapping operations.

Figure 5 is an example of an implementation of steps S210 and S250 shown in Figure 2. Figures 6, 7, and 8 are further explanations of Figure 5. Please refer to Figures 1, 2, 5, 6, 7, and 8 simultaneously. In this embodiment, the robotic arm 14 may correspond to a robotic arm posture, and the probe holder 13 may correspond to probe coordinates. Sensing data may include the robotic arm posture and probe coordinates, and the sensing data may include the pressure value and compression deformation of the robotic arm 14 pressing down on the patient's body surface.

Please continue referring to Figure 5. In step S210, the sensing data acquisition submodule 111 can obtain sensing data between the robotic arm 14 and the patient's body surface through sensor 131. In step S251, the simulated force feedback calculation submodule 113 can use the sensing data to build a model of the patient's body surface. Next, in step S252, the simulated force feedback calculation submodule 113 can use the patient's body surface model to build a virtual surface of the force feedback manual controller corresponding to the force feedback manual controller 31.

In detail, as shown in Figure 6, assuming that the projection points of the first sensor 131-1, the second sensor 131-2, the third sensor 131-3, and the fourth sensor 131-4 on the patient's body surface are points A, B, C, and D, respectively, and assuming that the projection point of the probe fixation device 13 on the patient's body surface is point E, the dynamic feedback calculation operator module 113 can use the sensing distance difference, probe coordinates, and robotic arm posture to calculate the coordinates of points A, B, C, D, and E to establish a model of the patient's body surface. Furthermore, since axes _L1 and _L2 may not intersect in 3D space, the simulated force feedback operator module 113 can divide the patient's body surface into four sub-planes to form a tetrahedral patient body surface model. Then, the simulated force feedback operator module 113 can simulate/project the patient's body surface model into a force feedback manual controller virtual surface at the force feedback manual controller 31 end.

In one embodiment, the patient's body surface may include a soft surface, wherein the thickness of the soft surface may indicate a distance tolerance value, and wherein the force feedback value may be associated with the distance tolerance value.

Please refer to Figure 5. In step S253, the simulated force feedback operator module 113 can determine whether the force feedback manual controller 31 has triggered force feedback. If the simulated force feedback operator module 113 determines that the force feedback manual controller 31 has triggered force feedback (the determination result of step S253 is "yes"), then in step S254, the simulated force feedback operator module 113 can obtain the force feedback value. Then, in step S255, the simulated force feedback operator module 113 can use the force feedback value to trigger the force feedback manual controller 31.

More specifically, as shown in Figures 7 and 8, assuming the simulated force feedback operator module 113 has established a virtual surface 60 for the force feedback manual controller, the simulated force feedback operator module 113 can obtain the force feedback value according to the nonlinear equation 700. It is worth noting that since the patient's body surface is not a hard steel surface, when the distance that the force feedback manual controller 31 presses down on the virtual surface 60 of the force feedback manual controller (i.e., the downward deformation) has not exceeded the distance tolerance value, the force feedback value obtained by the simulated force feedback operator module 113 will be 0. In other words, the force feedback manual controller 31 can press down a further distance beyond the tolerance value onto the patient's body surface (the virtual surface 60 of the force feedback manual controller) without the doctor feeling any force feedback through the force feedback manual controller 31. On the other hand, when the distance pressed down by the force feedback manual controller 31 onto the virtual surface 60 of the force feedback manual controller exceeds the distance tolerance value, the simulated force feedback calculation module 113 can obtain the force feedback value according to the nonlinear equation 700.

In one embodiment, the input device 32 can receive adjustment operations to adjust the distance tolerance value. Specifically, since different parts of a patient's body have different levels of flexibility, the physician can operate the input device 32 to adjust the distance tolerance value. Accordingly, when the physician operates the physician-side device 30 to observe different parts of the patient's body, the simulated force feedback operator module 113 will obtain a force feedback value based on different distance tolerance values (also known as "force feedback sensitivity").

In other embodiments, in addition to the distance tolerance value, the force feedback value may also be related to the following parameters:

(1) Slope: The force feedback value is the ratio of the force value from the patient's body surface to the distance from the hard surface.

(2) Error: The above-mentioned error in sensing distance and/or sensing horizontal distance.

(3) Ratio: The ratio of the sensing distance difference (mm) to the force feedback value (g).

Similarly, the physician can operate the input device 32 to adjust (1), (2) and (3) above so that the force feedback value is closer to the actual softness of the patient's body.

In summary, the medical system and method of this invention can adjust the robotic arm based on the sensor data obtained between the robotic arm and the patient's body surface. This eliminates the need for physicians to repeatedly check between the patient imaging display and the patient ultrasound imaging display to confirm the position of the ultrasound probe on the patient's body surface. Furthermore, the medical system and method of this invention can also use the sensor data to obtain force feedback values and use these values to trigger a manual force feedback controller on the physician's end device. Based on this, the physician can obtain a realistic sensation of pressing on the patient's body surface.

1: Medical System 10: Patient Terminal Device 11: Storage Media 111: Sensor Data Acquisition Submodule 112: Robotic Arm Adjustment Submodule 113: Simulated Force Feedback Calculation Submodule 12: Processor 13: Probe Fixer 131: Sensor 14: Robotic Arm 20: Network 30: Physician Terminal Device 31: Force Feedback Manual Controller 32: Input Device 33: Patient Image Display device 34: Patient ultrasound image display device 40: Patient-end camera S210, S230, S250, S231, S232, S251, S252, S253, S254, S255: Step 131-1: First sensor 131-2: Second sensor 131-3: Third sensor 131-4: Fourth sensor x, y, z: Robotic arm coordinates Rx, Ry, Rz: Robotic arm posture A, B, C, D, E: Points _L1 , _L2 : Axis 60: Force feedback manual controller Virtual surface 700: Nonlinear equation

Figure 1 is a schematic diagram of a medical system according to an embodiment of the present invention. Figure 2 is a flowchart of a medical method according to an embodiment of the present invention. Figure 3 is an example of an embodiment of steps S210 and S230 shown in Figure 2. Figure 4 is an example of an embodiment of step S230 shown in Figure 2. Figure 5 is an example of an embodiment of steps S210 and S250 shown in Figure 2. Figures 6, 7, and 8 are further explanations of Figure 5.

S210, S230, S250: Steps

Claims (12)

A medical system includes: a patient-side device, including a storage medium, a processor, a probe holder, and a robotic arm, wherein the storage medium stores multiple sub-modules, and the multiple sub-modules include a sensing data acquisition sub-module, a robotic arm adjustment sub-module, and a simulated force feedback calculation sub-module, wherein the processor is coupled to the storage medium, the probe holder, and the robotic arm and accesses and executes the multiple sub-modules, wherein the probe holder includes a sensor; and a physician-side device, communicatively connected to the patient-side device, wherein the physician-side device includes a force feedback manual controller, wherein the sensing data acquisition sub-module acquires sensing data between the robotic arm and the patient's body surface through the sensor; The robotic arm adjustment submodule uses the sensed data to adjust the robotic arm; The simulated force feedback calculation submodule uses the sensed data to obtain a force feedback value, and the simulated force feedback calculation submodule uses the force feedback value to trigger the force feedback manual controller, wherein the robotic arm corresponds to a robotic arm posture, and the robotic arm adjustment submodule performs a translation operation to move the reference point from the robotic arm to the probe holder; the robotic arm adjustment submodule uses the robotic arm posture, 3D rotation coordinate transformation operation, and vector mapping operation to calculate the movement distance and movement component corresponding to the probe holder. The medical system of claim 1, wherein the sensing data includes the sensing distance difference between the robotic arm and the patient's body surface, and the sensing data includes the sensing horizontal distance between the robotic arm and the patient's body surface, wherein the robotic arm adjustment submodule uses the sensing distance difference and the sensing horizontal distance to calculate a vertical angle; the robotic arm adjustment submodule uses the vertical angle to adjust the robotic arm. The medical system as claimed in claim 1, wherein the probe fixator corresponds to the probe coordinates, wherein the sensing data includes the robotic arm posture and the probe coordinates, and the sensing data includes the pressure value and the amount of deformation of the robotic arm pressing down on the patient's body surface. The medical system as described in claim 1, wherein: the simulated force feedback operator module uses the sensed data to establish a patient's body surface model; the simulated force feedback operator module uses the patient's body surface model to establish a virtual surface corresponding to the force feedback manual controller. The medical system as claimed in claim 1, wherein the patient's body surface includes a soft surface, wherein the thickness of the soft surface indicates a distance tolerance value, and wherein the force feedback value is associated with the distance tolerance value. The medical system as described in claim 5, wherein the physician-side device further includes an input device, wherein the input device receives adjustment operations to adjust the distance tolerance value. A medical method is suitable for a medical system including a patient-end device and a physician-end device, wherein the patient-end device includes a storage medium, a processor, a probe holder, and a robotic arm, wherein the storage medium stores multiple sub-modules, and the multiple sub-modules include a sensing data acquisition sub-module, a robotic arm adjustment sub-module, and a simulated force feedback calculation sub-module, wherein the probe holder includes a sensor, and wherein the physician-end device includes a force feedback manual controller, wherein the medical method includes the following steps: The sensing data acquisition sub-module acquires sensing data between the robotic arm and the patient's body surface via the sensor; The robotic arm adjustment sub-module adjusts the robotic arm using the sensing data; and The simulated force feedback calculation submodule obtains a force feedback value using the sensed data, and triggers the force feedback manual controller using the force feedback value. The robotic arm corresponds to a robotic arm posture. The steps by which the robotic arm adjustment submodule adjusts the robotic arm using the sensed data include: The robotic arm adjustment submodule performs a translation operation to move a reference point from the robotic arm to the probe holder; and The robotic arm adjustment submodule calculates the movement distance and movement components corresponding to the probe holder using the robotic arm posture, 3D rotation coordinate transformation, and vector mapping. The medical method of claim 7, wherein the sensing data includes the sensing distance difference between the robotic arm and the patient's body surface, and the sensing data includes the sensing horizontal distance between the robotic arm and the patient's body surface, wherein the step of the robotic arm adjustment submodule adjusting the robotic arm using the sensing data includes: The robotic arm adjustment submodule calculating a vertical angle using the sensing distance difference and the sensing horizontal distance; and The robotic arm adjustment submodule adjusting the robotic arm using the vertical angle. The medical method as described in claim 7, wherein the probe fixator corresponds to the probe coordinates, wherein the sensing data includes the robotic arm posture and the probe coordinates, and the sensing data includes the pressure value and the amount of deformation of the robotic arm pressing down on the patient's body surface. The medical method as described in claim 7, wherein the step of obtaining a force feedback value by the simulated force feedback operator module using the sensed data, and triggering the force feedback manual controller by the simulated force feedback operator module using the force feedback value, comprises: establishing a patient's body surface model by the simulated force feedback operator module using the sensed data; and establishing a virtual surface for the force feedback manual controller corresponding to the force feedback manual controller by the simulated force feedback operator module using the patient's body surface model. The medical method as described in claim 7, wherein the patient's body surface includes a soft surface, wherein the thickness of the soft surface indicates a distance tolerance value, and wherein the force feedback value is associated with the distance tolerance value. The medical method as described in claim 11, wherein the physician-side device further includes an input device, and wherein the medical method further includes the following steps: receiving an adjustment operation by the input device to adjust the distance tolerance value.