HK1237483A1 - Remote presence system including a cart that supports a robot face and an overhead camera - Google Patents

Description

Telepresence system including cart supporting robot face and overhead camera

This application is a divisional application of patent applications entitled "telepresence system including a cart supporting a robot face and an overhead camera", filed 3/2011, 1, and filed 201180012056.3 (international application number PCT/US 2011/026599).

Technical Field

The disclosed subject matter relates generally to the field of robotic telepresence (tele-presence) systems.

Background

Robots have been used in a variety of applications ranging from remote control of hazardous materials to assisting in the performance of surgical procedures. For example, U.S. Pat. No.5,762,458 issued to Wang et al discloses a system that allows a surgeon to perform minimally invasive medical procedures (medical procedures) through the use of robotically controlled instruments. One of the robotic arms in the Wang system moves an endoscope with a camera. The camera allows the surgeon to view the surgical field (surgical area) of the patient.

Mobile telepresence robots under the trademark RP-7, introduced by InTouch technologies, inc. The InTouch robot is controlled by a user at a remote station. The remote station may be a personal computer with a joystick that allows the user to remotely control the movement of the robot. Both the robot and the remote station have cameras, monitors, speakers, and microphones to allow for two-way video/audio communication. The robot camera provides video images to a screen at the remote station so that the user can view the robot's surroundings and move the robot accordingly.

InTouch also provides a system sold as VisitOR that includes a robot face attached to a boom. The boom and robot face may be installed into an operating room. Use of a robot face in an operating room may require disinfection of the face. Additionally, VisitOR requires the installation of a boom arm in the operating room. This may increase the cost and complexity of installing such a system.

Drawings

FIG. 1 is an illustration of a telepresence system;

fig. 2 is a perspective view of the cart (cart) of the system;

FIG. 3 is a rear view of the articulated arm and robot face of the cart;

FIG. 4 is an enlarged perspective view of an overhead camera (overhead camera) of the cart;

FIG. 5 is an illustration of a display user interface of a remote station;

fig. 6 is a diagram showing a user interface showing video images captured by the robot camera and video images captured by the overhead camera being displayed simultaneously.

Detailed Description

Disclosed is a telepresence system that includes a cart. The cart includes a robot face having a robot monitor, a robot camera, a robot speaker, a robot microphone, and an overhead camera. The system also includes a remote station coupled to the robot face and the overhead camera. The remote station includes a station monitor, a station camera, a station speaker, and a station microphone. The remote station may display video images captured by the robot camera and/or overhead camera. For example, the cart may be used in an operating room where an overhead camera may be placed over a sterile field (steriefield) to provide a more beneficial vantage point for viewing the procedure. A user at a remote station may conduct a teleconference through the robot face and may also obtain a view of the medical procedure through the overhead camera.

Referring more particularly to the drawings by reference numbers, FIGS. 1, 2 and 3 illustrate a telepresence system 10. The system 10 includes a cart 12 coupled to a remote control station 14. The cart 12 has a robot face 16 and an overhead camera 18. The remote control station 14 may be coupled to the cart 12 through a network 20. The network 20 may be, for example, a packet-switched network such as the internet, or a circuit-switched network such as the Public Switched Telephone Network (PSTN), or other broadband system. Alternatively, the cart 12 may be coupled to the remote station 14 through a satellite.

The remote control station 14 may include a computer 22 having a monitor 24, a camera 26, a microphone 28, and a speaker 30. The computer 22 may also include an input device 32 such as a joystick or a mouse. The control station 14 is typically located remotely from the cart 12. Although only one remote control station 14 is shown, the system 10 may include a plurality of remote stations 14. In general, any number of carts 12 may be coupled to any number of remote stations 14 or other carts 12. For example, one remote station 14 may be coupled to a plurality of carts 12, or one cart 12 may be coupled to a plurality of remote stations 14 or a plurality of carts 12. The system may include an arbiter (not shown) that controls access between the cart 12 and the remote station 14.

As shown in fig. 3, the cart 12 may include an articulated arm 40 that supports and may move the robot face 16. The articulated arm 40 may have active joints 42 and 44 that allow the robot face 14 to be panned (pan) and tilted (activejoint), respectively. The active joints 42 and 44 may move in response to commands provided by the remote station. Joints 42 and 44 may include position sensors 46 and 48, respectively, that provide position feedback of arm 40.

Referring to fig. 2 and 3, each robot face 16 includes camera(s) 50, monitor 52, microphone(s) 54, and speaker(s) 56. The robot camera 50 is coupled to the remote monitor 24 so that a user at the remote station 14 can view video images captured by the robot camera 50. Likewise, a robot monitor 52 is coupled to the remote camera 26 so that personnel at the surgical site can view the user of the remote station 14. Both microphones 28 and 54 and speakers 30 and 56 allow for audible communication between the system operator and personnel at the surgical site.

The overhead camera 18 may be coupled to a boom 60. Boom 60 may include a plurality of (active or passive) joints 62. The joints 62 may include position sensors to provide feedback regarding the position of the overhead camera 18.

As shown in fig. 4, the cart 12 may include a boom microphone 64 and a detachable handle 66. The overhead microphone 64 can provide an alternative sound source. The detachable handle 66 can be used to move the boom 60 and overhead camera 18. If the cart 12 is used in a sterile field, such as in an operating room, the handle 66 may be replaced with a sterile handle prior to each medical procedure to allow a surgeon in the sterile field to position the boom during the procedure.

Referring again to fig. 2, the cart 12 may include a linear drive 70, which linear drive 70 may be remotely or locally actuated to vary the height of the robot face 16 and overhead camera 18. Varying the height allows the cart 12 to be rolled through a door and then actuated to move the face 16 and camera 18 to an elevated position. For example, the face 16 and camera 18 may be lowered to allow the cart 12 to be moved into the operating room. The camera 18 may then be raised to provide the desired view of the operating table. The cart 12 may include a laser pointer 72 and/or directional lighting (not shown) on the boom 60. The cart 12 may also include a local control panel 74 to move the articulated arm 40, the drive 70, and/or the boom 60. The linear drive 70 is also advantageous in moving the face 16 to be at substantially the same level as a person whether the person is standing, sitting or lying in a prone position.

The robot face 16 may include a processor, hard drive, and other circuitry that enables the face 16 to function as a computer. The face 16 may include an input panel 76 that allows the user to provide input. For example, an operator of the remote station may provide one or more questions through the robot face 16, with the user of the cart providing answers through the input panel 76.

The system 10 may have the same or similar specific components and software as the robotic system known as RP-7, provided by InTouch technologies, assignee of goratata, california, and embodied in the system described in U.S. patent 6,925,357, incorporated by reference.

FIG. 5 illustrates a display user interface ("DUI") 120 that may be displayed at the remote station 14. The DUI120 may include a robot viewing area 122 that displays video images captured by the robot camera and/or overhead camera. The DUI120 may also include a station viewing area 124 that displays video images provided by the camera of the remote station 14. The DUI120 may be part of an application program stored and operated by the computer 22 of the remote station 14.

The DUI120 may include a graphical switch 126 that allows a user to select between video images provided by the robot camera and video images provided by the overhead camera. The DUI120 may also have a graphical switch 128 that allows the user to select to simultaneously display video images from the robot and overhead camera as shown in fig. 6. Video images from both cameras can be streamed from the cart to the remote station. The images may be merged by presenting a central rectangle (e.g., 320 x 480 central region) for each image. By manipulating a cursor on either image, a zoom or highlight feature may be utilized. The system may also automatically pan the camera lens when the cursor is moved out of the displayed field of view.

The system can automatically present video images from the cameras with the best view of the object. For example, the system may utilize pattern recognition techniques to determine which video image provides a clearer image of the object. The system may determine which camera is closer to the object and provide images from cameras closer to the object. The system may utilize position feedback from the cart to determine the proximity of the camera relative to the object. The system may also have sensors such as lasers, sonar, etc. that can determine the proximity of the camera to the object. The system may use feedback and/or sensors to determine which camera is closer to the object.

The system may automatically move the cameras so that each camera is directed to the same or substantially the same field of view. For example, if the robot face is pointed at an object, the overhead camera may be automatically moved to capture a video image of the same object. Likewise, if the overhead camera is capturing a video image of an object, the robot face may be automatically moved to point to the same object. This enhances the "presence" of remote operators, since these remote operators are facing the same object that the overhead camera is viewing.

The DUI120 may have a graphical switch 130 that allows the user to switch between sounds captured by the robot microphone or overhead microphone. The system may automatically switch between microphones based on the characteristic(s) of the sound captured by the microphones. For example, the system may switch to the microphone that provides the highest hearing clarity, or to the microphone that is closest to the person or object that generated the sound.

The DUI120 may include a positioning display 138 that provides a positioning of the robot face. The "change" button 140 may be selected to change the default robot face in a new session. The "change" button 140 may be used to select and control different robot faces in a system having multiple robot faces. The user may initiate and terminate a session through selection box 142. When the user selects this box to initiate a session, block 142 changes from "connect" to "disconnect". System settings and system support may be selected via buttons 144 and 146.

Both the robot viewing area 122 and the station viewing area 124 may have associated graphics to vary the visual display as well as the audio display. Each zone may have an associated graphical audio slider 148 to vary the audio level of a selected microphone and another slider 152 to vary the volume of a speaker.

The DUI120 may have slider bars 150, 154, and 156 to vary the zoom, focus, and brightness, respectively, of the selected camera. By selecting one of the graphical camera icons 158, a still picture can be taken at the robot face or at the remote station. The still picture may be an image that is presented at the corresponding region 122 or 124 when the camera icon 158 is selected. Capturing and playing video may be performed through the graphical icon 160. Returning to real-time video may be resumed after taking a still picture, captured video, or reviewing a slide show by selecting the graphical "live" button 162.

Still pictures may be loaded from disk for viewing by selecting icon 164. The stored still images are reviewed by selecting button 166. The number of images displayed relative to the total number of images is shown by the graphical box 168. By moving the slider bar 170, the user can quickly pass through still images in a slideshow fashion or through captured video clips. The captured video image may be paused through the selection circle 174. Playback can be resumed through the same button 174. Video or still images may be removed from the active list via button 172. The video or still image may be transmitted to the robot by selecting icon 176. For example, a doctor at a remote station may transmit an x-ray picture to a screen of the robot.

The system may provide the ability to annotate 184 the image displayed in regions 122 and/or 124. For example, a doctor at the remote station may annotate some portion of an image captured by the robot face camera. The annotated image may be stored by the system. The system may also allow for annotation of the image sent to the robot face by icon 176. For example, a doctor may send an x-ray picture to the robot face, which is displayed by the robot screen. The doctor may annotate the radiograph to indicate the portion of the radiograph to personnel located at the robot site. This may assist in allowing the physician to indicate the person at the robot site.

The display user interface may include a graphical input 186 that allows the operator to turn the remote station and the view of the remote camera on and off.

Referring to fig. 1, the cart 12 may be used in an operating room. For example, boom 60 may be moved to place overhead camera 18 over operating table 200. Overhead camera 18 may be located over a sterile field. The robot face 16 may be placed adjacent to the sterile field. With such a configuration, a person can conduct a two-way video conference through the robot face 16. The overhead camera 18 can provide a more desirable view of the patient as well as the surgical procedure. This would allow a physician at the remote station to view the procedure and hold a video conference to provide instructions, guidance, etc. to personnel at the surgical site.

It will be obvious to those having skill in the art that many changes may be made to the details of the above-described embodiments without departing from the underlying principles of the invention. While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art. In the claims, the conjunction "and" is inclusive, the conjunction "or" is exclusive, and the conjunction "and/or" is inclusive or exclusive. The embodiments of the invention in which exclusive property or privilege is claimed are defined as follows.

Claims (10)

1. A telepresence system, comprising:

a cart including an overhead camera, a speaker, a microphone, and a robot face having a robot monitor and a robot camera; and the number of the first and second groups,

a remote station coupled to the robot face and the overhead camera, the remote station including a station monitor, a station camera, a station speaker, and a station microphone, wherein at least one of the robot camera and the overhead camera is controllable from the remote station.

2. The system of claim 1, wherein the remote station is capable of displaying video images from the robot camera or the overhead camera.

3. The system of claim 1, wherein the station monitor simultaneously displays video images from the robot camera and video images from the overhead camera.

4. The system of claim 1, wherein the cart comprises an overhead microphone.

5. The system of claim 4, wherein the remote station includes an input that allows a user to switch between the robot microphone and the overhead microphone.

6. The system of claim 1, wherein the cart comprises an articulated arm coupled to and capable of moving the robot face, and the cart comprises a boom coupled to the overhead camera.

7. The system of claim 6, wherein the robot face includes an input panel and is operable as a computer.

8. The system of claim 6, further comprising a laser pointer attached to the boom.

9. A telepresence system, comprising:

a cart comprising a speaker, a microphone, and a robot face having a robot monitor and a robot camera that move together in at least two degrees of freedom; and

a remote station coupled to the robot face and the overhead camera, the remote station including a station monitor, a station camera, a station speaker, a station microphone, and an input device for controlling movement of the robot face.

10. A remote vision system, comprising:

a cart comprising an overhead camera, a speaker, a microphone, a second camera, and a monitor, the second camera and the monitor moving together in at least two degrees of freedom; and

a remote station coupled to the overhead camera and the second camera, the remote station including a station monitor, a station camera, a station speaker, a station microphone, and an input device for controlling movement of the robot face.