WO2009036782A1

WO2009036782A1 - Information processing apparatus and method for remote technical assistance

Info

Publication number: WO2009036782A1
Application number: PCT/EP2007/008125
Authority: WO
Inventors: Franco Tecchia; Sandro Baccinelli; Marcello Carrozzino; Massimo Bergamasco
Original assignee: Vrmedia S.R.L.; Sidel Participations
Priority date: 2007-09-18
Filing date: 2007-09-18
Publication date: 2009-03-26
Also published as: WO2009112063A2; WO2009112063A9; WO2009112063A3

Abstract

A virtual community communication system where two or more technicians carry or access an Augmented Reality (AR) - enhanced apparatus to communicate and exchange, over a LAN or the Internet, information regarding assembly or servicing or maintenance operations performed on complex machinery. Data streams exchange between the peers of the virtual community is performed by means of a centralised server. Various arrangements are presented that can be selected based on the needs of the operation to be performed, such as the number of members of the community and the type of communication equipment. The system is applicable to any application of the virtual community communication system and is optimized for application to industrial machinery.

Description

TITLE

INFORMATION PROCESSING APPARATUS AND METHOD FOR REMOTE TECHNICAL ASSISTANCE

DESCRIPTION Field of the invention

The present invention relates to an information processing method for remote assistance during assembly or maintenance operations.

Moreover, the invention relates to an apparatus that carries out such method.

Description of the prior art

Present-day industrial machinery have a very complex design, and as a consequence maintenance or repairing operations need the intervention of a variety of specialists, often only available as members of the manufacturer's technical staff.

When machinery needs intervention, it is often required for the or each expert to travel to the site where the machinery is hosted. Travelling is expensive and takes a long time, both factors influencing the total cost of a servicing operation.

On the other hand, a large part of the servicing operations would not really require in-deep knowledge of a machine working principle or internal structure, and could be performed by an in-situ technician who should have at least some step-by-step basic notions on the manual tasks to be performed on the machine.

Several attempts have been made to put the in-situ a technician in touch with a remote expert using a variety of communication means, ranging from voice communication to more complex data transmission technology.

In particular, Augmented Reality (AR) has been used for man-machine interaction, since it presents a major potential for supporting industrial operational processes. It overlays computer-generated graphical information onto a physical (real) world, by means of a see-trough near-eye display controlled by a computer. The field of view of the observer is enriched with the computer-generated images .

For example EP1157314 discloses an AR system for transmitting first information data from a technician at a first location to a remote expert at a second location. A sensor system is provided for data acquisition at the technician' s site, and for evaluating them at the expert's site, assigning then real objects to stored object data, which are provided at the technician's site.

US2002010734 disclose an internetworked augmented reality (AR) system, which is mainly dedicated to entertainment and consists of one or more local stations and one or more remote stations networked together. The remote stations can provide resources not available at a local AR Station such as databases, high performance computing (HPC) , and methods by which a human can interact with the person (s) at the local station . The above known systems, cannot assure that a remote expert has a clear and updated view on the actions to do, since an exchange of a number of high- quality video streams is required.

Summary of the invention It is therefore a feature of the invention to provide an advanced communication system used to create a virtual team of technicians performing assembly or servicing operations of complex machinery in a collaborative way forming a virtual community of geographically distributed experts and technicians.

In particular, one or more of the participating members can be located at different geographical sites with respect to the machinery, providing remote help to the in-situ technicians.

It is another feature of the invention to specify a system and a method which, in concrete operational situations, permits an effective virtual co- participation of the remote experts to the in-situ technicians actions and decisions, in order to enhance their technical ability up to perform some of the servicing operation without the need of the physical presence of the expert. It is a further feature of the invention to provide an information processing apparatus and method for remote technical assistance in which resolution, frame rate, and network latency allow a practical, effective and fast operation. These and other features are accomplished with one exemplary method for remote assistance during assembly or maintenance operations, the method comprising the steps of: providing at least one technician at a first location and at least one expert at a second location, exchanging information data between said at least one technician and said at least one expert, through a set of communication channels, including audio, voice and interactive graphics, as well as 3D data, wherein the information data are selected among video images, graphics and speech signals of the technician and wherein additional information data in the form of augmented-reality information are transmitted from the remote expert at the second location to the in situ technician at the first location highlighting specific objects in the field of view of the technician, said expert being equipped with a computer and videoconferencing devices; said technician being equipped with a wearable computer having a WI-FI antenna associated to said wearable computer for data transmission; a headset connected to said computer including headphones, a noise-suppressing microphone, one near-eye see-trough AR display, and a miniature camera mounted on the display itself used to capture what is in the front of view of the technician; characterised in that said least one technician and said at least one expert are arranged respectively at an in-situ-node and at a remote node of a network, said nodes communicating and exchanging data through the internet via a centralised communication server and in that the data are streamed via a hardware-accelerated video compression means. According to another aspect of the invention, an apparatus for remote assistance during assembly or maintenance operations comprises: means for exchanging information data between at least one technician at a first location and at least one expert at a second location through a set of communication channels, including audio, voice and interactive graphics, as well as 3D data, wherein the information data are a collection of video images, graphics and speech signals of the technician and wherein additional information data in the form of augmented-reality information are transmitted from a remote expert at the second location to an in situ technician at the first location highlighting specific objects in the field of view of the technician, a computer and videoconferencing devices to be used by said expert; a unit to be used by said technician comprising a wearable computer having a wi-fi antenna associated to said wearable computer for data transmission; a headset connected to said computer including headphones, a noise-suppressing microphone, one near- eye see-trough AR display, and a miniature camera mounted on the display itself used to capture what is in the front of view of the technician; characterised in that means are provided for communicating and exchanging data between at least two nodes through the internet via a centralised communication server and hardware-accelerated video compression means for streaming said data.

Advantageously, a hand held camera is provided connected to said computer equipped with a light source for lighting desired targets.

Advantageously, additional automated remote computing nodes are provided to create additional video feeds, in particular auxiliary fixed cameras that are positioned by the technicians and that can be controlled by the remote experts for pan, zoom and tilt movements.

Preferably, the organisation of a multitude of in- situ technicians and remote experts is established in a distributed virtual community for the exchange of knowledge, wherein at least one on-situ technician at one node and at least one remote expert at another nodes are provided communicating and exchanging data with each other.

Therefore, a virtual community of skilled specialists is created where members communicate by means of internetworked computers and several input/output devices. The virtual community can therefore be conceptualised as a group of technicians each of them equipped with a computer (computing node) plus some automated remote computing node used to provide additional video feeds. Each computing node exchanges data over a wide-area communication network. Some of these nodes can share the same physical space while other can be located at multiple geographical locations .

Preferably, the use of Augmented Reality is provided to overlap special visual markers on the objects falling inside the field of view of the operator .

Advantageously, said headset can also be equipped with a 3DOF tracking system, used to measure rotational head moments of the skilled technician, in such a way to compensate such movements in terms of visual displacement of the computer generated graphical markers that are overlapped on the field of view of the technician.

Preferably, said video compression means comprise H.264 Compression Technology.

Advantageously, video streaming is associated to VoiceOverIP technology.

Preferably, the hardware-accelerated video compression means are arranged in such a way that the video streams and audio streams are compressed and combined, preferably 384 Kbit/s uplink and 384 Kbit/s down link.

Brief description of the drawings The invention will be now shown with the following description of an exemplary embodiment thereof, exemplifying but not limitative, with reference to the attached drawings, in which: figure 1 shows an architecture of a virtual community communication system for remote technical assistance; figure 2 shows a particular embodiment of an architecture of a virtual community communication system for remote technical assistance where the nodes are arranged as sub-communities according to affinity criteria; figure 3 shows the architecture of figure 1 where at the computing nodes in-situ technicians, remote experts and remotely controlled video-cameras looking at the machinery are indicated; figures 4 to 6 show an on-field technician equipped with a wearable computing system and a special head set integrating an Augmented Reality see-trough display; - figures 7 to 8 show an on-field technician equipped also with a hand held camera; figure 9 shows an an in-situ fixed node, composed by a camera mounted on a tripod; figure 10 shows a Graphic Technician Interface of the application running at a technician' s node where to a technician three different streaming video data are presented. figure 11 shows a block diagram of a preferred working unit of an apparatus according to the invention; - figure 12 shows a data communication scheme applied to the architecture of the virtual community communication system for remote technical assistance of figure 1 using the preferred working units of figure 11; - figure 13 shows a data communication scheme applied to a different embodiment of a virtual community communication system for remote technical assistance, using a peer to peer architecture, and using the preferred working units of figure 11.

Description of the preferred embodiments

With reference to figure 1, an information processing apparatus and method are provided to establish a virtual community of geographically distributed experts and technicians for remote assistance during assembly or servicing operations of complex devices. The technician (s) and the expert (s) are arranged at nodes 1-N. Nodes 1-N communicate with one another and exchange data through the internet via a centralised communication server 8.

In addition, the centralised Communication Server 8 is used for monitoring the data, checking the data traffic, controlling the access rights and storing usage statistics.

In figure 2, the capability of the system is shown to group technicians in sub-communities which can be created according to various criteria, such as affinity in terms of servicing scenario, physical contiguity etc.

In particular, the presence of a centralised server allows for a dynamic management of how the technicians are grouped in sub-teams. In fact, multiple virtual teams composed by some in-situ technicians, some automated cameras and some remote experts can operate at the same time on multiple locations. Members of one team can be dynamically be allocated to another team even for a limited amount of time: this maximises the possibility that experts with specific know-how can quickly be contacted and involved in the assembly/servicing operation. In addition, in-situ technicians in a particular operation can quickly be transformed in remote experts for another particular operation, changing their roles amongst the teams. This dynamical architecture assures that even the skills and knowledge of the highly trained technicians is at disposal of the collectivity.

An example of remote technical assistance through the invention is shown in figure 3, where a network is illustrated managed by centralised communication server 8. Industrial machinery 11, for example large machinery located in an industrial plant, has to be serviced or assembled or inspected by technicians 9 and by auxiliary fixed video cameras 10. The experts 9 advise the technicians on how to operate.

In particular, the architecture is shown of the virtual community communication system, where the computing nodes, such as one or more remote nodes where experts 12 are present, one or more in-situ mobile nodes where a technician 9 is present, and fixed nodes 10 with remotely controllable video cameras, communicate and exchange data through the internet via centralised communication server 8.

In-situ technicians 9 use wearable equipment and move freely around the machinery 11. One or more among auxiliary remotely controlled video-cameras 10 can also be placed around the machinery 11 to provide extra video streams of the operations being performed by the technicians. Pan, zoom and tilt of these auxiliary cameras 10 can be controlled by the remote experts 12, who can adjust them in order to obtain desired images of the machine. Remote experts 12 are connected to the internet from one or more remote locations and are equipped with standard laptop computers 14 and videoconferencing devices, such as voice communication headphones 13.

The remote experts 12 receive and examine all the information coming from the technicians 9 and the cameras 10 and can consequently send back manipulation instructions by means of voice or by remotely controlling the display of special dynamic graphical markers (described hereafter with reference to figure

10) that appear on the field of view of the in-situ technicians by means of the Augmented Reality display.

With reference to figures 4-6, a on-field technician a wearable computing system 1 and a special head set 4 integrating an Augmented Reality see-trough display. The wearable AR-based apparatus is composed of a backpack 3 containing a portable computer and a helmet 4 where a video camera 5, headphones 6 with a microphone 6A and a see-through display 7 are mounted.

With reference to figures 7 and 8 an in-situ technician 12 wearing the AR-based apparatus 1 can hold an additional hand-held camera 2, having a lighting system, preferably with white LEDs, connected to the computer and that can be used to show to the remote experts 12 portions of the real scene that would be impractical to show using the video camera mounted on the headset or the fixed video cameras.

With reference to figure 9, and as previously indicated in figure 4, additionally to the computing nodes associated with in-situ technicians and remote experts, a third additional kind of computing node can be inserted in the community, comprising a remote controlled high-quality video-camera 10. It is mounted on a tripod 15 that can be placed around the machinery 11 (see fig. 4) to provide additional view-points on the operations. For example, in figure 9 instead of machinery a computer station 30 is shown, for example for remotely instructing technicians on how to assemble or service the station, or for training purposes. Each camera 10 is equipped with motorised Pan, Zoom and Tilt support that can be controlled by the remote experts 12. The camera 10 can either be a stand-alone network camera, equipped with video compression and network streaming capabilities, or a device connected to a computer 20, capable of acquiring, compressing and transmitting video data over the network and to the centralized communication server.

Figure 10 shows what the remote expert sees on the screen of its laptop, as seen by fixed camera 10 of figure 9, as well as by the micro camera on the headset or the hand held camera, and what kind of visual feedback can produce that will be overlapped on the field of view of the in-situ technician. Figure 10 can be the Graphic Technician Interface of the application running at the expert site. Figure 10 can be however also the Graphic Technician Interface of the application running at the technicians site.

To the expert the following is presented using three different streaming video data: in 31 video data is displayed coming from the fixed camera of a fixed in-situ node; in 32 video data are shown coming from the hand-held camera operated by the in-situ operator; in 33 video data coming from the helmet camera worn by the in-situ operator. The settings of each of these views can be customized using a system of slider and buttons 34. In particular, the in-situ fixed camera can be remotely operated modifying its orientation and its zooming. The technician at the technician's site or the expert can select which of these views is currently the active view 5 and have an audio/textual chat 36 with the other operators of the community. Moreover, the expert can draw enhancing symbols and markers, 37 or 38, using a selected technician interface mouse, pen, touch-screen etc. (not shown) on the active view, causing this information to appear on the see-through display worn by the in-situ operator. The latter, in this way, can be guided with extreme precision in actions, since the guidance is contextualized in the physical space on the field of view. In the same way the expert can send other kind of useful graphical information that is superimposed on the field of view of the in-situ operator, like cad drawings, text, 3D data, animations etc.

It is advantageous that the technician at the technician' s site has a see through monitor, so that the technician can see contemporaneously and on a same screen the images of the site and the images sent by the remote expert.

Optionally, the headset can also be equipped with a 3DOF tracking system, used to measure rotational head moments of the skilled technician. This is used to compensate such movements in terms of visual displacement of the computer generated graphical markers that are overlapped on the field of view of the technician. Say for example that the technician is looking at a complex control panel populated by a variety of controls: the remote expert is pointing the attention to a specific object overlapping graphical markers around it. This correspondence is obviously valid only as long as the in-situ technician does not translate or rotate the head. While translational movements are not very frequent in a typical maintenance operation, small rotational movements can occur frequently with a consequent loss of the correspondence between the objects and the overlapped markers. The presence of a 3DOF tracking system on the headset allows to compensate for such rotational movements, helping to track the correct object-marker correspondence. The system also takes in account the inevitable delays occurring in the communication between the in-situ technician and the remote expert.

Finally, for an effective communication over narrow-band links to the internet the system make explicit use of video and audio compression technology. In particular, the video streams and audio streams are compressed and combined in order to stay within the limits of the standard UMTS data plans (384 Kbit/s uplink and 384 Kbit/s down link) . The system is also equipped with adaptive algorithms that increase the quality of the video-audio-data streams when the availability of larger bandwidth is detected. In particular, H.264 Compression Technology can be used.

In a preferred embodiment, summarized by the block diagram of Fig. 11, the apparatus according to the invention has a computing system worn by the user that, in an advantageous embodiment of the invention, controls: a see-trough near eye display or a standard display; an auxiliary standard display; a RFID or barcode reader; two or more video cameras; a H.264 compression technology; input devices (like keyboard, mouse, etc) .

These controls are in turn managed centrally through the internet by the centralised communication server, as shown in figure 12, which illustrates a data communication scheme applied to the architecture of the virtual community communication system for remote technical assistance of figure 1 using the preferred working units of figure 11. Notwithstanding reference has been made to the presence of a centralised communication server, such a server can be virtually organized as shown in figure 12, in an alternative data communication scheme applied to a different embodiment of a virtual community communication system for remote technical assistance, using a peer to peer architecture, and using the preferred working units of figure 11.

The foregoing description of a specific embodiment will so fully reveal the invention according to the conceptual point of view, so that others, by applying current knowledge, will be able to modify and/or adapt for various applications such an embodiment without further research and without parting from the invention, and it is therefore to be understood that such adaptations and modifications will have to be considered as equivalent to the specific embodiment. The means and the materials to realise the different functions described herein could have a different nature without, for this reason, departing from the field of the invention. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation.

Claims

1. A method for remote assistance during assembly or maintenance operations, the method comprising the steps of: providing at least one technician at a first location and at least one expert at a second location, exchanging information data between said at least one technician and said at least one expert, through a set of communication channels, including audio, voice and interactive graphics, as well as 3D data, wherein the information data are selected among video images, graphics and speech signals of the technician and wherein additional information data in the form of augmented-reality information are transmitted from the remote expert at the second location to the in situ technician at the first location highlighting specific objects in the field of view of the technician, said expert being equipped with a computer and videoconferencing devices; said technician being equipped with a wearable computer having a wi-fi antenna associated to said wearable computer for data transmission; a headset connected to said computer including headphones, a noise-suppressing microphone, one near-eye see-trough AR display, and a miniature camera mounted on the display itself used to capture what is in the front of view of the technician; characterised in that said least one technician and said at least one expert are arranged respectively at an in-situ-node and at a remote node of a network, said nodes communicating and exchanging data through the internet via a centralised communication server and in that the data are streamed via a hardware-accelerated video compression means.

2. An apparatus for remote assistance during assembly or maintenance operations, comprises: means for exchanging information data between at least one technician at a first location and at least one expert at a second location through a set of communication channels, selected among audio, voice and interactive graphics, as well as 3D data, wherein the information data are a collection of video images, graphics and speech signals of the technician and wherein additional information data in the form of augmented-reality information are transmitted from a remote expert at the second location to an in situ technician at the first location highlighting specific objects in the field of view of the technician, a computer and videoconferencing devices to be used by said expert; a unit to be used by said technician comprising a wearable computer having a wi-fi antenna associated to said wearable computer for data transmission; a headset connected to said computer including headphones, a noise-suppressing microphone, one near- eye see-trough AR display, and a miniature camera mounted on the display itself used to capture what is in the front of view of the technician; characterised in that means are provided for communicating and exchanging data between at least two nodes through the internet via a centralised communication server and hardware-accelerated video compression means for streaming said data.

3. Apparatus according to claim 2, wherein a hand held camera is provided connected to said computer equipped with a light source for lighting desired targets.

4. Apparatus according to claim 2, wherein additional automated remote computing nodes are provided to create additional video feeds, in particular auxiliary fixed cameras that are positioned by the technicians and that can be controlled by the remote experts for pan, zoom and tilt movements.

5. Method according to claim 1, wherein the organisation of a multitude of in-situ technicians and remote experts is established in a distributed virtual community for the exchange of knowledge, wherein at least one on-situ technician at one node and at least one remote expert at another nodes are provided communicating and exchanging data with each other.

6. Apparatus according to claim 2, wherein Augmented Reality means are provided to overlap special visual markers on the objects falling inside the field of view of the operator.

7. Apparatus according to claim 6, wherein said headset can also be equipped with a 3DOF tracking system, used to measure rotational head moments of the skilled technician, in such a way to compensate such movements in terms of visual displacement of the computer generated graphical markers that are overlapped on the field of view of the technician.

8. Apparatus according to claim 2, wherein said video compression means comprise H.264 Compression

Technology.

9. Apparatus according to claim 8, wherein said video streaming is associated to VoiceOverIP technology.

10. Apparatus according to claim 2, wherein said hardware-accelerated video compression means are arranged in such a way that the video streams and audio streams are compressed and combined, preferably 384 Kbit/s uplink _.and 384 Kbit/s down link.