JP7779039B2 - Support system, support device, support method, and program - Google Patents

Description

The present invention relates to a support system, a support device, a support method, and a program.

A support system is known in which a video image observed by a worker is transmitted to a supporter in a remote location, and the supporter gives instructions to the worker using a slider while observing the video image (see, for example, Patent Document 1). Another related technology is Patent Document 2.
Patent Document 1: JP 2006-209664 A Patent Document 2: JP 2012-18605 A

In a support system, it is desirable to be able to communicate the details of work that involves operating equipment. Simply sharing a slideshow between the supporter and the worker makes it difficult to communicate the details of work that involves operating equipment.

To solve the above problem, a first aspect of the present invention provides a support system including a support-side device and a work-side device. The support-side device may include a support-side display unit. The support-side display unit may display a virtual model that resembles equipment. The support-side device may include a detection unit. The detection unit may detect the body movement of the supporter relative to the virtual model. The support-side device may include a transmission unit. The transmission unit may transmit information about the body movement relative to the virtual model. The work-side device may include a work-side display unit. The work-side device may display a virtual image showing the body movement based on the received body movement information.

The body movement detected by the detection unit may include position information about at least a portion of the supporter's body in three-dimensional space.

A reference point marker may be provided at a predetermined position on the equipment. The virtual model may be associated with a reference point provided at a position corresponding to the reference point marker and with reference coordinates, which are three-dimensional coordinates based on the reference point. The body movement detected by the detection unit may include position information for at least a part of the supporter's body in the reference coordinates. The work-side device may include a marker detection unit. The marker detection unit may detect reference point markers on the equipment. The work-side device may include an adjustment unit. The adjustment unit may adjust the position at which the virtualized image is superimposed on the image of the equipment based on the detection result of the reference point marker and the position information included in the received body movement information.

The support side device may further include a selection unit. The selection unit may select a virtual model corresponding to the facility from among multiple virtual models.

The selection unit may select a virtual model corresponding to the facility from among multiple virtual models based on instructions from the supporter.

An identification image for identifying the equipment may be displayed on the equipment. The work-side device may include a reading unit. The reading unit may read the identification image. The work-side device may further include an identification information transmission unit. The identification information transmission unit may transmit identification information obtained by reading the identification image with the reading unit to the support-side device. The selection unit may select a virtual model corresponding to the equipment from among the multiple virtual models based on the identification information received from the work-side device.

The work-side device may further include an imaging unit. The imaging unit may capture an image of the equipment to obtain an equipment image. The work-side device may further include an equipment image transmission unit. The equipment image transmission unit may transmit the equipment image to the support-side device. The support-side device may include a model generation unit. The model generation unit may generate a virtual model based on the equipment image received from the work-side device.

The work-side device may further include an imaging unit. The imaging unit may capture an image of the equipment to obtain an equipment image. The work-side device may further include a correction information generation unit. The correction information generation unit may generate correction information based on the equipment image. The work-side device may further include a correction information transmission unit. The correction information transmission unit may transmit the correction information to the support-side device. The support-side device may include a model correction unit. The model correction unit may correct the virtual model based on the correction information received from the work-side device. The correction information generation unit may generate correction information by comparing the virtual model received from the support-side device with the equipment image.

A second aspect of the present invention provides a support method for supporting work using a support-side device and a work-side device capable of communicating with the support-side device. The support method may include a step of displaying a virtual model that simulates equipment in the support-side device. The support method may include a step of detecting, in the support-side device, physical movements of a supporter corresponding to the virtual model. The support method may include a step of transmitting, in the support-side device, information about the physical movements of the supporter relative to the virtual model. The support method may include a step of receiving, by the work-side device, information about the physical movements. The support method may include a step of displaying, in the work-side device, a virtual image that represents the physical movements based on the received information about the physical movements.

A third aspect of the present invention provides a support-side device. The support-side device may be communicably connected via a network to a working-side device at a work site where equipment is installed. The support-side device may include a support-side display unit. The support-side display unit may display a virtual model that resembles the equipment. The support-side device may include a detection unit. The detection unit may detect the body movements of a supporter in accordance with the virtual model. The support-side device may include a transmission unit. The transmission unit may transmit information about the body movements of the supporter relative to the virtual model to the working-side device.

A fourth aspect of the present invention provides a program for causing a computer to function as the above-mentioned support device.

Note that the above summary of the invention does not list all of the necessary features of the present invention. Subcombinations of these features may also constitute inventions.

1 is a diagram showing an overview of a remote support system 1 according to a first embodiment of the present invention. 1 is a diagram illustrating an example of a remote support system 1 according to a first embodiment of the present invention. FIG. 2 is a schematic diagram showing an example of a virtual model 7 in a support side device. 10 is a schematic diagram showing an example of a composite image 10 in a work-side apparatus. FIG. FIG. 10 is a diagram showing an example of posture data representing the motion of a supporter. FIG. 10 is a diagram showing an example of point cloud data representing the actions of a supporter. 10 is a flowchart showing an example of processing content of a support-side device in the remote support method according to the first embodiment; 8 is a flowchart showing an example of the processing content in step S50 of FIG. 7. 10 is a flowchart showing an example of processing content of a work-side apparatus in the remote assistance method according to the first embodiment; 10 is a flowchart showing an example of processing content in step S80 of FIG. 9. FIG. 2 is a diagram illustrating another example of the remote support system 1 according to the first embodiment of the present invention. FIG. 10 is a schematic diagram showing an example of a virtual model 7 in a support side device according to a second embodiment of the present invention. FIG. 10 is a schematic diagram showing an example of a composite image 80 in a work-side apparatus according to a second embodiment of the present invention. FIG. 10 is a diagram illustrating an example of a remote support system 1 according to a second embodiment of the present invention. FIG. 10 is a diagram illustrating another example of the remote support system 1 according to the second embodiment of the present invention. 10 is a flowchart illustrating an example of processing content of a support-side device in a remote support method according to a second embodiment. 17 is a flowchart showing an example of the processing content in step S160 of FIG. 16. 10 is a flowchart showing an example of processing content of a work-side apparatus in a remote support method according to a second embodiment; 20 is a flowchart showing an example of the processing content in step S310 of FIG. 18. FIG. 11 is a schematic diagram showing an example of a composite image 90 in a work-side apparatus according to a third embodiment of the present invention. 22 illustrates an example computer 2200 in which aspects of the present invention may be embodied, in whole or in part.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

Figure 1 is a diagram showing an overview of a remote support system 1 according to a first embodiment of the present invention. The remote support system 1 includes a support-side device 2 and a work-side device 3. The support-side device 2 and the work-side device 3 are communicatively connected to each other via a communication network 4. In one example, the support-side device 2 is a head-mounted display (HMD) type virtual reality wearable computer terminal. The support-side device 2 displays a virtual model 7 simulating equipment 5 to the supporter. The virtual model 7 may be a 3D hologram image.

While looking at the virtual model 7, the supporter actually moves their fingers, hands, etc. to operate it in the virtual space, demonstrating an example to the worker. The support-side device 2 uses motion capture technology to detect the supporter's physical movements relative to the virtual model 7. The support-side device 2 then transmits information about the physical movements relative to the virtual model 7 to the work-side device 3 via the communication network 4.

The work-side device 3 displays a body movement image, which is a virtualized image showing the body movement based on the received body movement information. The body movement image may be a video that virtually reproduces the movements of the supporter's fingers, hands, etc. In one example, the work-side device 3 is an augmented reality wearable computer terminal that uses a head-mounted display (HMD). The worker receives support from the images displayed on the work-side device 3 and performs operations on the equipment 5.

A reference point marker 6 may be provided at a predetermined position P _M0 in the equipment 5. Marker coordinates (X _M , Y _M , Z _M ) may be set based on the position P _M0 of the reference point marker 6. A reference point 8 may be provided on the virtual model 7 at a position P ₀ corresponding to the position P _M0 of the reference point marker 6. Furthermore, reference coordinates (X ₀ , Y ₀ , Z ₀ ) that are three-dimensional coordinates at the reference point P ₀ may be provided on the virtual model 7. The detected body movement may include position information about at least a part of the supporter's body relative to the reference coordinates (X ₀ , Y ₀ , Z ₀ ). In this example, in the coordinates of the support-side device 2, the direction from the supporter toward the virtual model 7 is defined as the Z-axis direction, and a plane perpendicular to the Z-axis direction is defined as the XY plane. In the XY plane, the X-axis direction and the Y-axis direction are defined as directions that are orthogonal to each other. Similarly, in the coordinate system of the work-side device 3, the direction from the worker toward the equipment is defined as the Z-axis direction, and the plane perpendicular to the Z-axis direction is defined as the XY plane. Within the XY plane, the X-axis direction and the Y-axis direction are perpendicular to each other.

Figure 2 is a diagram showing an example of a remote support system 1 in the first embodiment of the present invention. The support-side device 2 in this example includes a communication control unit 101, a detection unit 102, and a support-side display unit 103. The support-side display unit 103 displays a virtual model 7 that resembles equipment 5. The support-side display unit 103 may have a holographic lens display. The detection unit 102 detects the body movements of the supporter relative to the virtual model 7. The detection unit 102 imports motion capture data indicating the supporter's movements from a motion capture device 106. The communication control unit 101 functions as a transmission unit that transmits information about the body movements relative to the virtual model 7 (equipment model). As a result, the support-side device 2 can detect the supporter's relative movements relative to the virtual model 7 that resembles equipment 5 in virtual space and send the information to the work-side device 3.

The support-side device 2 in this example may include a virtual model storage unit 104, a selection unit 105, a screen control unit 110, a sensor 112, a microphone 114, a speaker 116, and an audio control unit 118. The virtual model storage unit 104 is a database that pre-stores information about virtual models 7. The virtual model storage unit 104 may store multiple types of virtual models 7 corresponding to multiple types of equipment 5. However, the virtual model storage unit 104 does not have to be included in the support-side device 2. In this case, the virtual model 7 is downloaded from an external device. The selection unit 105 selects a virtual model 7 corresponding to the equipment 5 from among the multiple virtual models. The selection unit 105 may select a virtual model 7 corresponding to the equipment 5 from among the multiple virtual models based on instructions from the supporter. Alternatively, the selection unit 105 may select a virtual model 7 corresponding to the equipment 5 from among the multiple virtual models based on identification information received from the work-side device 3.

The screen control unit 110 may realize virtual reality by integrating the virtual model 7, which is a 3D hologram image, with motion capture data. The screen control unit 110 may combine the hologram image with the physical environment recognized by the imaging unit and the visible light tracking camera. The screen control unit 110 may receive inertial measurement information from the sensor 112, combine it with data from the imaging unit and the visible light tracking camera that recognize the environment, and display a hologram image on the support-side display unit 103, which is a holographic lens display with a transparent lens. The microphone 114, speaker 116, and audio control unit 118 enable conversation between the supporter and the worker via the working-side device 3.

The sensor 112 may include a depth sensor (distance sensor), an imaging unit (camera), an inertial measurement unit, multiple visible light cameras for tracking, and an environmental recognition unit. The visible light camera for tracking detects head movement. Two infrared cameras for eye tracking detect eye movement. The depth sensor calculates the distance to the object, its shape, etc. by emitting a laser from a light source to an object and measuring the reflected light. The inertial measurement unit may include an accelerometer, a gyroscope, and a magnetometer. The inertial measurement unit detects three-dimensional inertial motion (translational and rotational motion in three orthogonal axes). The accelerometer may detect translational motion, and the gyroscope may detect rotational motion.

The work-side device 3 includes a communication control unit 201 and a work-side display unit 202. The work-side device 3 may also include an imaging unit 204, a screen control unit 210, a sensor 212, a microphone 214, a speaker 216, and an audio control unit 218.

The communication control unit 201 functions as a receiving unit that receives physical movement information from the support-side device 2. The working-side display unit 202 displays a virtualized image (physical movement image) showing the physical movement based on the received physical movement information. In particular, the working-side display unit 202 may display the physical movement image superimposed on an equipment image obtained by photographing the equipment 5. The imaging unit 204 may be a camera. The imaging unit 204 may generate the equipment image by photographing the equipment 5. The equipment image may include not only the equipment 5 but also a background image of the surrounding environment of the equipment 5.

Sensor 212 may be similar to sensor 112 described above. Sensor 212 may include a depth sensor (distance sensor), an inertial measurement unit, multiple visible light cameras for tracking, and an environmental recognition unit. The visible light camera for tracking detects head movement. Two infrared cameras for eye tracking detect eye movement. The depth sensor calculates the distance to the object, its shape, etc. by emitting a laser from a light source to an object and measuring the reflected light. The inertial measurement unit may include an accelerometer, a gyroscope, and a magnetometer. The inertial measurement unit detects three-dimensional inertial motion (translational motion and rotational motion in three orthogonal axes). The accelerometer may detect translational motion, and the gyroscope may detect rotational motion.

The microphone 214, speaker 216, and audio control unit 218 enable conversation between the supporter and the worker via the support-side device 2. The screen control unit 210 has various functions, such as a marker detection unit 241, an adjustment unit 242, and a reading unit 243. These functions will be described later.

Figure 3 is a schematic diagram showing an example of a virtual model 7 in the support-side device 2. Figure 4 is a schematic diagram showing an example of a composite image 10 in the work-side device 3. Note that coordinates are displayed in Figures 3 and 4 for explanatory purposes. Coordinates do not need to be displayed on the actual virtual model 7 and composite image 10.

The composite image 10 in Fig. 4 may be an image obtained by superimposing an equipment image 11 obtained by capturing an image of the equipment 5 using the imaging unit 204 and a body movement image 12, which is a virtualized image showing body movement. As shown in Fig. 4, a reference point marker 6 is provided in the equipment 5 at a predetermined position _PMO . The reference point marker 6 may have an identification image 9 for identifying the equipment 5. In one example, the identification image 9 may be a barcode image. Unlike the case in Fig. 4, the reference point marker 6 and the identification image 9 may be provided separately. In the example shown in Fig. 4, the body movement image 12 is a virtualized image of the movement of the supporter's hands and fingers.

The screen control unit 210 shown in FIG. 2 may function as a marker detection unit 241 that detects reference point markers 6 on the equipment 5. The screen control unit 210 may also function as an adjustment unit 242. The adjustment unit 242 may adjust the position at which the body movement image 12 is superimposed on the equipment image 11 based on the detection result of the reference point markers 6 and the position information included in the received body movement information. The screen control unit 210 may also function as a reading unit 243 that reads the identification image 9. In this case, the communication control unit 201 also functions as an identification information transmission unit that transmits the identification information obtained by reading the identification image 9 with the reading unit 243 to the support-side device 2. The selection unit 105 in the support-side device 2 selects a virtual model 7 corresponding to the equipment 5 from among multiple virtual models based on the identification information received from the work-side device 3.

The information on the body movements detected by the detection unit 102 may include position information on at least a part of the supporter's body in three-dimensional space. The position information may include body movement point coordinate values ( _XRi , _YRi , _ZRi ) shown in Fig. 3. The body movement point coordinate values ( _XRi , _YRi , _ZRi ) refer to the coordinates (Xi, Yi, Zi) of each body movement point 24 on the body when the reference point ( _X0 , _Y0 , _Z0 ) is used as the reference in the reference coordinates (X, _Y , _{Z )} . Note that in Figs. 3 and 4, the direction perpendicular to the paper surface is the Z-axis direction. The body motion coordinate values (X _Ri , Y _Ri , Z _Ri ) are expressed as X _Ri =X _i −X ₀ , Y _Ri =Y _i −Y ₀ , and Z _Ri =Z _i −Z ₀ .

The screen control unit 210 may adjust the position (XM + _XRi , YM + _YRi , ZM + _ZRi ) at which the body movement image 12 is superimposed on the equipment image 11 based on the detection result of the reference point marker 6 (marker coordinates _XM , _YM , _ZM ) _and the coordinate values ( _XRi , _YRi , _ZRi ) of the body movement point 24 included in _the received body movement _information .

Figure 5 is a diagram showing an example of posture data representing the movements of a support worker. Figure 6 is a diagram showing an example of point cloud data representing the movements of a support worker. Motion capture data obtained by reading the movements of a worker using motion capture technology may be posture data as shown in Figure 5, or point cloud data as shown in Figure 6.

The posture data expresses a movement based on the recognition results of the positions of parts (shown as black circles in the figure) that are basic elements of the movement, such as the joints of the fingers. The posture data may be expressed by the three-dimensional coordinates of each position of multiple parts that are basic elements. Points that are basic elements of the movement are called body motion points 24. The position information may be expressed by body motion coordinate values ( _XRi , _YRi , _ZRi ) (i = 1, 2, 3, ... n) of each position of the (n) body motion points 24.

6 indicates the behavior of a plurality of extracted points (shown as dots in the figure). Therefore, each extracted point may be regarded as a body motion point 24. The point cloud data may express position information as body motion coordinate values ( _XRi , _YRi , _ZRi ) (i=1, 2, 3, . . . N) of the extracted points.

Figure 7 is a flowchart showing an example of the processing content of the support-side device 2 in the remote support method of the first embodiment. The remote support method supports work using the support-side device 2 and the work-side device 3. The selection unit 105 selects a field worker based on instructions from the supporter (step S10). The supporter can instruct the field worker using a pointing device or the like. The voice control unit 118 initiates a voice call between the selected field worker and the supporter (step S20). The selection unit 105 selects a virtual model 7 corresponding to the facility 5 to be supported from among multiple virtual models stored in the virtual model storage unit 104 (step S30).

The selection unit 105 may select a virtual model based on instructions from a support person. The selection unit 105 may specify a virtual model based on a signal from a pointing device or the like. The selection unit 105 may select a virtual model 7 corresponding to the equipment from among multiple virtual models based on identification information received from the work-side device 3.

The support-side display unit 103 displays a virtual model 7 that resembles the equipment 5 (step S40). The supporter operates the displayed virtual model 7 in the virtual space. The detection unit 102 detects the supporter's physical movements relative to the virtual model 7 and transmits information about the detected physical movements to the work-side device 3 (step S50). The voice control unit 118 ends the voice call (step S60). Note that the worker may communicate equipment identification information to the supporter via voice call. However, the voice call processing (steps S20 and S60) and the on-site worker selection processing (step S10) may be omitted.

FIG. 8 is a flowchart showing an example of the processing content in step S50 of FIG. 7. The communication control unit 101 transmits a start code (step S51). The start code is information indicating the start of processing. Until the operation ends (step S52: NO), the total number (n) of body motion points 24 is acquired each time a transmission period is reached (step S53: YES) (step S54). The initial value i = n. Next, unless i = 0 (step S55: NO), the detection unit 102 acquires body motion coordinates ( _XRi = _{Xi - X0} , _YRi = Yi - _Y0 , _ZRi = Zi _{- Z0 )} for one body motion point ₂₄ (Xi, _Yi , Zi) using motion capture data (step S56). The detection unit 102 sets i = i - 1 (step S57). i is decremented from the initial value n to n - 1. Until i=0 (step S55: NO), the detection unit 102 repeats the processes from step S55 to step S57.

As a result of steps S54 to S57, the detection unit 102 can acquire body motion coordinates ( _XRi = Xi - _X0 , YRi = Yi - _Y0 , _ZRi = _Zi - _{Z0 )} based on the motion capture data for each of the body motion points 24 where i = ₁ , ₂ , ..., n. When i = 0 (step S55: YES), acquisition of body motion coordinates for all (n) body motion points has been completed. The communication control unit 101 transmits information on the total number (n) of body motion points 24 and information on the body motion coordinates ( _XRi = _Xi - _X0 , _YRi = _Yi - _Y0 , _ZRi = _Zi - _Z0 ) as position information (step S58).

When an instruction to end the operation is given (step S52: YES), the communication control unit 101 sends an end code to the work-side device 3 (step S59). The process then returns.

Figure 9 is a flowchart showing an example of the processing content of the working-side device 3 in the remote support method of the first embodiment. The voice control unit 218 initiates a voice call between the supporter and the worker (step S70). The communication control unit 201 receives body movement information from the support-side device 2, and the working-side display unit 202 displays a body movement image, which is a virtualized image showing the body movement based on the received body movement information (step S80). When the support processing is completed, the voice control unit 218 ends the voice call (step S90).

Figure 10 is a flowchart showing an example of the processing content in step S80 of Figure 9. The communication control unit 201 functions as a receiving unit that receives various information from the support side device 2. When the communication control unit 201 receives a start code from the support side device 2 (step S101: YES), it activates the imaging unit 204 (step S102). The imaging unit 204 takes an image of the surrounding environment including the equipment 5 and generates image data. The screen control unit 210 acquires the generated image data as the equipment image 11.

If the communication control unit 201 receives an end code (step S103: Yes), it assumes that the assistance process has ended and stops the image capture unit 204 (step S104). Then, the process returns. Until the communication control unit 201 receives an end code (step S103: No), when the communication control unit 201 receives body movement information (step S105: Yes), the screen control unit 210 acquires the total number (n) of body movement points 24 and the body movement coordinates ( _XRi = Xi - _X0 , _YRi = _Yi - _Y0 , _ZRi = _Zi - _Z0 ) for each body movement point 24 (i = 1, 2, ... _n ) (step S106). The initial value i = n may be used.

The screen control unit 210 also functions as a marker detection unit 241 that detects the reference point marker 6. The screen control unit 210 acquires marker coordinates ( _XM , _YM , _ZM ) indicating the position of the reference point marker 6 as the detection result of the reference point marker 6 (step S107). Next, unless i=0 (step S108: NO), the screen control unit 210 draws a body movement image 12 based on the body movement information. The screen control unit 210 generates a composite image 10 by superimposing the equipment image 11 and the body movement image 12. The work-side display unit 202 displays the composite image 12 (step S109). The screen control unit 210 adjusts the position at which the body movement image 12 is superimposed on the equipment image 11 to (XM + _XRi , _YM + _YRi , ZM + _ZRi ) based on the marker coordinates ( _XM , _YM , _ZM ) that are the detection results of the reference point marker and _the body movement coordinates ( _XRi = _{Xi -X0} , _YRi = _{Yi -Y0} , _ZRi = _{Zi -Z0} ) included in the received position _information .

Next, the screen control unit 210 sets i = i - 1 (step S110). i is decremented from the initial value n to n - 1. Until i = 0 (step S108: NO), the detection unit 102 repeats the processing of steps S109 and S110. As a result, the screen control unit 210 can display a virtualized body movement image 12 superimposed on the equipment image 11 for each body movement point 24 where i = 1, 2, ... n.

FIG. 11 is a diagram showing another example of the remote support system 1 according to the first embodiment of the present invention. In the remote support system 1 in FIG. 11, the working-side device 3 includes a correction information generation unit 220. Furthermore, in the working-side device 3, the communication control unit 201 may function as an equipment image transmission unit that transmits an equipment image 11 obtained by capturing an image of the equipment to the support-side device 2. Alternatively, the communication control unit 201 may function as a correction information transmission unit that transmits correction information generated by the correction information generation unit 220 to the support-side device 2. The correction information generation unit 220 may generate correction information based on the equipment image 11. In one example, the working-side device 3 receives a virtual model 7 from the support-side device 2 via the communication control unit 201. The correction information generation unit 220 may compare the virtual model 7 with the equipment image 11 to create correction information. Furthermore, the correction information may be an equipment image 11 with a reduced resolution to reduce the amount of data.

The support-side device 2 may include a model generation unit 120. The model generation unit 120 generates a virtual model 7 based on an equipment image 11 received from the work-side device 3. The model generation unit 120 can detect the dynamic state of the equipment 5 based on images captured by the work-side device 3. The model generation unit 120 can generate a virtual model that imitates the equipment 5 according to the dynamic state of the equipment 5.

The support-side device 2 may include a model correction unit 122. The model correction unit 122 may correct the virtual model 7 based on correction information received from the work-side device 3. In one example, the correction information may be a two-dimensional image of the equipment 5 with a low image quality. In one example, the virtual model 7 includes settings for the possible states of each part. The model generation unit 120 may detect the state of each part of the equipment 5 based on the correction information, and correct the virtual model 7 by changing the state of each part based on the detection results.

The remote support system 1 of the first embodiment of the present invention uses virtualized body movements of the supporter, which makes it possible to intuitively convey the operation details, compared to when virtualized sliders or the like are used. This improves the work efficiency of on-site workers and supporters. Furthermore, unlike simple chromakey compositing, there is no need to have the supporter stand and operate against a background of a specific color.

Figure 12 is a schematic diagram showing an example of a virtual model 7 in the support-side apparatus 2 in the second embodiment of the present invention. Figure 13 is a schematic diagram showing an example of a composite image 80 in the work-side apparatus 3 in the second embodiment of the present invention.

As shown in Fig. 12 , in this embodiment, the virtual model 7 may include a door provided with a handle 25. The virtual model 7 includes a plurality of operation parts, namely, an operation part 21 and an operation part 22. The assistant changes the state of the operation part 22 by specifying the operation part 22 in the virtual space. In the example of Fig. 12 , the operation part 22 is a door, and the assistant can perform an assisting operation of rotating the operation part 22 to a specific rotation position in the virtual space. Operation information indicating the content of the assisting operation at the operation part 22 of the equipment 5 is transmitted from the support-side apparatus 2 to the work-side apparatus 3. The operation information may include information on the position 23 ( _X1 , _Y1 , _Z1 ) of the operation part 22 and the operation part rotation angle ( _Rx , _Ry , _Rz ). The information regarding the position 23 ( _X1 , _Y1 , _Z1 ) of the operation part 22 may be the relative operation part coordinates ( _XR , _YR , _ZR ) of the position 23 of the operation part 22 based on the reference point _P0 . The relative operation part coordinates ( _XR , _YR , _ZR ) are expressed as _XR = _X1 - _X0 , _YR = _Y1 - _Y0 , _ZR = _Z1 - _Z0 .

As shown in FIG. 13 , the work-side apparatus 3 has a work-side display unit 202 that displays a composite image 80 obtained by combining an equipment image 11 obtained by capturing an image of the equipment 5 using the imaging unit 204 with an operation status image 30 that indicates the state of the equipment 5 after the assist operation. In this example, the composite image 80 may include an image of a door including a handle 33. The work-side apparatus 3 receives, as operation information, information on the coordinates ( _X1 , _Y1 , _Z1 ) of the operation portion 22 and the operation portion rotation angle ( _Rx , _Ry , _Rz ). The work-side apparatus 3 generates the operation status image 30 based on the operation information. Then, the work-side display unit 202 displays the composite image 80 generated by overlaying the operation status image 30 on the equipment image 11. The operation status image 30 includes multiple operation portion images 31 and 32. The operation portion image 32 can be in multiple states. In this example, the operation portion image 32 indicates the state of the door being opened or closed. The worker-side display unit 202 may display a message 35 and a symbol 36 relating to the content of the assist operation together with the operation status image 30. This makes it easier for the worker to understand the operation status.

Figure 14 is a diagram showing an example of a remote support system 1 according to the second embodiment of the present invention. The remote support system 1 includes a support side device 2 and a work side device 3. The support side device 2 and the work side device 3 may be communicably connected via a communication network 4. The support side device 2 may be a head-mounted display (HMD) type virtual reality wearable computer terminal.

The support-side device 2 in this example includes a communication control unit 131, an operation part detection unit 130, and a support-side display unit 103. In this example, the support-side display unit 103 displays the virtual model 7 as shown in FIG. 12. The support-side display unit 103 may have a holographic lens display. The operation part detection unit 130 detects the content of the support operation at the operation part of the equipment 5. In the example shown in FIG. 12, the operation part detection unit 130 detects the content of the support operation, which is to rotate the door at the operation part 22 of the equipment 5 to the operation part rotation angle.

The operating part detection unit 130 imports motion capture data indicating the content of the assisting operation from the motion capture device 106. The operating part detection unit 130 may identify the operating part designated by the assistant by determining whether there is a collision between the virtual model 7 and a body region corresponding to the assistant's body, or between the virtual model 7 and an instruction image (e.g., a slider, pointer, etc.) operated by the assistant. The operating part detection unit 130 may detect the assistant's body movements relative to the virtual model 7, similar to the detection unit 102 in the first embodiment. In this case, the operating part detection unit 130 may detect the content of the assisting operation by detecting body movements using the body movement detection technology in the first embodiment.

The communication control unit 131 functions as an operation information transmitting unit that transmits operation information indicating the content of the assisting operation. As a result, the support-side device 2 can detect the operation content at the operation part of the virtual model 7 that simulates the equipment 5 and send it to the working-side device 3. The communication control unit 131 can transmit information related to the coordinates ( _X1 , _Y1 , _Z1 ) of the operation part 22 and the operation part rotation angle ( _Rx , _Ry , _Rz ) to the working-side device 3 as operation information.

The support side device 2 in this example may include a virtual model storage unit 104, a selection unit 105, a screen control unit 110, a sensor 112, a microphone 114, a speaker 116, and an audio control unit 118. These components may be similar to those shown in FIG. 2. Therefore, repeated explanations will be omitted.

The work-side device 3 includes a communication control unit 231 and a work-side display unit 202. The work-side device 3 may include an imaging unit 204, a screen control unit 210, a sensor 212, a microphone 214, a speaker 216, and an audio control unit 218. The communication control unit 231 functions as a receiving unit that receives operation information. The screen control unit 210 superimposes an operation status image 30 indicating the state of the equipment 5 after the assisting operation onto the equipment image 11 to create a composite image. The work-side display unit 202 displays the operation status image 30 indicating the state of the equipment 5 after the assisting operation, superimposed onto the image of the equipment 5. Specifically, the work-side display unit 202 displays a composite image 80. The work-side display unit 202 may display a message 35 and a symbol 36 regarding the content of the assisting operation together with the operation status image 30.

The work-side device 3 may include an image generation unit 230 and a work-side virtual model storage unit 232. The work-side virtual model storage unit 232 stores a work-side virtual model that simulates the equipment 5. The image generation unit 230 generates an operation status image 30 that shows the state of the equipment after an assist operation, according to the assistance content specified for each operation portion 21, 22. The image generation unit 230 may function as a deformation unit that deforms the work-side virtual model based on the operation information. The work-side display unit 202 may display the deformed work-side virtual model as the operation status image 30. The work-side display unit 202 may change the display of the body region or the portion where the instruction image comes into contact with the virtual model 7. In the example shown in FIG. 13 , the image 32 of the operation portion where the assistance operation is being performed is displayed with a dotted line.

Apart from these points, the imaging unit 204, screen control unit 210, sensor 212, microphone 214, speaker 216, and audio control unit 218 have the same configuration as shown in Figure 2. Therefore, repeated explanations will be omitted.

Figure 15 is a diagram showing another example of the remote support system 1 in the second embodiment of the present invention. In the case shown in Figure 14, the operation part detection unit 130 detects the body movement of the supporter relative to the virtual model 7. However, the remote support system 1 of this embodiment is not limited to this case.

In the example shown in FIG. 15, the sensor 112 is omitted. The support side device 2 includes a pointing device 132. The pointing device 132 may be, for example, a mouse. The supporter operates the pointing device 132. The operation part detection unit 130 detects the operation part based on a signal from the pointing device 132. Specifically, the operation part detection unit 130 may detect the operation part by determining whether the cursor image of the pointing device 132 has collided with the virtual model 7. The other structure is the same as the structure shown in FIG. 14, so repeated explanation will be omitted.

Figure 16 is a flowchart showing an example of processing details of the support-side device in the remote support method of the second embodiment. In the remote support method, work is supported using the support-side device 2 and the work-side device 3. The selection unit 105 selects a field worker based on instructions from the supporter (step S120). The supporter can instruct the field worker using a pointing device or the like. The voice control unit 118 initiates a voice call between the selected field worker and the supporter (step S130). The selection unit 105 selects a virtual model 7 corresponding to the facility 5 to be supported from among multiple virtual models stored in the virtual model storage unit 104 (step S140). The selection unit 105 may receive identification information for the facility 5 from the work-side device 3 and select the virtual model 7 based on the identification information. However, the selection unit 105 may also select a virtual model based on instructions from the supporter. The supporter can point to the virtual model 7 using a pointing device or the like.

The support-side display unit 103 displays a virtual model 7 that resembles the equipment 5 (step S150). The virtual model 7 may be stored in the virtual model storage unit 104 or may be downloaded. The supporter operates the displayed virtual model 7 in virtual space. The operation part detection unit 130 detects the content of the support operation at the operation part 22 of the equipment. Operation information indicating the content of the support operation is sent to the work-side device 3 (step S160). The voice control unit 118 then ends the voice call (step S170). Note that the worker may communicate equipment identification information to the supporter via voice call. However, the voice call processing (steps S130 and S170) and the on-site worker selection processing (step S120) may be omitted.

Figure 17 is a flowchart showing an example of the processing content in step S160 of Figure 16. The communication control unit 101 transmits a start code and a facility ID (step S200). The start code is information indicating the start of processing. The facility ID is identification information for identifying the facility. Until the operation is completed (step S201: NO), the operation part detection unit 130 performs a collision determination (step S203) each time the transmission period is reached (step S202: YES).

In this embodiment, the virtual model 7 is modeled after the equipment 5. The virtual model 7 has multiple operating parts, namely, operating parts 21 and 22. When the operating parts 21 and 22 are identified by body movements, the operating part detection unit 130 acquires a body area image corresponding to the supporter's body based on data from the motion capture device 106. The body area image may be generated based on the coordinates of the body movement point 24. In one example, the body area image may be the posture data shown in FIG. 5 or the point cloud data shown in FIG. 6.

The operation part detection unit 130 performs a collision determination between the virtual model 7 and the body area image corresponding to the supporter's body (step S203). If one of the multiple operation parts 21 and 22 collides with the supporter's body area image, the colliding operation part 22 is selected. If the operation part 21 is selected by the pointing device 132, the operation part detection unit 130 may detect the operation part by performing a collision determination between the cursor image of the pointing device 132 and the virtual model 7. The collision determination (step S203) may be performed for each transmission cycle (step S202: YES).

If an operation part is found (step S203: YES), the operation part detection unit 130 acquires identification information (ID) of the operation part 22 (step S204). Then, the operation part detection unit 130 may acquire operation part relative coordinates ( _XR , _YR , _ZR ) (where XR = _X1 - X0, _YR = _Y1 - _Y0 , _ZR = _Z1 - _Z0 ) which are relative coordinates of the position 23 ( _X1 , _Y1 , _Z1 ) of the operation part 22 with reference to the reference point _{P0 (X0, Y0 , Z0 )} . Furthermore, the operation part detection unit 130 acquires operation part rotation angles ( _Rx , _Ry , _Rz ) which indicate the orientation of _the operation part 22 ₍ step S206). The operation part rotation angles ( _Rx , _Ry , _Rz ) refer to the rotation angles of the X-axis, Y-axis, and Z-axis.

The communication control unit 131 transmits the operation information to the work-side apparatus 3. The operation information may include identification information (ID) of the operation part, relative coordinates of the operation part ( _XR , _YR , _ZR ), and rotation angles of the operation part ( _Rx , _Ry , _Rz ).

When an instruction to end the operation is given (step S201: NO), the communication control unit 131 sends an end code to the work-side device 3 (step S208). The process then returns.

Figure 18 is a flowchart showing an example of the processing details of the working-side device 3 in the remote support method of the second embodiment. The voice control unit 218 initiates a voice call between the supporter and the worker (step S300). The communication control unit 201 receives operation information from the support-side device 2. Based on the received operation information, the working-side display unit 202 displays an operation status image 30 indicating the state of the equipment after the support operation, superimposed on the equipment image 11 (step S310). When the support processing is completed, the voice control unit 218 terminates the voice call (step S320).

Figure 19 is a flowchart showing an example of the processing content in step S310 of Figure 18. The communication control unit 201 functions as a receiving unit that receives various information from the support-side device 2. When the communication control unit 201 receives a start code from the support-side device 2 (step S401: YES), it acquires the virtual model 7 (step S402). The virtual model 7 may be downloaded from an external device, received from the support-side device 2, or acquired from the working-side virtual model storage unit 232 in the working-side device 3.

Next, when the communication control unit 201 receives the start code from the support side device 2, it activates the imaging unit 204 (step S403). The imaging unit 204 captures an image of the surrounding environment including the equipment 5 and generates image data. The screen control unit 210 acquires the generated image data as the equipment image 11.

The screen control unit 210 also functions as a marker detection unit 241 that detects the reference point marker 6. The screen control unit 210 acquires marker coordinates ( _XM , _YM , _ZM ) indicating the position of the reference point marker 6 as the detection result of the reference point marker 6 (step S404). The screen control unit 210 may combine the virtual model 7 with an equipment image 11 obtained by capturing an image of the surrounding environment including the equipment 5, and display the combined image on the work-side display unit 202 (step S405).

Until the communication control unit 201 receives an end code (step S406: NO), when the communication control unit 201 receives operation information (operation part information) (step S407: YES), the image generation unit 230 acquires the identification information (ID) of the operation part 22, the operation part relative coordinates ( _XR , _YR , _ZR ), and the operation part rotation angle ( _Rx , _Ry , _Rz ) (step S408).

The image generation unit 230 draws an operation state image 30 indicating the state of the equipment 5 after the assist operation, based on the identification information (ID), the operation portion relative coordinates ( _XR , _YR , _ZR ), and the operation portion rotation angles ( _Rx , _Ry , _Rz ) of the operation portion 22. In other words, the image generation unit 230 may draw (redraw) the operation state image 30 by deforming the virtual model 7 synthesized in step S405.

The screen control unit 210 generates a composite image 80 by overlaying and combining the operation status image 30 on the equipment image 11. The work-side display unit 202 displays the composite image 80 (step S409). When the operation part rotation angle ( _Rx , _Ry , _Rz ) changes over time, the operation status image 30 may also change and be displayed accordingly. The equipment image 11, operation status image 30, and composite image 80 may be three-dimensional images.

If the communications control unit 201 receives an end code (step S406: Yes), it assumes that the assistance process has ended and stops the imaging unit 204 (step S410). Then, the process returns.

The remote support system 1 of the second embodiment of the present invention can dynamically change and present virtual objects, making it possible to intuitively communicate operation details. This can improve the work efficiency of on-site workers and work assistants. In particular, it becomes possible to accurately communicate to workers complex work details that involve the physical movements of the assistant and the operation of the equipment. Specifically, it becomes possible to accurately communicate the status and movement of the equipment to workers. Since an equipment status image can be displayed superimposed on a background that also includes the equipment image, it becomes easy to accurately communicate the status of which parts of the equipment.

Also, unlike simple chromakey compositing, there is no need for the supporter to stand and move against a background of a specific color.

Figure 20 is a schematic diagram showing an example of a composite image 90 on the work-side apparatus 3 in the third embodiment of the present invention. As shown in Figure 20, the content of the first embodiment and the content of the second embodiment can be integrated to display the composite image 90 on the work-side apparatus 3. Specifically, the work-side display unit 202 may display both the body movement image 12 in the first embodiment and the operation status image 30 in the second embodiment. In this case, the remote support system 1 may have the configurations shown in Figures 2 and 11 and the configurations shown in Figures 14 and 15. Repetitive explanations will be omitted.

Figure 21 shows an example of a computer 2200 in which aspects of the present invention may be embodied, in whole or in part. Programs installed on the computer 2200 may cause the computer 2200 to function as or perform operations associated with an apparatus or one or more sections of the apparatus according to embodiments of the present invention, and/or to perform methods or steps of methods according to embodiments of the present invention. Such programs may be executed by the CPU 2212 to cause the computer 2200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 2200 according to this embodiment includes a CPU 2212, RAM 2214, a graphics controller 2216, and a display device 2218, which are interconnected by a host controller 2210. The computer 2200 also includes input/output units such as a communications interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input/output controller 2220. The computer also includes legacy input/output units such as a ROM 2230 and a keyboard 2242, which are connected to the input/output controller 2220 via an input/output chip 2240.

The CPU 2212 operates according to programs stored in the ROM 2230 and RAM 2214, thereby controlling each unit. The graphics controller 2216 retrieves image data generated by the CPU 2212 into a frame buffer or the like provided in the RAM 2214 or into the graphics controller itself, and causes the image data to be displayed on the display device 2218.

The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides the programs or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

ROM 2230 stores therein boot programs and the like that are executed by computer 2200 upon activation, and/or programs that depend on the hardware of computer 2200. I/O chip 2240 may also connect various I/O units to I/O controller 2220 via parallel ports, serial ports, keyboard ports, mouse ports, etc.

The programs are provided on a computer-readable medium such as a DVD-ROM 2201 or an IC card. The programs are read from the computer-readable medium, installed on the hard disk drive 2224, RAM 2214, or ROM 2230, which are also examples of computer-readable media, and executed by the CPU 2212. The information processing described in these programs is read by the computer 2200, resulting in cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing information manipulation or processing in accordance with the use of the computer 2200.

For example, when communication is performed between computer 2200 and an external device, CPU 2212 may execute a communication program loaded into RAM 2214 and instruct communication interface 2222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 2212, communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in RAM 2214, hard disk drive 2224, DVD-ROM 2201, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer processing area, etc., provided on the recording medium.

The CPU 2212 may also cause all or necessary portions of a file or database stored on an external recording medium such as the hard disk drive 2224, DVD-ROM drive 2226 (DVD-ROM 2201), IC card, etc. to be read into the RAM 2214, and perform various types of processing on the data on the RAM 2214. The CPU 2212 then writes the processed data back to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. CPU 2212 may perform various types of processing on data read from RAM 2214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the program's instruction sequence, and write the results back to RAM 2214. CPU 2212 may also search for information in files, databases, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, CPU 2212 may search for an entry that matches a condition specified by the attribute value of the first attribute from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The programs or software modules described above may be stored on computer-readable media on or near computer 2200. Recording media such as a hard disk or RAM provided within a server system connected to a dedicated communications network or the Internet can also be used as computer-readable media, thereby providing the programs to computer 2200 via the network.

The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," or the like, and it should be noted that processes can be performed in any order, unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that it is necessary to perform the processes in this order.

1...Remote support system, 2...Support side device, 3...Work side device, 4...Communication network, 5...Equipment, 6...Reference point marker, 7...Virtual model, 8...Reference point, 9...Identification image, 10...Synthetic image, 11...Equipment image, 12...Body movement image, 21...Operation part, 22...Operation part, 23...Position, 24...Body movement point, 25...Handle, 30...Operation state image, 31...Image, 32...Image, 33...Handle, 35... Message, 36... Symbol, 80... Composite image, 90... Composite image, 101... Communication control unit, 102... Detection unit, 103... Support side display unit, 104... Virtual model storage unit, 105... Selection unit, 106... Motion capture device, 110... Screen control unit, 112... Sensor, 114... Microphone, 116... Speaker, 118... Audio control unit, 120... Model generation unit, 122... Model correction unit, 130... Operation part detection unit, 13 1...Communication control unit, 132...Pointing device, 201...Communication control unit, 202...Work side display unit, 204...Imaging unit, 210...Screen control unit, 212...Sensor, 214...Microphone, 216...Speaker, 218...Audio control unit, 220...Correction information generation unit, 230...Image generation unit, 231...Communication control unit, 232...Work side virtual model storage unit, 241...Marker detection unit, 242...Adjustment unit, 243...Reading unit, 2200 ...Computer, 2201...DVD-ROM, 2210...Host controller, 2212...CPU, 2214...RAM, 2216...Graphics controller, 2218...Display device, 2220...I/O controller, 2222...Communication interface, 2224...Hard disk drive, 2226...DVD-ROM drive, 2230...ROM, 2240...I/O chip, 2242...Keyboard

Claims (8)

a support-side device including a support-side display unit that displays a virtual model that imitates a facility, a detection unit that detects a body movement of a supporter relative to the virtual model, and a transmission unit that transmits information about the body movement relative to the virtual model;
a work-side device including a work-side display unit that displays a virtual image representing the body movement based on the received information about the body movement,
the detection unit detects the body motion, the body motion including position information about at least a part of the body of the support person relative to the virtual model;
the transmitting unit transmits the information about the body movement including the position information;
the work-side display unit displays a composite image in which the virtualized image is superimposed on the image of the equipment at a position corresponding to the position information ,
a reference point marker is provided at a predetermined position on the equipment;
the virtual model is provided with a reference point provided at a position corresponding to the reference point marker and a reference coordinate that is a three-dimensional coordinate based on the reference point;
the body motion detected by the detection unit includes the position information of at least a part of the body of the supporter in the reference coordinate system;
The work-side device is
a marker detection unit that detects a reference point marker in the facility;
and an adjustment unit that adjusts a position at which the virtualized image is to be combined with the image of the equipment based on a detection result of the reference point marker and the position information included in the received information on the body movement.
Support system. the body motion detected by the detection unit includes the position information of at least a part of the body of the supporter in a three-dimensional space;
The assistance system according to claim 1 . The support system according to claim 1 , wherein the support-side device further comprises a selection unit that selects the virtual model corresponding to the facility from a plurality of virtual models. the selection unit selects the virtual model corresponding to the facility from the plurality of virtual models based on an instruction from the supporter.
The assistance system according to claim 3 . An identification image for identifying the equipment is displayed on the equipment,
The work-side device includes a reading unit that reads the identification image;
an identification information transmitting unit that transmits identification information obtained by reading the identification image with the reading unit to the support-side device,
the selection unit selects a virtual model corresponding to the equipment from among a plurality of virtual models based on the identification information received from the work-side device.
The assistance system according to claim 3 . a support-side device including a support-side display unit that displays a virtual model that imitates a facility, a detection unit that detects a body movement of a supporter relative to the virtual model, and a transmission unit that transmits information about the body movement relative to the virtual model;
a work-side device including a work-side display unit that displays a virtual image representing the body movement based on the received information about the body movement,
the detection unit detects the body motion, the body motion including position information about at least a part of the body of the support person relative to the virtual model;
the transmitting unit transmits the information about the body movement including the position information;
the work-side display unit displays a composite image in which the virtualized image is superimposed on the image of the equipment at a position corresponding to the position information,
The work-side device includes an imaging unit that captures an image of the facility and obtains a facility image;
an equipment image transmission unit that transmits the equipment image to the support-side device,
the support-side device further includes a model generation unit that generates the virtual model based on the equipment image received from the work-side device.
Support system. a support-side device including a support-side display unit that displays a virtual model that imitates a facility, a detection unit that detects a body movement of a supporter relative to the virtual model, and a transmission unit that transmits information about the body movement relative to the virtual model;
a work-side device including a work-side display unit that displays a virtual image representing the body movement based on the received information about the body movement,
the detection unit detects the body motion, the body motion including position information about at least a part of the body of the support person relative to the virtual model;
the transmitting unit transmits the information about the body movement including the position information;
the work-side display unit displays a composite image in which the virtualized image is superimposed on the image of the equipment at a position corresponding to the position information,
The work-side device includes an imaging unit that captures an image of the facility and obtains a facility image;
a correction information generation unit that generates correction information based on the equipment image;
a correction information transmitting unit that transmits the correction information to the support-side device,
the support-side device further includes a model correction unit that corrects the virtual model based on the correction information received from the work-side device.
Support system. the correction information generation unit compares the virtual model received from the support-side device with the facility image to generate the correction information.
The assistance system according to claim 7 .