WO2016084142A1 - Work assistance system and work assistance method - Google Patents

Abstract

A work assistance system comprising: a camera which outputs a captured image of a target object, which is an object to be worked on, and of the surroundings thereof; a storage unit which stores a first model representing a three-dimensional model of the target object; a detection unit which detects, from the image, both a hand of a user performing the work, and the target object; a relationship determination unit which determines from the image whether the hand and the target object overlap each other; a condition determination unit which, if the hand and the target object do not overlap each other, extracts first information about the target object from the image and second information about the target object from the first model, and determines the working conditions on the basis of the results of a comparison between the first information and the second information; and an output unit which outputs information based on the working conditions.

Description

Work support system and work support method

The present invention relates to a work support system.

Conventionally, in the work of assembling parts, the position where the parts are attached is displayed superimposed on the parts in the work space using a projector, and when the individual parts are assembled, the operator depresses the switch. A technique has been proposed in which the assembly state of the part is determined from an image, and if it passes, the process proceeds to the next assembly process.

Also, using the markers attached to the work target and the tool, the position of the work target and the tool position are specified from the image, and if the position of the tool is included in the work target position, the work is performed in the correct procedure. A technique for determining that a device is broken has been proposed.

In addition, it is possible to determine from the worker's posture, position, moving direction and work procedure data whether the work that the worker is trying to carry out is correct and by detecting that the touch sensor attached to the work object is touched. Techniques for determining that work has been performed on a target and techniques for determining that work has been completed correctly by detecting changes in sensor data obtained from the work target have been proposed (for example, Patent Documents). 1).

JP 2005-250990 A

By using conventional technology, it is possible to construct a work support system that presents work procedures to the worker in order and navigates the worker.

However, the method of operating a separately prepared switch, which is a method for shifting to the next work procedure, requires an operator to perform an operation that is not performed in normal work. For this reason, such a method may hinder smooth progress of work.

On the other hand, when the method for navigating the work is a method for detecting that the work has been performed on the work object by the marker or the touch sensor, as described in the related art or Patent Document 1, the influence on the work flow. There are few possibilities. However, such a method has a problem that it is often impossible or difficult to attach a marker or a touch sensor in the case of installing or assembling a large-scale facility. Such a method can determine that some work has been performed on the work target, but cannot determine whether the work target is in the correct state.

On the other hand, by using the technique for determining the state of the work object based on the image and the technique for detecting the change in the sensor data described in Patent Document 1, it is determined that the work has been correctly completed and automatically It is possible to construct a work support system that presents information on the next work procedure.

However, when the method of constantly determining the state of the work target based on the image is used, there is a problem that the possibility that the work is erroneously determined is increased.

Also, it is considered that the method of determining the end of work based on sensor data is often difficult to install the sensor or to operate the sensor, such as installation and assembly.

An object of the present invention is to provide a work support system for presenting work procedures to a worker in order and navigating the worker at an appropriate timing without adding special devices such as markers and sensors to the work target. By determining the status of the work target and automatically switching the work content to be presented to the worker based on the determination result, the worker can be surely performed without interfering with the normal work of the worker. It is to provide a technique for effectively navigating.

In order to solve the above problems, the present invention is a work support system, which outputs a photographed image of a target object that is a work target and the periphery of the target object, and a three-dimensional view of the target object. A storage unit that stores a first model indicating a model, a detection unit that detects a hand of the user who performs the work and the object from the image, and whether the hand and the object overlap each other When the relationship determination unit for determining from the above and the hand and the object do not overlap, the first information that is the information of the object is extracted from the image, and the second information that is the information of the object is the second information A state determination unit that determines a working state based on a result extracted from one model and compares the first information and the second information; and an output unit that outputs information according to the working state. .

According to the present invention, it is possible to navigate the worker effectively so that the work can be performed reliably.

Issues, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

It is a block diagram which shows the physical structure of the work assistance system of Example 1. FIG. It is explanatory drawing which shows the operation | work in the work assistance system of Example 1. FIG. It is explanatory drawing which shows the point cloud data of the installation of the work assistance system of Example 1. FIG. It is explanatory drawing which shows the data format stored in the work procedure of Example 1. FIG. It is explanatory drawing which shows the area | region of the work target of Example 1, and a work space three-dimensional model. It is a functional block diagram which shows the process by the work assistance system of Example 1. FIG. 3 is a flowchart illustrating processing by the work support system according to the first embodiment. It is explanatory drawing which shows the point cloud data of working space data of Example 1. FIG. FIG. 3 is an explanatory diagram illustrating a three-dimensional model representing hand and arm shapes according to the first embodiment. It is explanatory drawing which shows the work space data after deleting the point cloud data corresponding to the operator's hand and arm of Example 1. FIG. It is explanatory drawing which shows the area | region of the work object in the working space data of Example 1. FIG. It is explanatory drawing which shows the screen which displayed the information regarding the work procedure and the area | region of work object on the distance image acquired by the depth sensor of Example 1. FIG. It is explanatory drawing which shows the area | region of an operator's hand of Example 1. FIG. It is explanatory drawing which shows the screen displayed after one work procedure of Example 1 is complete | finished. 10 is a flowchart illustrating processing by the work support system according to the second embodiment. FIG. 10 is an explanatory diagram illustrating an image acquired as working space data according to the second embodiment. It is explanatory drawing which shows the area | region of the operator's hand of Example 2. FIG. 12 is a flowchart illustrating processing by the work support system according to the third embodiment. 10 is a flowchart illustrating processing by the work support system according to the fourth embodiment. It is explanatory drawing which shows the space data during work when the operator of Example 4 is carrying out work while holding a tool or the like.

About the work support system of this embodiment that navigates a worker's work by presenting the work content and work object to the worker at an appropriate timing in various work such as maintenance or inspection of construction or facilities This will be described below.

Embodiment 1 of the present invention will be described with reference to FIGS.

FIG. 1 is a block diagram illustrating a physical configuration of the work support system according to the first embodiment.

FIG. 1 shows a configuration of a work support system when a general work support system executes the processing of this embodiment.

The work support system according to the present embodiment includes an information processing apparatus 101, an input apparatus 102, an output apparatus 103, a work space model 104, a work procedure 105, a normal time model 106, a storage device 107, and a work space data 115.

The information processing apparatus 101 is an apparatus for executing various programs stored in the storage device 107. The information processing apparatus 101 includes at least one processor and a memory.

The processor included in the information processing apparatus 101 is, for example, a CPU. The memory included in the information processing apparatus 101 includes a ROM that is a nonvolatile storage element and a RAM that is a volatile storage element. The ROM stores an immutable program (for example, BIOS).

The RAM is a high-speed and volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores a program stored in the storage device 107 and data used when the program is executed.

The storage device 107, the work space model 104, the work procedure 105, the normal time model 106, and the work space data 115 are large-capacity and nonvolatile storage devices such as a magnetic storage device (HDD) and a flash memory (SSD), for example. Yes, it stores a program executed by the information processing apparatus 101 and data used when the program is executed.

The input device 102 includes a device that acquires, as the working space data of the present embodiment, data obtained by photographing a work target that is an object on which the worker performs work and a space where the work target exists. The input device 102 receives an instruction from the worker and sends it to the program. The input device 102 may include an input device in a general computer such as a keyboard, a mouse, or a touch panel.

In the first embodiment, the input device 102 includes a depth sensor that acquires a distance image having a pixel value as a distance to an object. The input device 102 converts the distance image acquired by the depth sensor into point cloud data representing information of the target space as a set of points on the three-dimensional space by using the focal length and the center position of the depth sensor. The working space data in this embodiment is mainly expressed as point cloud data.

The output device 103 is a device that presents the results of processing by the work support system of the present embodiment to the worker. The output device 103 may include, for example, a monitor used in a general work support system, or may include a head-mounted display device called an HMD (head mounted display).

The work space model 104 is a storage area for storing a three-dimensional work space model that represents the entire space in which work is performed. The work space three-dimensional model may be represented by point cloud data in the work space model 104 in the same manner as the work space data.

It is desirable for general work support systems to store information related to multiple works. The work support system of this embodiment also uses a different work space three-dimensional model for each work. For this reason, the work space model 104 stores point cloud data indicating the work space three-dimensional model and an identifier (for example, name) for identifying the work in which the work space three-dimensional model is used in association with each other.

Note that the work space model 104 may store information on a three-dimensional model using a polygon (polygon) generally used in CG or the like. The program that accesses the work space model 104 may divide the polygon in the work space model 104 until it reaches a predetermined size, and extract only the coordinates of the vertexes of each polygon after the division. Thus, the program can easily convert the polygon representing the three-dimensional model into point cloud data indicating the extracted coordinates.

The work procedure 105 includes a series of work procedures for carrying out the work, information for specifying the work object that is the object to be worked, and information on the position of the work object in the work space three-dimensional model. The work procedure 105 may include information indicating the state of the work target such as the size, shape, color, and temperature of the work target.

The normal-time model 106 includes a normal-time object (three-dimensional) model that represents an ideal state (position, size, shape, color, temperature, and the like) after the work of each work object indicated by the work procedure 105. The three-dimensional model stored in the normal-time model 106 may be expressed by point cloud data, like the working space data and the working space three-dimensional model.

It is considered that the ideal state of a 3D model corresponding to a certain work target may differ depending on the work procedure even during the same work. For this reason, the normal model 106 includes point cloud data representing a work target and information related to related work and work procedures (a work name 401 described later, work procedure areas 403 and 407, and the like), Hold in correspondence.

The normal model 106 may hold a three-dimensional model using polygons. A program that accesses the normal model 106 may convert the polygon into point cloud data by the method described above.

The working space data 115 includes working space data indicating the working space acquired during the work. The working space data 115 includes point cloud data indicating the shape, size, and position of an object in the working space, and may include data indicating the temperature of the object in the working space.

The storage device 107 includes an information presentation program 108, an input program 113, a correspondence calculation program 114, a hand detection program 109, a work target detection program 110, a relationship determination program 111, and a state determination program 112. The information presentation program 108 is a program that controls other programs held in the storage device 107 and presents information to the worker according to the contents of the work procedure 105.

The input program 113 is a program for acquiring working space data using the input device 102 after the work starts. The correspondence calculation program 114 is a program for obtaining a correspondence between the work space data and the work space three-dimensional model. The hand detection program 109 is a program for detecting an area of the worker's hand from the working space data.

The work target detection program 110 is a program that detects a work target area in the working space data based on the correspondence obtained by the correspondence calculation program 114.

The relationship determination program 111 determines whether the work target area and the hand area overlap. Specifically, for example, the relationship determination program 111 includes an area of the operator's hand detected by the hand detection program 109 within a predetermined position including the area of the work target detected by the work target detection program 110. Are determined by determining whether they exist.

When it is determined that the work target area and the hand area do not overlap, the state determination program 112 extracts the data of the work target area from the working space data, and further, the normal target object corresponding to the work target The model is extracted from the normal model 106.

Then, the state determination program 112 compares the extracted work area data with the extracted normal object model to determine whether the work object is in an appropriate state, that is, whether the work has been properly completed. Determine.

FIG. 2 is an explanatory diagram showing work to which the work support system of the first embodiment is applied.

The work support system shown in FIG. 2 includes the work support system shown in FIG. 1 and the work target on which the worker works. The work support system shown in FIG. 2 includes a terminal 201, a touch panel 202, a depth sensor 203, a helmet 204, an HMD 205, equipment 206, and a lever 207.

The terminal 201 is a portable terminal that presents information regarding work to the worker or accepts instructions from the worker in the work support system. The terminal 201 may be any device as long as it has an input device and an output device, and may be a tablet terminal or a smartphone.

The work support system shown in FIG. 1 may be implemented by the terminal 201, the depth sensor 203, and the HMD 205. A computer connected to the terminal 201 by wire or wireless implements the information processing apparatus 101, the storage device 107, the work space model 104, the work procedure 105, the normal model 106, and the work space data 115 shown in FIG. May be.

The terminal 201 includes, for example, a touch panel 202 as the input device 102 and the output device 103. The touch panel 202 is operated by an operator. The worker controls the operation of the work support system by inputting an instruction to the touch panel 202.

For example, the worker operates the touch panel 202 when starting or ending work in the work support system, or when returning to the original work procedure when the work support system erroneously shifts to the next work procedure. .

The depth sensor 203 measures the distance between the work target and an object around the work target and the depth sensor 203. The depth sensor 203 acquires the measurement result as working space data, and the input program 113 stores the obtained working space data in the working space data 115. The depth sensor 203 is installed in the worker or in the vicinity of the worker.

The depth sensor 203 shown in FIG. 2 is installed in the helmet 204, so that it is possible to acquire information on an object existing in the worker's field of view including the work target and the worker's hand from the worker's viewpoint. And since the state determination program 112 of a present Example can determine the state of work based on the work object recognized by the user, it can determine the state of work accurately.

The depth sensor 203 may be installed at any position as long as the shape of the object existing in the space including the work target and the worker's hand can be acquired. For this reason, the depth sensor 203 may be installed in the HMD 205. Alternatively, the depth sensor 203 may be installed at any position on the worker's body as long as it can acquire information on the space including the work target, such as the worker's shoulder or chest. Further, the depth sensor 203 may be installed on a ceiling or the like near the worker.

The HMD 205 is a head-mounted display device. The HMD 205 corresponds to the output device 103 in FIG.

The facility 206 is a device or a structure having a work target. The worker works on a work target included in the facility 206 in order to perform maintenance or change of the facility 206. In the facility 206, a lever 207, a meter 208, a meter 209, a switch 210, a switch 211, a lamp 212, a lamp 213, a lamp 214, and the like, which are work objects, are arranged.

In the work space model 104, point cloud data indicating the shape of the facility 206 is stored in advance. The input program 113 and the input device 102 (depth sensor 203) may acquire point cloud data indicating the shape of the facility 206 in advance before work and store it in the work space model 104. The administrator of the work support system may set point cloud data indicating the shape of the facility 206 in the work space model 104 in advance before work.

FIG. 3 shows an image when a three-dimensional work space model expressing the facility 206 as point cloud data is arranged in the three-dimensional space.

FIG. 3 is an explanatory diagram illustrating the point cloud data 301 of the facility 206 of the work support system according to the first embodiment.

The point cloud data 301 shown in FIG. 3 represents the shape of the facility 206 by a set of points in a three-dimensional space. The point cloud data 301 shown in FIG. 3 represents only information regarding the position (coordinates) of each point in the three-dimensional space. However, the depth sensor 203 may acquire the color information or temperature information of each point together, and the input device 102 may store the point cloud data 301 and the color information or temperature information in the work space model 104.

At this time, when the depth sensor 203 acquires the point cloud data 301, the depth sensor 203 acquires a color image or a temperature image taken from the same direction, and the correspondence between the color image and the point cloud data 301 or the temperature image and the point cloud data 301. The color information or temperature information of each point in the point cloud data 301 may be acquired on the basis of the correspondence relationship with the

FIG. 4 is an explanatory diagram showing a data format stored in the work procedure 105 of the first embodiment.

The work procedure 105 includes a name 401 and areas 402 to 410.

Name 401 indicates an identifier such as a name given to the work. The work of this embodiment includes a plurality of work procedures. The identifier of the name 401 is expressed by an arbitrary character string, for example. The work procedure 105 indicates a plurality of work procedures included in a plurality of works. The information presentation program 108 presents a work name 401 to the worker in order to select a work to be started.

The area 402 indicates the number of work procedures included in the work indicated by the name 401. Areas 403 to 406 include information on the first work procedure included in the work indicated by the name 401, and areas 407 to 410 include information on the last work procedure included in the work indicated by the name 401.

The area 403 and the area 407 include an identifier (for example, a name) assigned to each work procedure. The identifiers of the area 403 and the area 407 are expressed by an arbitrary character string, for example.

The area 404 and the area 408 are information indicating detailed work contents of each work procedure, and may be expressed by a character string that can be understood by the worker. Further, the area 404 and the area 408 may include an image, a moving image, or CG indicating the work content. In addition, the facility 206 including the work target may be reflected in the image or the like indicated by the region 404 and the region 408.

The area 405 and the area 409 include work target identifiers (for example, names) in each work procedure. When the worker operates a plurality of work targets according to one work procedure, the area 405 and the area 409 may include a plurality of work target identifiers.

The area 406 and the area 410 indicate the position coordinates on the work space three-dimensional model of the work object indicated by the area 405 and the area 409. An area 406 and an area 410 indicate an identifier (for example, name) of the work space three-dimensional model including the work object, and a three-dimensional area of the work object on the work space three-dimensional model.

For example, the area 406 and the area 410 include information on the area of the work target expressed as a combination of the center position of the cuboid circumscribing the work target on the work space three-dimensional model and the sizes in the x, y, and z axis directions. It may be held. In addition, the area 406 and the area 410 may hold information on the work target area expressed as a combination of an angle representing rotation of a cuboid circumscribing the work target and a rotation matrix.

Furthermore, when the work target has a complicated shape, the work target area may be expressed by a combination of a plurality of rectangular parallelepipeds or a combination of a rectangular parallelepiped and another shape such as a cylinder or a sphere. As described above, the contents of the region 406 and the region 410 may be expressed by any method as long as the method can express the three-dimensional region to be worked on the work space three-dimensional model.

FIG. 5 is an explanatory diagram illustrating a work target area and a work space three-dimensional model according to the first embodiment.

FIG. 5 is an image obtained by superimposing and displaying the work target area registered in the work procedure 105 on the work space three-dimensional model of FIG.

The work target area shown in FIG. 5 is represented by a rectangular parallelepiped. A region 501 shown in FIG. 5 is a region corresponding to the lever 207 shown in FIG. Regions 502 and 503 correspond to the meters 208 and 209, regions 504 and 505 correspond to the switches 210 and 211, and regions 506, 507, and 508 correspond to the lamps 212, 213, and 214.

FIG. 6 is a functional block diagram illustrating processing by the work support system according to the first embodiment.

The work support system according to the first embodiment includes a work space data input unit 2001, a work space data-work space three-dimensional model correspondence calculation unit 2002, a work information presentation unit 2003, a work target region detection unit 2005, and a hand detection unit 2004. And a functional unit such as a work target-hand relationship determination unit 2006 and a work target state determination unit 2007. The functional units included in the work support system according to the first embodiment may be implemented by each program illustrated in FIG. 1 or may be implemented by a physical device.

The working space data input unit 2001 acquires the shape around the work target being worked as working space data. The working space data-working space three-dimensional model correspondence calculating unit 2002 compares the acquired working space data with the working space three-dimensional model, thereby obtaining data indicating the shape of the facility 206 including the work object. Extract.

On the other hand, the hand detection unit 2004 detects an area of the worker indicating the shape of the worker's hand from the acquired work space data. The work target area detection unit 2005 detects a work target area indicating the shape of the work target based on the work procedure 105 from the data of the facility 206 including the work target.

The work target-hand relationship determination unit 2006 determines the positional relationship between the detected hand region and the detected work target region. If the work target-hand relationship determining unit 2006 determines that the detected hand region does not overlap the detected work target region as a result of the determination, the work target state determining unit 2007 determines that the work target state determining unit 2007 uses the working space data. The target data is extracted, and further, the normal object model indicating the correct shape of the work object after the operation is extracted from the normal model 106. Then, the work target state determination unit 2007 compares the extracted work target data with the extracted normal object model.

Then, as a result of the comparison, when the work target state determination unit 2007 determines that the work target area has an appropriate shape, the work information presentation unit 2003 outputs that the work has been properly completed.

The details of the processing when the processing shown in FIG. 6 is executed by the program shown in FIG. 1 will be described below.

FIG. 7 is a flowchart illustrating processing by the work support system according to the first embodiment.

First, the information presentation program 108 presents a list of work that can be performed to the worker, and causes the worker to select the work to be performed (601). Specifically, the information presentation program 108 extracts at least one work name 401 from the work procedure stored in the work procedure 105, and outputs the extracted at least one name as a work list via the output device 103. Output.

In addition, the information presentation program 108 accepts the name of the work selected using the keyboard, mouse, touch panel, or the like included in the input device 102.

After step 601, the information presentation program 108 acquires at least one work procedure corresponding to the work name selected in step 601 from the work procedure 105, and further, the work space tertiary corresponding to the selected work name. An original model is acquired from the workspace model 104 (602).

After step 602, the information presentation program 108 sets the first work procedure in the obtained work procedure as an execution target (603). Specifically, the information presentation program 108 outputs the first work procedure via the output device 103 in step 603. Thus, the worker can recognize the first work procedure.

The work procedure output here is the content indicated by the area 404 of the work procedure 105, and may be an image or a moving image showing the facility 206.

After step 603, the input program 113 acquires the point cloud data of the working space data via the depth sensor 203 provided in the input device 102 (604). The input program 113 stores the acquired point cloud data in the working space data 115.

FIG. 8 is an explanatory diagram showing point cloud data of the working space data of the first embodiment.

The working space data shown in FIG. 8 is point cloud data that three-dimensionally shows the shape of an object existing in the space including the work target being worked and the hand of the worker. The point cloud data shown in FIG. 8 includes point cloud data 701 corresponding to a part of the work space three-dimensional model, and point cloud data 702 and 703 indicating the shape of the operator's hand.

Here, the origin of the working space data is the position of the depth sensor. In general, the work space data acquired by the input program 113 and the work space three-dimensional model stored in the work space model 104 are expressed in different coordinate systems.

The depth sensor of the first embodiment acquires point cloud data that is a three-dimensional model to be worked, so that the state determination program 112 can accurately determine the state of the work in the process described later.

After step 604, the hand detection program 109 detects the position and posture of the operator's hand and arm from the working space data acquired in step 604 (605). The hand detection program 109 may use any method as long as it detects the position and orientation of a specific shape from the point cloud data as a method for detecting the hand and arm.

For example, the hand detection program 109 may hold in advance a three-dimensional model representing the shape of the hand and arm during work as shown in FIG. The hand detection program 109 detects the position and posture of the operator's hand and arm by detecting the position where the pre-held three-dimensional model of the hand and arm and the work space data most closely match. You may do it.

FIG. 9 is an explanatory diagram showing a three-dimensional model 801 representing the hand and arm shapes of the first embodiment.

A three-dimensional model 801 shown in FIG. 9 is a three-dimensional model using a polygon showing the shape of the right hand. The hand detection program 109 may divide such polygons, and convert and hold a three-dimensional model of hands and arms into point cloud data in advance by the method described above.

In addition, the hand detection program 109 is, for example, an ICP (Iterative Closest Point) algorithm (method for detecting a position where the two point cloud data (the three-dimensional model 801 and the working space data in this embodiment) best match). S. Rusinkiewicz and M. Levoy, "Efficient Variants of the ICP Algorithm", Third International Conference on 3D Digital Imaging and Modeling, 2001), or, NDT (Normal Distributions Transformation) (P. Biber and W. Straser, "Th e Normal Distributions Transform: A New Approach to Laser ScanMatching Proceedings of the 2003, IEEE / RSJ International ConfencementInsertient3, etc.

In step 605, the hand detection program 109 uses these methods to convert the hand and arm three-dimensional model 801 so as to match the working space data, that is, a working matrix, that is, working. The position and orientation of the three-dimensional model 801 in the spatial data are obtained.

Further, when these methods are used, the hand detection program 109 needs to acquire an initial position when performing alignment. When it is predicted in advance that the working space data shown in FIG. 8 can be obtained, for example, in the coordinate system of the point cloud data, the right elbow side is arranged in the lower right front and the right hand is arranged in the upper left back. For this reason, the hand detection program 109 may acquire in advance the average position and posture of the predicted hand and arm as the initial position.

Alternatively, the hand detection program 109 sets a plurality of representative positions and postures as initial positions, and based on the result of the best match between the working space data and the three-dimensional model of the hands and arms, You may seek posture.

Further, the hand detection program 109 may hold a three-dimensional model of the left hand in addition to the three-dimensional model 801 shown in FIG. The hand detection program 109 can detect the positions and postures of both hands and arms of the worker in the working space data by using the above-described method used for the three-dimensional model 801 of the right hand.

After step 605, the correspondence calculation program 114 deletes the point cloud data corresponding to the hand and arm detected in step 605 from the working space data. Here, the correspondence calculation program 114 can detect any position and posture of the hand and arm from the working space data, and can be deleted by specifying the point cloud data corresponding to the hand and arm. A method may be used.

The correspondence calculation program 114 converts the three-dimensional model 801 of hands and arms into point cloud data in the working space data, for example, using the conversion matrix for performing rotation and translation obtained in step 605. Thereafter, the correspondence calculation program 114 uses a set of points in the working space data within a predetermined distance from each point of the converted point cloud data as point cloud data corresponding to the hands and arms of the worker. Delete from medium space data.

FIG. 10 is an explanatory diagram showing the working space data after deleting the point cloud data corresponding to the hand and arm of the worker of the first embodiment.

The correspondence calculation program 114 leaves the point cloud data 701 indicating the facility 206 in the working space data shown in FIG. 10, but deletes the point cloud data from the areas 902 and 903 where the point cloud data 702 and 703 existed.

Further, the correspondence calculation program 114 obtains a transformation matrix representing the correspondence between the working space data and the work space three-dimensional model after removing the point cloud data corresponding to the operator's hand and arm (606). The correspondence calculation program 114 can obtain information indicating the correspondence by obtaining a position where the working space data and the work space three-dimensional model most closely match.

The correspondence calculation program 114 obtains the position and orientation in which the two three-dimensional data are best matched as a method for obtaining the correspondence between the working space data and the work space three-dimensional model, and information indicating the obtained position and orientation. Any method may be used as long as the information is acquired as information indicating the correspondence.

The correspondence calculation program 114 may use, for example, the aforementioned ICP algorithm or NDT. Then, the correspondence calculation program 114 may obtain a conversion matrix that rotates and translates the work space data so as to best match the three-dimensional work space model as information indicating the correspondence.

After step 606, the work target detection program 110 uses the information indicating the correspondence relationship between the work space data obtained in step 606 and the work space three-dimensional model, and the work object corresponding to the work procedure being executed ( The area in the working space data (corresponding to the area 405 of the work procedure 105) is obtained (607).

The work object detection program 110 obtains the work space three-dimensional model as a transformation matrix that rotates and translates so as to best match the work space data, and is then placed on the work space three-dimensional model shown in FIG. Read the work area. Then, the work target detection program 110 converts the read work target area using the obtained conversion matrix. Thereby, the work target detection program 110 can obtain a work target area in the working space data.

FIG. 11 is an explanatory diagram showing work target areas in the working space data of the first embodiment.

11 are areas corresponding to the area 501 and the area 502 in FIG. Areas 1001 and 1002 are work target areas in the working space data.

After step 607, the information presentation program 108 outputs the information indicating the area in the working space data of the work target obtained in step 607 together with the output work procedure via the output device 103 (608). .

FIG. 12 is an explanatory diagram showing a screen 1100 that displays information on the work procedure and the work target area on the distance image acquired by the depth sensor 203 of the first embodiment.

12 displays the distance image acquired by the depth sensor 203, the contents of the work procedure (the area 404 of the work procedure 105, etc.) and the work target area. The screen 1100 includes an area 1101 and an area 1102. An area 1101 displays the contents of the work procedure being executed as a character string. An area 1102 displays a work target area in the work procedure being performed.

The area 1102 shown in FIG. 12 only shows the work target area. However, when the work procedure 105 also stores information such as a symbol indicating an operation method (including a graphic such as an arrow or a circle) and an image or a moving image, the information presentation program 108 may display a symbol such as an arrow or a circle at the work target position. Further, an image or a moving image may be displayed in the area 1102 or in the vicinity of the area 1102.

When the input device 102 uses the depth sensor 203 that can acquire a color image and a distance image from the same direction, the information presentation program 108 obtains a conversion matrix for converting the position on the distance image into the position on the color image. Also good. Then, the information presentation program 108 may display the area 1101 and the area 1102 on the color image by using this conversion matrix.

In addition, when the output device 103 uses an HMD of a type that can directly see the scenery in front, the information presentation program 108 may display only the area 1101 and the area 1102 on the HMD.

After step 608, the relationship determination program 111, based on the area of the operator's hand in the work space data detected in step 605 and the work target area in the work space data obtained in step 607, It is determined whether the operator's hand overlaps the work target (609).

Here, the relationship determination program 111 is close to the three-dimensional model of the hand portion of the point cloud data corresponding to the hand and the arm deleted from the working space data in the working hand space data. Only the point cloud data determined in step 606 is extracted as point cloud data corresponding to the hand. Then, the relationship determination program 111 sets the extracted point cloud data area as point cloud data of the worker's hand area in the working space data.

FIG. 13 is an explanatory diagram illustrating an area of the hand of the worker according to the first embodiment.

An area 1201 shown in FIG. 13 is an area corresponding to the hand. An area 1001 shown in FIG. 13 is the work target area obtained in step 607.

For example, the relationship determination program 111 obtains a rectangular parallelepiped circumscribing the point cloud data corresponding to the hand as shown in an area 1201 in FIG. 13, and determines the area 1201 as the hand area. In step 609, the relationship determination program 111 determines whether the operator's hand overlaps the work target by determining whether the area 1201 of the obtained worker's hand exists in the work target area 1001. Determine.

Further, the relationship determination program 111 according to the present embodiment, when the operator's hand area 1201 exists within a predetermined position range including the work target area 1001, the operator's hand overlaps the work target. You may judge.

After step 609, the relationship determination program 111 determines whether the area 1201 of the operator's hand overlaps the area 1001 to be worked according to the determination result of step 609 (610). When it is determined that the worker's hand area 1201 overlaps the work target area 1001, the relationship determination program 111 needs to determine that the worker is performing the work and determine the work state. Judge that there is no. Then, the processing shown in FIG.

If it is determined in step 610 that the area 1201 of the worker's hand does not overlap the area 1001 to be worked, the relationship determination program 111 causes the worker to finish one of the work procedures, and this time It is determined that the timing is to be determined. Then, the relationship determination program 111 causes the state determination program 112 to execute step 611.

Step 609 and Step 610 are executed, the positional relationship between the worker's hand and the work target is determined, and when the worker's hand is separated from the work target, Step 611 is executed, whereby the relationship of Embodiment 1 The determination program 111 can efficiently extract an appropriate timing for determining the state of the work target.

In step 611, the state determination program 112 extracts point cloud data corresponding to the work target from the work space data based on the work area 1001 in the work space data obtained in step 607. Further, the state determination program 112 extracts from the normal-time model 106 a normal-time object (three-dimensional) model representing an ideal state of the work target after work according to the work procedure set in step 603 or 614. To do.

In step 611, the state determination program 112 calculates a difference between the point cloud data of the work target in the working space data and the acquired normal object (three-dimensional) model. The state determination program 112 may use any method as long as it calculates the difference between the point cloud data to be worked and the normal object model in step 611. A specific method is shown below.

When the normal model 106 holds a normal object model having the same position and orientation as the point cloud data indicating the work target area in the work space three-dimensional model, the state determination program 112 acquires the normal object acquired. A difference in distance between each point of the point cloud data of the model and the point cloud data of the work target closest to them is obtained. And the state determination program 112 calculates | requires the sum total of the calculated | required several difference as a difference of the point cloud data of the work object in work space data, and the acquired normal time object model.

If the position and orientation of the work target held by the work space model 104 and the position and posture of the normal object model held by the normal time model 106 are different, the administrator or the like A normal object model corresponding to an area obtained by enlarging a work target area in the space model 104 by a predetermined ratio is stored in advance. Then, the state determination program 112 acquires the point cloud data of the work target area enlarged by the ratio from the work space data 115 as the work target point cloud data.

Then, the state determination program 112 uses the method described in step 605 and step 606 to align the positions of the normal object model and the point cloud data of the work target. The state determination program 112 converts one point cloud data so as to best match the other point cloud data using the result of matching the positions.

Thereafter, the state determination program 112 calculates the difference in the distance between the point cloud data of the work target and the point cloud data of the normal object model using the above-described method of calculating the sum of the distances of the points, The difference between the work target area and the normal object model may be obtained.

In step 611, the state determination program calculates the similarity between the point cloud data of the work target and the ideal normal three-dimensional model as the difference in distance between the point cloud data of the work target and the ideal normal three-dimensional model. You may ask for.

When the obtained difference is larger than the predetermined threshold, the state determination program 112 determines in step 611 that the state of the work target is not appropriate. If the obtained difference is equal to or smaller than the predetermined threshold, the state determination program 112 determines in step 611 that the state of the work target is appropriate.

State determination by determining the state of the work target based on the difference between the 3D model of the work target extracted from the working space data and the 3D model of the work target extracted from the normal 3D model The program 112 can accurately determine whether the work state is appropriate.

After step 611, the state determination program 112 determines whether the state of the work target is appropriate according to the determination result in step 611 (612). If the work target state is appropriate, one work procedure has been completed without any problem, and the information presentation program 108 executes step 613.

If the work target state is not appropriate, the processing in FIG. 7 returns to Step 604. When returning from step 612 to step 604, the information presentation program 108 may use the output device 103 to present information indicating that the work has not been completed by a message or a symbol to the worker.

In step 613, the information presentation program 108 uses the output device 103 to present the worker with information indicating that the work procedure being performed has been completed. For example, the information presentation program 108 may present the message 1301 of FIG. 14 to the worker.

FIG. 14 is an explanatory view showing a screen displayed after one work procedure of the first embodiment is completed.

The message 1301 shown in FIG. 14 is displayed in step 613 and indicates that one work procedure has been completed without any problem.

After step 613, the information presentation program 108 refers to the work procedure 105 and sets the next work procedure as the work procedure to be executed (614). In step 614 or when returning from step 612 to step 604, by outputting whether or not the work procedure has been completed, the information presentation program 108 determines whether or not the worker can appropriately perform a series of work. The state of the work to be shown can be presented to the worker. Thus, the work support system according to the first embodiment can appropriately navigate the worker.

After step 614, the information presentation program 108 checks whether all work procedures in the work procedure to be executed have been completed (615). If all work procedures have not been completed, the next work procedure is executed. Therefore, step 604 is executed. When all work procedures are completed, the information presentation program 108 executes Step 616.

In step 616, the information presentation program 108 uses the output device 103 to present information indicating that all work procedures have been completed to the worker. For example, the information presentation program 108 may present a message such as “work has been completed” to the worker. After presenting the information to the worker, the information presentation program 108 ends the process shown in FIG.

In the first embodiment described above, the output device 103 may include a device that outputs sound or sound such as a speaker in addition to a display device such as a monitor or HMD generally used in a computer. Then, the information presentation program 108 may present information to the worker by sound or voice.

In step 609, the relationship determination program 111 may determine whether the worker's hand exists in the work target area related to the work procedure that is not being executed. When the relationship determination program 111 determines that the work target and the hand do not overlap and the worker's hand exists in the work target area related to the work procedure different from the work procedure currently being executed, The presentation program 108 may use an output device 103 to present a warning to the worker indicating that work is being performed on a different work target.

As a result, the information presentation program 108 can cause the user to review the work procedure and appropriately navigate the work.

In step 610, the relationship determination program 111 may determine whether the hand region overlaps the work target region based on the results of steps 604 to 610 executed a plurality of times in the past.

Specifically, the relationship determination program 111 may determine whether the worker's hand exists in the work target area by executing Steps 604 to 609 a plurality of times at predetermined time intervals. In this case, the relationship determination program 111 indicates that the worker's hand exists in the work target area for a predetermined time, and then the worker's hand does not exist in the work target area for a predetermined time. If it is determined, it may be determined in step 610 that the hand region does not overlap the work target region.

This prevents the relationship determination program 111 from erroneously determining that the work procedure has been completed even when the worker speaks a hand from the work object accidentally for a short time during the work procedure. The work state can be determined by the state determination program 112 at an appropriate timing.

Further, in step 610 described above, the relationship determination program 111 may identify whether the hand is one hand or both hands. Specifically, when the relationship determination program 111 determines that both hands are present in the work target area before a certain time and one hand is not present in the work target area after that time, It may be determined that the work has been completed, and step 611 may be executed.

By determining whether or not the hand overlaps the work target based on the positions of both hands and one hand, the relationship determination program 111 can be used, for example, in a work where it is necessary to always support the work target with one hand. In addition, it is possible to extract an appropriate timing for determining the work state.

In step 610, when the relationship determination program 111 determines that the predetermined time has passed, the information presentation program 108 determines that the operator's hand is present in the work target area. , Information indicating warning or attention indicating that the work is delayed or the like may be presented to the worker.

In step 611, the state determination program 112 extracts the point cloud data corresponding to the work target from the work space data based on the work target area in the work space data obtained in step 607. The point cloud data may be stored in a database such as the storage device 107 in association with the position of the work target on the work space model 104 and the work target in the work procedure 105.

Thereby, since the work support system according to the first embodiment holds data representing a time-series change regarding each work target, for example, when the work to be performed is an inspection of the work target, the administrator can change the work target over time. Etc. can be confirmed.

In addition, when an error occurs in the determination result in step 612 and the work procedure does not proceed even though the work procedure being executed has been completed, or the work procedure being executed has not been completed. Regardless, if the work procedure has proceeded next, the information presentation program 108 may accept a correct work procedure input by the operator via a keyboard, mouse, touch panel, or the like. Then, the information presentation program 108 may set the accepted work procedure.

Further, as a method of shifting to the correct work procedure, the information presentation program 108 acquires the movement of the worker's hand with the depth sensor, and when the worker's hand makes a predetermined movement, the information presentation program 108 shifts to the correct work procedure. May be.

In step 611 described above, it is determined whether the difference between the point cloud data of the work target extracted from the working space data and the normal object model is larger than a predetermined threshold value. The information content to be presented to the operator may be changed based on the magnitude of the difference.

For example, when the difference obtained in step 611 is smaller than the predetermined first threshold, the information presentation program 108 indicates that the work procedure being executed has been completed (such as “XX procedure has been completed”). May be presented to the operator. If the difference obtained in step 612 is smaller than the predetermined second threshold (> first threshold), the information presentation program 108 indicates that it is close to the correct state (such as “turn a little more”). May be presented to the operator.

In this case, the work procedure 105 may hold the threshold value according to the work content and the content presented to the worker in advance for each work procedure.

Further, as a result of executing Step 604 to Step 611 a plurality of times, when a predetermined time has elapsed since the time when the state of the work target is not determined to be appropriate, the information presentation program 108 uses the output device 103 to The operator may be presented with a warning or a reminder indicating that is delayed. Thereby, the information presentation program 108 can appropriately navigate the worker.

According to the first embodiment, the work support system determines whether one work procedure in a continuous work is finished by determining whether the work target area and the hand area overlap each other. To do. For this reason, the work support system according to the first embodiment can determine the state of the work target at an appropriate timing, and can effectively switch the work procedure information presented to the worker. As a result, it is possible to navigate effectively so that the worker can reliably perform the work.

Further, since the work support system according to the first embodiment determines the state of the work target based on the three-dimensional point cloud data, it is possible to improve the determination accuracy.

Embodiment 2 of the present invention will be described with reference to FIGS.

In the first embodiment, a depth sensor capable of acquiring three-dimensional data is used as the means for inputting the working space data in the input device 102. However, in the second embodiment, a general image is acquired as the means for inputting the working space data. Use the camera that you want. That is, the work support system according to the second embodiment includes a camera instead of the depth sensor 203, and the camera according to the second embodiment acquires the work space data as a two-dimensional image.

The configuration in Example 2 is the same as in FIG. However, the work space three-dimensional model held by the work space model 104 is a three-dimensional model using polygons (polygons) generally used in CG or the like, and the normal model 106 is an ideal work target. Holds an image representing the state. Further, the process executed by the state determination program 112 from the information presentation program 108 is the process shown in FIG.

FIG. 15 is a flowchart illustrating processing by the work support system according to the second embodiment.

In the flowchart shown in FIG. 15, steps 601 to 603, step 608, step 610, and steps 612 to 616 are the same processing as in the first embodiment. The processing shown in FIG. 15 is different from the processing shown in FIG. 7 in steps 1401 to 1406 in the second embodiment.

Hereinafter, step 1401 to step 1406 will be described. In step 1401, first, the input program 113 acquires an image which is working space data using a camera.

FIG. 16 is an explanatory diagram showing an image acquired as working space data of the second embodiment.

The image shown in FIG. 16 is acquired by the camera of Example 2 and shows the working space data. The image shown in FIG. 16 includes an equipment 1501 and an image 1502 and an image 1503 of the worker's hand.

After step 1401, the hand detection program 109 detects the operator's hand and arm from the working space data acquired in step 1401 (1402). The method for detecting the hand and arm in step 1402 may be any method as long as the hand detection program 109 recognizes a region having a specific shape from the image.

For example, the hand detection program 109 may detect the area of the operator's hand and arm from the working space data by detecting an area having the same color as the hand and arm. Alternatively, the hand detection program 109 holds a plurality of representative template images related to the hand and arm of the worker in advance and superimposes the template image on the work space data while changing the position and rotation angle in the template image. The area of the operator's hand and arm may be detected from the working space data by obtaining the template image that best matches the working space data and its position and rotation angle.

After step 1402, the correspondence calculation program 114 deletes the detected hand and arm regions from the working space data (1403). Specifically, when the hand detection program 109 detects an area having the same color as the hand and arm as the worker's hand and arm, the correspondence calculation program 114 is detected in the working space data. The hand and arm regions may be deleted by painting all the pixels in the hand and arm regions with a single color (for example, black).

Further, when the hand detection program 109 detects an area of the hand and arm of the worker using the template image, the correspondence calculation program 114 displays the template image when the template image is superimposed on the working space data. The hand and arm regions may be deleted by painting all the pixels of the region in the working space data to be overlaid with a single color.

In step 1403, the correspondence calculation program 114 further obtains a correspondence between the work space data after deleting the hand and arm regions of the worker and the work space three-dimensional model. The correspondence calculation program 114 may obtain the correspondence by using a method of aligning positions so that the image of the work space data and the work space three-dimensional model best match.

For example, the correspondence calculation program 114 uses the edge information extracted from the image and the CG model (V. Lepetit, L. Vacchetti, D. Thallmann and P. Fua, “Fully Automated and Stable Registration Registration”. The correspondence relationship may be obtained by using ISMAR 2003, pp. 93-102, 2003).

When this method is used, the correspondence calculation program 114 extracts edge information even in the vicinity of the boundary between the deleted worker's hand and arm regions, and therefore excludes such extracted edge information. . In addition, the correspondence calculation program 114 similarly uses the region of the operator's hand and arm and the boundary when using another method as a method for obtaining the correspondence between the work space data and the work space three-dimensional model. The correspondence is obtained after excluding the extracted information.

When the correspondence is obtained by the method as described above, the correspondence calculation program 114 obtains, as the correspondence, a transformation matrix that performs rotation and translation of the work space three-dimensional model so as to best match the work space data. .

After step 1403, the work target detection program 110 uses the transformation matrix representing the correspondence between the work space data obtained in step 1403 and the work space three-dimensional model to perform each work related to the work procedure being performed. The target area in the working space data is obtained (1404). That is, the work object detection program 110 according to the second embodiment obtains a corresponding area on the image from the three-dimensional area defined on the work space three-dimensional model in step 1404.

When the work target area on the work space three-dimensional model is registered as a cube as shown in FIG. 5, the work target detection program 110 uses the perspective projection conversion using the conversion row example obtained as a result of step 1403. To convert the coordinates of each vertex of the cube to a position on the image. Then, the work target detection program 110 obtains a region on the work target image by connecting the positions of the converted vertices on the image with straight lines based on the relationship between the vertices.

Furthermore, even if the area of the work target on the work space three-dimensional model has an arbitrary shape, the work target detection program 110 selects a plurality of characteristic points at edges or vertices on the area, The original position is converted to a position on the image by perspective projection conversion. Then, the work object detection program 110 obtains an area on the work object image by obtaining an area represented by connecting the converted points.

After step 1404, the information presentation program 108 executes step 608. After step 608, the relationship determination program 111, based on the area of the operator's hand in the working space data detected in step 1402 and the work target area in the working space data obtained in step 1404, As in step 609 in the first embodiment, it is determined whether the worker's hand overlaps the work target (1405).

Specifically, in step 1405, the relationship determination program 111 determines whether the area of the worker's hand exists in the work target area or a predetermined position range including the work target area. To do.

The relationship determination program 111 selects only the pixel corresponding to the hand portion from the pixels inside the hand and arm regions deleted from the work space data in step 1403, and the area of the worker's hand in the work space data. Choose as.

FIG. 17 is an explanatory diagram illustrating an area of the hand of the worker according to the second embodiment.

An area 1601 shown in FIG. 17 indicates a hand area. An area 1602 shown in FIG. 17 is a work target area.

For example, the relationship determination program 111 obtains a rectangle circumscribing the pixel determined to correspond to the hand, and determines the rectangular area as the hand area. In step 1405, the relationship determination program 111 determines whether the obtained area 1601 of the worker's hand overlaps the work target area 1602.

The relationship determination program 111 executes step 610 thereafter. In Step 610 of the second embodiment, when it is determined that the area 1601 of the worker's hand does not overlap the area 1602 to be worked, the state determination program 112 starts Step 1406.

In step 1406, the state determination program 112 extracts an image corresponding to the work target from the work space data based on the work target area in the work space data obtained in step 1404. In addition, the state determination program 112 acquires an image representing an ideal state of the corresponding work target from the normal model 106.

In step 1406, the state determination program 112 further compares the work target image in the working space data with the ideal work target image, and calculates the difference between the two. At this time, the work target image in the working space data and the image registered as the normal object model do not need to be images obtained by photographing the object from the same position and direction. The state determination program 112 may obtain the difference between the two after aligning the position so that the work target image in the work space data and the ideal work target image are best matched.

As a method for aligning the positions of two images, the state determination program 112 is a two-dimensional tracking technology (Hirohiko Sagawa, Yudai Urano, Tsuneya Kurihara, “Development of a work support system using simple AR based on two-dimensional tracking,” Human Interface Symposium. 2013 paper collection, 2013) or the like may be used.

After step 1406, the state determination program 112 executes step 612.

According to the second embodiment, even when a two-dimensional image is used, the work support system can determine the state of the work target at an appropriate timing and effectively switch the work procedure information presented to the worker. It becomes possible.

Embodiment 3 of the present invention will be described with reference to FIG.

In Example 1, when it is determined that the operator's hand does not overlap the work target, the process proceeds to a process of calculating the difference between the work target data and the normal object model in the working space data. The work support system according to the third embodiment further executes a process of determining whether a pre-registered worker's hand action has been performed.

Furthermore, the work support system according to the third embodiment is configured so that the work target data in the working space data is obtained when the operator's hand does not exist in the work target area after the operation of the predetermined hand is performed. And a process for calculating a difference between the normal object model and the normal object model.

The configuration of the work support system of the third embodiment is the same as that shown in FIG. However, the work procedure 105 of the third embodiment holds information regarding hand movements for determining hand movements.

手 の Hand movements vary depending on the work procedure. For this reason, the administrator or the like may store information related to hand movements in association with information on each work procedure in the format of the work procedure 105 shown in FIG. In addition, the work support system according to the third embodiment may include a storage area that holds the name of the work procedure and the information related to the hand movement.

Further, the information related to the hand motion may be, for example, time series data of a rotation matrix representing a position coordinate representing the hand trajectory and a direction. Further, the information related to the hand motion may be any information as long as it is information representing a change in the hand trajectory and posture, such as a method of representing the symbol as a moving direction.

FIG. 18 is a flowchart illustrating processing by the work support system according to the third embodiment.

In the processing shown in FIG. 18, Steps 601 to 608, Step 609, and Steps 611 to 616 are the same as those in the first embodiment. On the other hand, steps 1701 and 1702 are different from the processing shown in FIG. The process of step 1701 is executed by the relationship determination program 111 of the third embodiment.

After step 608, the relationship determination program 111 obtains the hand position and posture from the hand area detected in step 605. Then, the relationship determination program 111 associates the time-series data based on the hand position and posture obtained in step 608 executed multiple times in the past up to the time when step 608 starts with the work procedure 105 being executed. It is determined whether or not the predetermined operation has been performed by comparing the registered information regarding the operation of the predetermined hand (1701).

In step 605, since the area of the worker's hand is specified from the working space data, the relationship determination program 111 regards, for example, the position of the center of gravity of the area of the hand as the position of the hand. Alternatively, the relationship determination program 111 may use a method in which the position of the hand region detected from the working space data and the position of the detailed hand model are matched to determine the palm center position as the hand position.

Further, in step 1701, the relationship determination program 111 can obtain a hand posture by obtaining a rectangular parallelepiped that best circumscribes the region of the hand of the worker and calculating a rotation matrix representing the direction. In addition, the relationship determination program 111 may use a method of calculating a rotation matrix representing the direction of the palm by matching the position of the hand region detected from the working space data with the detailed hand model.

Further, the relationship determination program 111 compares well-known time-series data comparison methods for comparing the time-series data of hand positions and postures up to execution of step 1701 with predetermined hand movements, for example, , Dynamic programming (DP matching) (Uchida, “DP Matching Overview: Basics and Various Extensions”, IEICE Technical Report, PRMU 2006-166, pp. 31-36, 2006), etc. may be used.

Thereby, the relationship determination program 111 calculates an index representing the difference and similarity between the two from the result of comparing the two. Then, the relationship determination program 111 may determine whether a predetermined operation has been performed by comparing a predetermined threshold value with the calculated index.

For example, as a result of comparison, when an index indicating that the difference is larger as the value is larger is calculated, the relationship determination program 111 determines that a predetermined operation has been performed when the calculated index is smaller than a predetermined threshold. To do.

Also, as a result of comparison, when a similarity indicating a larger value as the difference is smaller is calculated, the relationship determination program 111 determines that a predetermined operation has been performed when the calculated similarity is greater than a threshold value. When the relationship determination program 111 determines that a predetermined operation has been performed, the relationship determination program 111 stores the determined time in the storage device 107 or the like. Thereafter, the relationship determination program 111 executes Step 609.

After step 609, the relationship determination program 111 determines whether or not the hand overlaps the work target after the operator's hand has performed a predetermined motion based on the determination results in step 1701 and step 609 ( 1702). Specifically, the relationship determination program 111 performs a predetermined operation before starting the step 1702 and if the operator's hand does not exist in the work target area after the predetermined operation is performed. If it is determined, it is determined that the work target and the hand do not overlap. Then, the relationship determination program 111 causes the state determination program 112 to execute step 611.

Alternatively, when it is determined that a predetermined operation has been performed at a time that has been determined to go back a predetermined time from the time when it is determined that the worker's hand does not exist in the work target area, the relationship determination program 111 It may be determined that the work target and the hand do not overlap, and the state determination program 112 may execute step 611.

The third embodiment described above can also be applied to the second embodiment in which an image acquired from a camera is used as working space data.

In Step 1701 described above, it was determined whether the predetermined hand movement was performed based only on the comparison result between the time series data including the position and posture of the worker's hand and the predetermined hand movement. However, the relationship determination program 111 may further determine that the predetermined hand movement has been performed only when the hand movement is performed within a predetermined position range from the work target area.

In step 1701, the relationship determination program 111 may extract the hand direction and the direction of hand movement. If there is a work target area in the direction of the extracted hand movement and the hand movement is close to the predetermined movement, the relationship determination program 111 determines that the predetermined hand movement has been performed. good.

Thereby, the relationship determination program 111 can determine that the timing at which the specific movement indicating the end of the work procedure is performed is an appropriate timing for determining the work state. For example, the relationship determination program 111 can proceed to step 611 for determining the state of the work target at the timing when the “pointing” operation is performed on the work target.

In step 1701, the relationship determination program 111 extracts the direction of hand movement regardless of whether it is determined that a predetermined hand motion has been performed, and the work target in the extracted direction is being executed. It may be determined whether the work target is related to the work procedure.

When it is detected that the hand is moving toward an area that is not a work target related to the work procedure being performed, for example, a position where a different work target is detected, the relationship determination program 111 displays the information. The program 108 and the output device 103 may be used to present a warning to the worker indicating that work is being performed on a different work target. Thus, the information presentation program 108 can cause the worker to review the work procedure and appropriately navigate the work.

Thereby, the relationship determination program 111 can prompt the worker to confirm the content of the work again, and can appropriately navigate the worker.

In step 1702, when a predetermined period of time has elapsed when the operator's hand does not perform the predetermined operation, the relationship determination program 111 uses the information presentation program 108 and the output device 103 to delay the operation, for example. The operator may be presented with a warning or alert to indicate that he is doing.

According to the third embodiment described above, in order to execute the process of determining whether the state of the work target is appropriate based on whether the worker has performed a predetermined operation and whether the hand has been released from the work target, The work support system can determine the end of the work procedure at an appropriate timing after the work is appropriately executed. Thus, the work support system according to the third embodiment can effectively navigate the worker at an appropriate timing.

Embodiment 4 of the present invention will be described with reference to FIGS.

The work support system of Example 1 shifted to a process of calculating the difference between the work target data in the working space data and the normal object model based on the relationship between the worker's hand and the work target.

The work support system according to the fourth embodiment assumes a case where the worker performs a work while holding a tool or the like, determines the positional relationship between the work target area and the worker's hand, and further determines the worker's hand. Does not overlap the work target area, it is determined whether a tool or the like exists between the operator's hand and the work target area.

Then, when it is determined that a tool or the like exists, the work support system according to the fourth embodiment determines whether the area of the tool or the like overlaps the area to be worked. Furthermore, the work support system according to the fourth embodiment executes a process of calculating the difference between the work target data in the work space data and the normal object model when the hand and the tool do not overlap the work target area. Then, it is determined whether the work procedure has been completed normally.

The configuration of the work support system of the fourth embodiment is the same as the configuration of the work support system of FIG. However, the process of the work support system according to the fourth embodiment is partially different from the process of the work support system of FIG.

FIG. 19 is a flowchart illustrating processing by the work support system according to the fourth embodiment.

In the processing of FIG. 19, steps 601 to 609 and steps 611 to 616 are the same as those in the first embodiment. On the other hand, steps 1801 and 1802 are different processes from those shown in FIG. The process of step 1801 is executed by the relationship determination program 111 of the third embodiment.

After step 609, the relationship determination program 111 detects an object such as a tool, and determines whether the object such as a tool is covered by the work target (1801). The relationship determination program 111 may execute Step 1801 particularly when it is determined in Step 609 that the operator's hand does not overlap the work target area. As a result, when the worker's hand is away from the predetermined position range including the work target, but the worker is working on the work target using an object such as a tool, the relationship determination program 111 prevents erroneous determination that one work procedure has been completed in step 1802.

FIG. 20 is an explanatory diagram showing the working space data when the worker of the fourth embodiment is working while holding a tool or the like.

In FIG. 20, an area 1901 is point cloud data corresponding to an object such as a tool. The area 1001 is the work target area obtained in step 607, similarly to the area 1001 shown in FIGS. 11 and 13. The area 1201 is the area of the operator's hand obtained in step 609, as in the area 1201 shown in FIG.

The area 1901 is the work target area calculated in step 607, and the area 1202 is the area of the worker's hand calculated in step 609.

In step 1801, the relationship determination program 111 extracts point cloud data in the space between the work target area 1001 and the worker hand area 1202 from the working space data. When the extracted point cloud data is in contact with the work target area and the worker's hand area, the relationship determination program 111 determines that the tool or the like held by the worker exists in the work target area. be able to.

Also, the relationship determination program 111 may hold in advance a three-dimensional model such as a tool to be used in the same manner as the hand detection method described in the first embodiment. The relationship determination program 111 extracts point cloud data between the work target area and the worker's hand area from the working space data, and the extracted point cloud data and a three-dimensional model such as a tool. The positions may be aligned so that and match best.

The relationship determination program 111 may determine that the operator is holding the tool or the like when the difference between the three-dimensional model of the tool and the extracted point cloud data is smaller than a predetermined threshold.

In this case, the relationship determination program 111 may specify a region such as a tool in the working space data based on the result of alignment between the three-dimensional model and the extracted point cloud data, as in the case of hand detection. . Then, the relationship determination program 111 may further determine whether the tool or the like overlaps the work target area based on the overlap between the specified tool area and the work target area. The work procedure 105 may hold a three-dimensional model such as a tool to be used in association with each work procedure.

After step 1801, the relationship determination program 111 determines whether both the operator's hand and an object such as a tool overlap the work target area based on the determinations in steps 609 and 1801 (1802). If both the operator's hand and the object such as the tool are not included in the work target area (does not overlap), the state determination program 112 starts step 611, and the operator's hand and the object such as the tool are working. If it is included (overlapping) in the target area, the input program 113 starts step 604.

Example 4 can also be applied to Example 2 in which an image acquired from a camera is used as working space data.

According to the fourth embodiment described above, even when the worker is holding a tool or the like and performing the work, the process for determining the state of the work target can be performed at an appropriate timing, so this work can also be performed for a more general work. Examples are applicable.

In addition, this invention is not limited to the above-mentioned Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described.

Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

Also, each of the above-described configurations, functions, processing units, processing procedures, etc. may be realized in hardware by designing a part or all of them, for example, with an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as a program, a table, or a file that realizes each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

Also, the control lines or information lines indicate what is considered necessary for the explanation, and not all control lines or information lines on the product are necessarily shown. In practice, almost all the components are connected to each other.

Claims (15)

A work support system,
A camera that outputs an image of an object that is a target of work and the periphery of the object;
A storage unit storing a first model indicating a three-dimensional model of the object;
From the image, a detection unit for detecting the hand of the user who performs the work and the object;
A relationship determination unit that determines from the image whether the hand and the object overlap;
When the hand and the object do not overlap, the first information that is information of the object is extracted from the image, the second information that is information of the object is extracted from the first model, and the first A state determination unit that determines a working state based on a result of comparing one information with the second information;
An output unit for outputting information according to the work state. The work support system according to claim 1,
The first model shows an ideal shape of the object after the work,
The camera outputs the image showing a three-dimensional model of the object and an object existing around the object;
The detection unit detects a second model, which is a three-dimensional model of the object indicated by the image, and a three-dimensional model of the hand from the image,
The relationship determination unit determines whether the three-dimensional model of the hand and the second model overlap,
When the three-dimensional model of the hand and the second model of the object do not overlap, the state determination unit extracts information on the second model of the object as the first information, and further, the first model The work support system characterized by extracting the 2nd information which is the information of the target object from. The work support system according to claim 1,
The camera outputs an image obtained by photographing a space including a user's field of view as seen from the viewpoint of the user who performs the work as an image obtained by photographing the object and the periphery of the object. . The work support system according to claim 1,
The camera outputs a plurality of the images,
The relationship determination unit includes a second position within a predetermined position range in which the hand includes the object after the hand exists in the predetermined position range including the object for a first predetermined time or more. A work support system that determines that the hand and the object do not overlap when it is detected from the plurality of images that it does not exist for a predetermined time or more. The work support system according to claim 1,
The camera outputs a plurality of the images,
The detection unit detects both hands of the user from the plurality of images,
The relationship determination unit is configured to determine whether the user's hand is in a predetermined position range including the target object after both hands of the user exist in the predetermined position range including the target object for a first predetermined time or longer. When it is detected from the plurality of images that the hand does not exist for a second predetermined time, it is determined that the hand and the object do not overlap. The work support system according to claim 1,
The camera outputs a plurality of the images,
The relationship determination unit
Acquiring the hand movement from the plurality of images;
After the hand performs a predetermined action in a predetermined position range including the object, the hand does not exist in the predetermined position range including the object for a predetermined time or more. A work support system that determines that the hand and the object do not overlap when detected from an image. The work support system according to claim 1,
The camera outputs a plurality of the images,
The relationship determination unit
Acquiring the hand movement from the plurality of images;
After the hand performs a predetermined motion in the direction in which the object is present, the plurality of images indicate that the hand does not exist within a predetermined position range including the object for a predetermined time or longer. A work support system that, when detected, determines that the hand and the object do not overlap. The work support system according to claim 1,
The relationship determination unit
It is determined using the image whether there is an object that is neither the object nor the hand between the object and the hand,
When the object is present between the object and the hand, it is determined using the image whether the object is included in a predetermined position range including the object,
If the hand does not exist within a predetermined position range including the object and the object does not exist within a predetermined position range including the object, the hand and the object do not overlap. A work support system characterized by determining that The work support system according to claim 1,
The state determination unit
By comparing the first information and the second information, a difference between the object detected from the image and the first model is obtained,
A work support system that determines, based on the difference, whether the work is properly completed as the work state. The work support system according to claim 9,
The storage unit stores information on a next object that is an object of a work performed next to the work,
When the state determination unit determines that the work is properly completed, the output unit determines that the work is properly completed and information on the next object as information according to the work state. Output,
When the state determination unit determines that the work is not properly finished, the output unit outputs that the work is not properly finished as information according to the work state. Work support system. The work support system according to claim 9,
The camera outputs a plurality of the images at a predetermined time,
The state determination unit obtains a plurality of differences between the object detected from the plurality of images and the first model;
When the state determination unit determines that the work is not properly completed at the predetermined time based on the plurality of differences, the output unit outputs a warning as information according to the work state. A work support system characterized by The work support system according to claim 1,
The storage unit holds information about other objects that are objects of work different from the work,
The relationship determination unit determines whether the hand is outside a predetermined position range including the object and the hand exists in a predetermined position range including the other object,
When the hand is outside a predetermined position range including the target object and the hand exists in a predetermined position range including the other target object, the output unit outputs information indicating a warning. A work support system characterized by that. The work support system according to claim 1,
The camera outputs a plurality of the images,
The relationship determination unit
The direction in which the hand moves is acquired from the plurality of images,
Determining whether the direction in which the hand moves is a direction toward a predetermined position range including the object;
When the direction in which the hand moves is not a direction toward a predetermined position range including the object, the output unit outputs information indicating a warning. A work support method by a work support system,
The work support system includes:
A processor and memory;
A camera that outputs an image obtained by photographing an object that is a target of work and the periphery of the object;
The method
A storage procedure in which the processor stores a first model indicating a three-dimensional model of the object;
A detection procedure in which the processor detects, from the image, a user's hand performing the work and the object;
A relationship determination procedure for the processor to determine whether the hand and the object overlap;
When the hand does not overlap the object, the processor extracts first information that is information on the object from the image, and extracts second information that is information on the object from the first model. And a state determination procedure for determining a work state based on a result of comparing the first information and the second information;
An output procedure in which the processor outputs information according to the work state. The work support method according to claim 14,
The first model shows an ideal shape of the object after the work,
The method includes a step in which the camera outputs the image showing a three-dimensional model of the object and an object existing around the object;
The detection procedure includes a procedure in which the processor detects a second model, which is a three-dimensional model of the object indicated by the image, and a three-dimensional model of the hand from the image.
The relationship determination procedure includes a procedure in which the processor determines whether the three-dimensional model of the hand and the second model overlap each other,
In the state determination procedure, when the three-dimensional model of the hand and the second model of the object do not overlap, the processor extracts the information of the second model of the object as the first information, and A work support method comprising a procedure of extracting second information which is information of the object from the first model.

Images