WO2024201737A1 - Robot control device, robot control method, and program - Google Patents

Abstract

The present invention provides a robot control device comprising: an image acquisition unit that acquires data which is generated by a vision sensor connected to a movable part of a robot and relates to an image containing a detection object on a workpiece moving in a transport direction; a storage unit that stores a first correspondence between a first position of the movable part in the optical axis direction of the vision sensor that intersects the transport direction and a first detection object target position at which the detection object should be located in the image when the movable part is in the first position and a second correspondence between a second position of the movable part in the optical axis direction and a second detection object target position at which the detection object should be located in the image when the movable part is in the second position; and a target position setting unit that sets a target position at which the detection object should be located in the image on the basis of the first correspondence and the second correspondence and the position of the movable part in the optical axis direction.

Description

Robot control device, robot control method and program

This disclosure relates to a robot control device, a robot control method, and a program.

　Conventionally, there is known a production line equipped with a robot and a transport device that transports workpieces. In this production line, the robot follows the workpiece being transported in the transport direction by the transport device, and then starts working on the workpiece.

When a single visual sensor (e.g., a two-dimensional camera) that is connected to the moving part of the robot (e.g., a hand, end effector, etc.) and moves integrally with the moving part of the robot is used to make the robot follow the workpiece being transported in the transport direction, when the moving part of the robot moves in the optical axis direction of the visual sensor, the appearance of the detection target on the workpiece being transported in the transport direction, which is included in the image generated by the visual sensor, changes. If the appearance of the detection target included in the image changes, there is a risk that the moving part of the robot cannot be made to properly follow the workpiece being transported in the transport direction with conventional visual feedback that uses a single visual sensor.

Consequently, a fixed two-dimensional camera that is not connected to the moving part of the robot and does not move integrally with the moving part of the robot is used, and the transport of the workpiece by the transport device is stopped when the detection target is positioned on the optical axis of the two-dimensional camera (see, for example, JP 2012-161850 A). Therefore, with this technology, the appearance of the detection target contained in the image captured by the two-dimensional camera does not change, and the moving part of the robot can be made to follow the workpiece. On the other hand, with this technology, it is necessary to use a fixed two-dimensional camera, and the detection target needs to be on the optical axis of the two-dimensional camera.

JP 2012-161850 A

Even if the appearance of the detection target on the workpiece moving in the transport direction, which is included in the image generated by the visual sensor, changes as the visual sensor, which is connected to the moving part of the robot and moves integrally with the moving part of the robot, moves in the direction of the optical axis of the visual sensor, it is desirable to be able to appropriately control the position of the moving part of the robot to follow the workpiece moving in the transport direction based on the image generated by the visual sensor.

One aspect of the present disclosure is a robot control device that includes an image acquisition unit that acquires image data including a detection target on a workpiece moving in a conveying direction, generated by a visual sensor provided on a movable part of a robot; a storage unit that stores at least a first correspondence relationship that is a correspondence relationship between a first position of the movable part in an optical axis direction of the visual sensor, which is a direction intersecting the conveying direction, and a first detection target position that is a target position on the image where the detection target should be located when the movable part is located at the first position; and a second correspondence relationship that is a correspondence relationship between a second position of the movable part in the optical axis direction and a second detection target position that is a target position on the image where the detection target should be located when the movable part is located at the second position; and a target position setting unit that sets the target position on the image where the detection target should be located based on the first correspondence relationship, the second correspondence relationship, and the position of the movable part in the optical axis direction.

1 is a diagram illustrating an example of a robot system 100 to which a robot control device 2 of a first embodiment is applied. FIG. 2 is a diagram illustrating an example of functional blocks of the robot system 100 shown in FIG. 1 . FIG. 2 is a diagram showing a robot 1 and the like included in the robot system 100 shown in FIG. 1 as viewed from the top of FIG. 13 is a diagram for explaining an example of robot teaching executed at a movement start position X1 in the X direction of a movable part 11B (hand) of the robot 1. FIG. 5 is a diagram showing an example of an image IM captured by a visual sensor 14 (two-dimensional camera) in the robot teaching shown in FIG. 4. 13 is a diagram for explaining an example of robot teaching executed at a movement end position X2 in the X direction of a movable part 11B (hand) of the robot 1. FIG. 7 is a diagram showing an example of an image IM captured by a visual sensor 14 (two-dimensional camera) in the robot teaching shown in FIG. 6. 13 is a diagram for explaining an example of control (visual feedback) of the position of the movable part 11B (hand) of the robot 1 executed by the robot control unit 232. FIG. 9 is a diagram showing an example of the target position (Hz(X), Vt(X)) of the detection target DT on the image IM that is set by the target position setting unit 231B when the visual feedback shown in FIG. 8 is executed. FIG. 4 is a flowchart illustrating an example of a process executed by the robot control device 2 of the first embodiment. FIG. 13 is a diagram illustrating an example of a robot control device 2 according to a second embodiment.

Below, embodiments of the robot control device, robot control method, and program disclosed herein will be described with reference to the attached drawings.

First Embodiment
Fig. 1 is a diagram showing an example of a robot system 100 to which a robot control device 2 according to a first embodiment is applied. Fig. 2 is a diagram showing an example of functional blocks of the robot system 100 shown in Fig. 1. Fig. 3 is a diagram showing a robot 1 and the like included in the robot system 100 shown in Fig. 1 as viewed from the top of Fig. 1.
In the example shown in FIGS. 1 to 3, a robot system 100 includes a robot 1 and a robot control device 2 .
In the example shown in FIGS. 1 to 3, the robot system 100 does not include a teaching pendant, but in other examples, the robot system 100 may include a teaching pendant.

1 to 3, the robot 1 is a six-axis vertical articulated robot. The robot 1 includes a plurality of links 11, a plurality of joints 12, a plurality of actuators 13, a visual sensor 14, and a connecting member 15.
The multiple link parts 11 include a base part 11A, multiple movable parts 11C, and a movable part 11B. The multiple movable parts 11C include a part that functions as the torso of the robot 1 and a part that functions as the lower arm of the robot 1. The multiple movable parts 11C also include a part that functions as the upper arm of the robot 1 (more specifically, a part on the lower arm side of the upper arm of the robot 1 and a part on the wrist side of the upper arm of the robot 1). The multiple movable parts 11C also include a part that functions as the wrist of the robot 1 (more specifically, a part on the upper arm side of the wrist of the robot 1). The movable part 11B includes a part (hand) on the tip side of the wrist of the robot 1.

Each of the multiple joint portions 12 functions as a rotational joint.
The joints 12 include a joint 12 that connects the base 11A and the movable part 11C that functions as the torso of the robot 1. The joints 12 also include a joint 12 that connects the movable part 11C that functions as the torso of the robot 1 and the movable part 11C that functions as the lower arm of the robot 1. The joints 12 also include a joint 12 that connects the movable part 11C that functions as the lower arm of the robot 1 and the movable part 11C that functions as a part of the lower arm side of the upper arm of the robot 1. The joints 12 also include a joint 12 that connects the movable part 11C that functions as a part of the lower arm side of the upper arm of the robot 1 and the movable part 11C that functions as a part of the wrist side of the upper arm of the robot 1. The joints 12 also include a joint 12 that connects the movable part 11C that functions as a part of the wrist side of the upper arm of the robot 1 and the movable part 11C that functions as a part of the upper arm side of the wrist of the robot 1. The multiple joint parts 12 also include a joint part 12 that connects a movable part 11C that functions as the upper arm side part of the wrist of the robot 1 and a movable part 11B (hand) that functions as the tip side part of the wrist of the robot 1.
In another example, the joint portion 12 may function as a linear joint.

In the example shown in FIG. 1 to FIG. 3, the actuators 13 include an actuator 13 that rotates (turns) the movable part 11C that functions as the body of the robot 1 with respect to the base part 11A. The actuators 13 also include an actuator 13 that rotates the movable part 11C that functions as the lower arm of the robot 1 with respect to the movable part 11C that functions as the body of the robot 1. The actuators 13 also include an actuator 13 that rotates the movable part 11C that functions as the lower arm side part of the upper arm of the robot 1 with respect to the movable part 11C that functions as the lower arm side part of the upper arm of the robot 1. The actuators 13 also include an actuator 13 that rotates the movable part 11C that functions as the wrist side part of the upper arm of the robot 1 with respect to the movable part 11C that functions as the lower arm side part of the upper arm of the robot 1 (i.e., twists the upper arm of the robot 1). The actuators 13 also include an actuator 13 that rotates the movable part 11C that functions as the upper arm side part of the wrist of the robot 1 with respect to the movable part 11C that functions as the wrist side part of the upper arm of the robot 1. The multiple actuators 13 also include an actuator 13 that rotates movable part 11B (hand) that functions as the tip side part of the wrist of robot 1 relative to movable part 11C that functions as the upper arm side part of the wrist of robot 1 (i.e., twists the wrist of robot 1).
Each of the multiple actuators 13 is composed of a servo motor. The servo motor rotates a rotation shaft of the servo motor based on a control signal from the robot control device 2. Each of the multiple actuators 13 (servo motor) has an actuator sensor 13A. The actuator sensor 13A is composed of an encoder that detects the position and speed of the rotation shaft of the servo motor. The actuator sensor 13A (encoder) outputs actuator sensor data indicating the position and speed of the rotation shaft of the servo motor to the robot control device 2.
In other examples, an actuator other than a servo motor may be used as the actuator 13, and an actuator sensor other than an encoder may be used as the actuator sensor 13A.

1 to 3, the visual sensor 14 is a two-dimensional camera. The two-dimensional camera captures an image IM (see FIG. 5, etc.) including a detection target DT on a workpiece W1 being transported in a transport direction (negative side of the Y direction, the front side in FIG. 1, the right side in FIG. 3) by a conveyor C. In other words, the visual sensor 14 generates data of the image IM including the detection target DT on the workpiece W1 moving in the transport direction and transmits it to the robot control device 2.
The visual sensor 14 is connected to the movable part 11B (hand) that functions as the tip side part of the wrist of the robot 1 via a connection member 15. In other words, when the movable part 11B (hand) of the robot 1 is moved by the actuator 13, the visual sensor 14 moves integrally with the movable part 11B (hand). In other words, the visual sensor 14 is provided on the movable part 11B (hand) of the robot 1.

3, the optical axis direction of the visual sensor 14 is set to the X direction. That is, the optical axis direction of the visual sensor 14 is a direction intersecting the conveying direction (Y direction) of the workpiece W1, for example, a direction approximately perpendicular to the conveying direction (Y direction) of the workpiece W1. The optical axis of the visual sensor 14 and the conveyor C conveying the workpiece W1 may cross each other, for example, at a three-dimensional intersection (that is, the optical axis of the visual sensor 14 and the conveyor C conveying the workpiece W1 may have a relationship of, for example, a "twist position"). Note that the optical axis direction of the visual sensor 14 and the conveying direction of the workpiece W1 are not limited to being approximately perpendicular, and may include a slight angular deviation.
1 to 3, the visual sensor 14 is connected to the movable part 11B (hand) that functions as a part on the tip side of the wrist of the robot 1 via the connection member 15. Meanwhile, in another example, the visual sensor 14 may be connected to a movable part 11C other than the movable part 11B (hand) of the robot 1 via the connection member 15. For example, in an example in which the actuator 13 that rotates the movable part 11B (hand) of the robot 1 cannot be operated with respect to the movable part 11C that functions as a part on the upper arm side of the wrist of the robot 1, the visual sensor 14 may be connected to the movable part 11C that functions as a part on the upper arm side of the wrist of the robot 1 via the connection member 15.

In the example shown in FIGS. 1 to 3, the robot control device 2 is configured by a computer having a communication unit 21, a storage unit 22, and a processing unit .
The communication unit 21 is a communication interface and has an interface circuit for connecting the robot control device 2 to the actuator 13, visual sensor 14, etc. of the robot 1 via, for example, a signal line. The storage unit 22 is a memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory), or a storage such as a HDD (Hard Disk Drive) or an SDD (Solid State Drive). The storage unit 22 stores programs and various data used in the processing executed by the processing unit 23. The processing unit 23 is a processor and has a function as an image processing unit 231 and a function as a robot control unit 232.
The image processing unit 231 processes the image IM captured by the visual sensor 14 (two-dimensional camera), etc. The robot control unit 232 controls the robot 1 (visual feedback) based on the image IM captured by the visual sensor 14 (two-dimensional camera).

The image processing unit 231 has a function as an image acquisition unit 231A and a function as a target position setting unit 231B.
The image acquisition unit 231A acquires data of an image IM (see FIG. 5, etc.) including a detection target DT on a workpiece W1 moving in a conveying direction (right side in FIG. 3) generated by the visual sensor 14. That is, the image acquisition unit 231A acquires data of the image IM captured by the visual sensor 14 (two-dimensional camera).
The target position setting unit 231B sets a target position (Hz(X), Vt(X)) (see FIG. 9) of the detection target DT on the image IM captured by the visual sensor 14 (two-dimensional camera), which is used in the visual feedback executed by the robot control unit 232. In detail, the target position setting unit 231B sets the target position (Hz(X), Vt(X)) of the detection target DT on the image IM when the position (actual coordinate value) in the X direction of the movable part 11B (hand) of the robot 1 calculated by the position calculation unit 232A described later is X.

The robot control unit 232 has a function as a position calculation unit 232A and a function as a control unit 232B.
The position calculation unit 232A calculates the position (more specifically, the position in the X-direction, Y-direction, and Z-direction (X, Y, Z)) of the movable part 11B (hand) of the robot 1 connected to the visual sensor 14 via the connecting member 15, based on the position of the rotation axis of each of the multiple actuators 13 (servo motors) detected by each of the multiple actuator sensors 13A (encoders).
The control unit 232B executes control (visual feedback) of the position (X, Y, Z) of the movable part 11B (hand) of the robot 1 connected to the visual sensor 14 via the connecting member 15, based on the target position (Hz(X), Vt(X)) of the detection target DT on the image IM set by the target position setting unit 231B.

In the example shown in Figures 1 to 3, the control unit 232B executes control to move the movable part 11B (hand (more specifically, the hand holding the workpiece W2)) of the robot 1 closer to the workpiece W1 on the conveyor C in the X direction in visual feedback of the position (X, Y, Z) of the movable part 11B (hand) of the robot 1. Furthermore, the control unit 232B executes control to change the position of the movable part 11B (hand) of the robot 1 in the Y direction to follow the workpiece W1 transported by the conveyor C to the negative side in the Y direction (the right side in Figure 3). On the other hand, the control unit 232B does not execute control to change the position of the movable part 11B (hand) of the robot 1 in the Z direction.
In another example, the control unit 232B may execute control to change the position in the Z direction of the movable part 11B (hand) of the robot 1 in visual feedback of the position (X, Y, Z) of the movable part 11B (hand) of the robot 1. In detail, in this example, the control unit 232B executes control to bring the movable part 11B (hand) of the robot 1 closer to the workpiece W1 on the conveyor C in the X direction, and executes control to change the position in the Y direction of the movable part 11B (hand) of the robot 1 to follow the workpiece W1 transported by the conveyor C to the negative side in the Y direction (the right side in FIG. 3 ).

In the example shown in Figs. 1 to 3, before the control unit 232B controls the position (X, Y, Z) of the movable part 11B (hand) of the robot 1, for example, an operator performs robot teaching to register the position and posture of the robot 1. In detail, for example, an operator performs robot teaching to register the position and posture of the robot 1 at the movement start position X1 and movement end position X2 in the X direction of the movable part 11B (hand) of the robot 1.

Figure 4 is a diagram for explaining an example of robot teaching that is performed at the movement start position X1 in the X direction of the movable part 11B (hand) of the robot 1. Figure 5 is a diagram showing an example of an image IM captured by the visual sensor 14 (two-dimensional camera) during the robot teaching shown in Figure 4 (i.e., when the X direction position of the movable part 11B (hand) of the robot 1 is the movement start position X1).

4 and 5, for example, when performing robot teaching at the X-direction movement start position X1 of the movable part 11B (hand) of the robot 1, the operator sets the X-direction position (actual coordinate value) of the movable part 11B (hand) of the robot 1 to X1. Furthermore, for example, the operator appropriately sets the Y-direction position (actual coordinate value) and the Z-direction position (actual coordinate value) of the movable part 11B (hand) of the robot 1 so that the detection target DT on the workpiece W1 is included in the image IM captured by the visual sensor 14 (two-dimensional camera) as shown in FIG. 5. The memory unit 22 stores information indicating the positions of the rotation axes of each of the multiple actuators 13 (servo motors) detected by the actuator sensor 13A when, for example, the operator sets the X-direction position (movement start position X1), Y-direction position, and Z-direction position of the movable part 11B (hand) of the robot 1.

Next, the visual sensor 14 captures an image IM including the detection target DT on the workpiece W1 shown in Fig. 5, the image acquisition unit 231A acquires data of the image IM, and the storage unit 22 stores the data of the image IM. The target position setting unit 231B calculates the position (coordinates) (Hz1, Vt1) of the detection target DT on the image IM shown in Fig. 5. The target position setting unit 231B further sets the position (Hz1, Vt1) as the target position of the detection target DT on the image IM captured by the visual sensor 14 when the robot control unit 232 executes visual feedback (more specifically, when the movable part 11B (hand) of the robot 1 starts moving in the X direction when the visual feedback is executed).
Next, for example, an operator sets the X-direction position (actual coordinate value) of the movable part 11B (hand) of the robot 1 to the movement end position X2, and performs robot teaching at the X-direction movement end position X2 of the movable part 11B (hand) of the robot 1.

Fig. 6 is a diagram for explaining an example of robot teaching executed at the movement end position X2 in the X direction of the movable part 11B (hand) of the robot 1. Fig. 7 is a diagram showing an example of an image IM captured by the visual sensor 14 (two-dimensional camera) in the robot teaching shown in Fig. 6 (i.e., when the position of the movable part 11B (hand) of the robot 1 in the X direction is the movement end position X2).
6 and 7, for example, an operator changes the X-direction position (actual coordinate value) of the movable part 11B (hand) of the robot 1 from X1 to X2 without changing the Y-direction position (actual coordinate value) and Z-direction position (actual coordinate value) of the movable part 11B (hand) of the robot 1 from the state shown in Fig. 4, and executes robot teaching at the X-direction movement end position X2 of the movable part 11B (hand) of the robot 1. The storage unit 22 stores information indicating the positions of the rotation axes of each of the multiple actuators 13 (servo motors) when, for example, the operator changes the X-direction position of the movable part 11B (hand) of the robot 1 from X1 to X2.

Then, the visual sensor 14 captures an image IM including the detection target DT on the workpiece W1 shown in FIG. 6, the image acquisition unit 231A acquires the data of the image IM, and the memory unit 22 stores the data of the image IM. The target position setting unit 231B calculates the position (coordinates) (Hz2, Vt2) of the detection target DT on the image IM shown in FIG. 7. The target position setting unit 231B further sets the position (Hz2, Vt2) as the target position of the detection target DT on the image IM captured by the visual sensor 14 when the robot control unit 232 executes visual feedback (more specifically, when the movement of the movable part 11B (hand) of the robot 1 in the X direction ends when executing visual feedback).

1 to 3, in order to make the robot 1 follow the workpiece W1 being transported in the transport direction (Y direction), one visual sensor 14 (two-dimensional camera) is used, which is connected to the movable part 11B (hand) of the robot 1 via a connection member 15 and moves integrally with the movable part 11B (hand) of the robot 1. Therefore, when the movable part 11B (hand) of the robot 1 moves in the optical axis direction (X direction) of the visual sensor 14 as shown in Figs. 4 and 6, the appearance of the detection target DT on the workpiece W1 being transported in the transport direction (Y direction) included in the image IM generated (captured) by the visual sensor 14 (two-dimensional camera) changes as shown in Figs. 5 and 7.
In detail, the Y-direction position and Z-direction position of the movable part 11B (hand) of the robot 1 in the state shown in Figures 4 and 5 are the same (unchanged) as the Y-direction position and Z-direction position of the movable part 11B (hand) of the robot 1 in the state shown in Figures 6 and 7. Nevertheless, the position (coordinates) (Hz1, Vt1) of the detection target DT on the image IM captured by the visual sensor 14 (two-dimensional camera) in the state shown in Figures 4 and 5 differs from the position (coordinates) (Hz2, Vt2) of the detection target DT on the image IM captured by the visual sensor 14 (two-dimensional camera) in the state shown in Figures 6 and 7.
Therefore, if conventional visual feedback were applied, the horizontal position (coordinate) Hz1 of the detection target DT on the image IM shown in Figure 5 is different from the horizontal position (coordinate) Hz2 of the detection target DT on the image IM shown in Figure 7, so there is a risk that the robot control device 2 will erroneously determine that the position of the workpiece W1 in the transport direction (Y direction) has changed.

In consideration of the above, in the example shown in Figures 1 to 3, the memory unit 22 stores a first correspondence relationship, which is a correspondence relationship between the movement start position X1 in the X direction of the movable part 11B of the robot 1 and the target position (Hz1, Vt1) on the image IM where the detection target DT should be located when the movable part 11B of the robot 1 is located at the movement start position X1 (i.e., in the state shown in Figures 4 and 5). The memory unit 22 also stores a second correspondence relationship, which is a correspondence relationship between the movement end position X2 of the movable part 11B of the robot 1 in the X direction and the target position (Hz2, Vt2) on the image IM where the detection target DT should be located when the movable part 11B of the robot 1 is located at the movement end position X2 (i.e., in the state shown in Figures 6 and 7).

Furthermore, in the example shown in Figures 1 to 3, when the robot control unit 232 executes control (visual feedback) of the position of the movable part 11B (hand) of the robot 1, the target position setting unit 231B sets the target position (Hz(X), Vt(X)) on the image IM where the detection target DT should be located when the X-direction position of the movable part 11B (hand) of the robot 1 is neither the movement start position X1 nor the movement end position X2, based on the first correspondence relationship, the second correspondence relationship, and the X-direction position X of the movable part 11B of the robot 1.

In detail, the target position (Hz(X), Vt(X)) on the image IM where the detection target DT should be located, which is set by the target position setting unit 231B, is represented by a horizontal coordinate value Hz(X) and a vertical coordinate value Vt(X). The horizontal coordinate value Hz(X) changes in proportion to the change in the position of the movable part 11B (hand) of the robot 1 in the X direction. The vertical coordinate value Vt(X) also changes in proportion to the change in the position of the movable part 11B (hand) of the robot 1 in the X direction.

More specifically, the target position setting unit 231B calculates the horizontal target position Hz(X) of the detection target DT on the image IM based on the following formula:
Hz(X)=Hz1+((Hz2-Hz1)/(X2-X1))×(X-X1)
Furthermore, the target position setting unit 231B calculates the target position Vt(X) in the vertical direction of the detection target DT on the image IM based on the following formula.
Vt(X)=Vt1+((Vt2-Vt1)/(X2-X1))×(X-X1))

In the example shown in Figs. 1 to 3, the robot control unit 232 controls the position of the movable part 11B (hand) of the robot 1 (visual feedback) based on the position (Hz, Vt) of the detection target DT on the image IM captured by the visual sensor 14 (two-dimensional camera) and the target position (Hz(X), Vt(X)) on the image IM where the detection target DT should be located, set by the target position setting unit 231B. In detail, the robot control unit 232 controls the position (visual feedback) of the movable part 11B (hand) of the robot 1 so that the position (Hz, Vt) of the detection target DT on the image IM matches the target position (Hz(X), Vt(X)) of the detection target DT on the image IM.

In other words, in the example shown in Figures 1 to 3, the robot control unit 232 controls the position of the movable part 11B (hand) of the robot 1 in the X direction based on information stored in the memory unit 22 indicating the movement start position X1 and the movement end position X2 of the movable part 11B (hand) of the robot 1.

Fig. 8 is a diagram for explaining an example of control (visual feedback) of the position of the movable part 11B (hand) of the robot 1 executed by the robot control unit 232. Fig. 9 is a diagram showing an example of the target position (Hz(X), Vt(X)) of the detection target DT on the image IM set by the target position setting unit 231B when the visual feedback shown in Fig. 8 is executed.
As shown in FIG. 9, the trajectory of the target position (Hz(X), Vt(X)) of the detection target DT on the image IM set by the target position setting unit 231B is a straight line connecting the position (Hz1, Vt1) of the detection target DT on the image IM when the X-direction position of the movable part 11B (hand) of the robot 1 is the movement start position X1, and the position (Hz2, Vt2) of the detection target DT on the image IM when the X-direction position of the movable part 11B (hand) of the robot 1 is the movement end position X2.
8 and 9, the robot control unit 232 executes control to move the movable part 11B (hand) of the robot 1 closer to the workpiece W1 on the conveyor C in the X direction, and also executes control to change the position of the movable part 11B (hand) of the robot 1 in the Y direction to the negative side in the Y direction (the right side in FIG. 8) by following the workpiece W1 transported by the conveyor C to the negative side in the Y direction (the right side in FIG. 8). Specifically, the workpiece W2 (door) grasped by the movable part 11B (hand) of the robot 1 is attached to the workpiece W1 (car body) transported by the conveyor C to the negative side in the Y direction (the right side in FIG. 8).

Specifically, in the example shown in FIGS. 8 and 9, when the movable part 11B (hand) of the robot 1 approaches the workpiece W1 in the X direction, the robot control unit 232 controls the position of the movable part 11B (hand) of the robot 1 only in the transport direction (Y direction) of the workpiece W1 based on the position (Hz, Vt) of the detection target DT on the image IM and the target position (Hz(X), Vt(X)) of the detection target DT on the image IM set by the target position setting unit 231B.
That is, in the example shown in Figures 8 and 9, control of the Z-direction position of the movable part 11B (hand) of the robot 1 is different from control of the Y-direction position of the movable part 11B (visual feedback).

As described above, in the example shown in Figures 4 and 5, robot teaching is performed at position X1 where the movable part 11B (hand) of the robot 1 starts moving in the X direction, and in the example shown in Figures 6 and 7, robot teaching is performed at position X2 where the movable part 11B (hand) of the robot 1 ends moving in the X direction.

In another example, robot teaching may be performed at any two positions between the movement start position X1 and movement end position X2 of the movable part 11B (hand) of the robot 1 in the X direction.
In this example, the storage unit 22 stores a first correspondence relationship between one of two positions (first position) between the movement start position X1 and the movement end position X2 in the X direction of the movable part 11B of the robot 1 and a target position on the image IM where the detection target DT should be located when the movable part 11B of the robot 1 is located at the first position. The storage unit 22 also stores a second correspondence relationship between the other of two positions (second position) between the movement start position X1 and the movement end position X2 in the X direction of the movable part 11B of the robot 1 and a target position on the image IM where the detection target DT should be located when the movable part 11B of the robot 1 is located at the second position.
Furthermore, in this example, when the robot control unit 232 executes control (visual feedback) of the position of the movable part 11B (hand) of the robot 1, the target position setting unit 231B sets a target position (Hz(X), Vt(X)) on the image IM where the detection target DT should be located when the position of the movable part 11B (hand) of the robot 1 in the X direction is neither the first position nor the second position, based on the first correspondence relationship, the second correspondence relationship, and the position X in the X direction of the movable part 11B of the robot 1.
Also, in this example, similar to the examples shown in Figures 1 to 3, the robot control unit 232 controls the position of the movable part 11B (hand) of the robot 1 (visual feedback) based on the position (Hz, Vt) of the detection target DT on the image IM captured by the visual sensor 14 (two-dimensional camera) and the target position (Hz(X), Vt(X)) on the image IM where the detection target DT should be located, which is set by the target position setting unit 231B.

FIG. 10 is a flowchart for explaining an example of processing executed by the robot control device 2 of the first embodiment.
In the example shown in FIG. 10, in step S11, robot teaching is executed to register the position and posture of the robot 1 at a movement start position X1 in the X direction of the movable part 11B (hand) of the robot 1.
In detail, for example, an operator sets the X-direction position (actual coordinate value) of the movable part 11B (hand) of the robot 1 to X1, and also sets the Y-direction position (actual coordinate value) and the Z-direction position (actual coordinate value) of the movable part 11B (hand) of the robot 1 so that the detection target DT on the workpiece W1 is included in the image IM captured by the visual sensor 14 (two-dimensional camera) as shown in FIG. 5.

In step S11A, the storage unit 22 stores information indicating the positions of the rotation axes of the actuators 13 (servo motors) detected by the actuator sensors 13A. That is, the position and orientation of the robot 1 at the movement start position X1 in the X direction of the movable part 11B (hand) of the robot 1 are registered.
In step S11B, the image acquisition unit 231A acquires data of an image IM (see FIG. 5) including the detection target DT on the workpiece W1 captured by the visual sensor 14, and the memory unit 22 stores the data of the image IM.
In step S11C, the target position setting unit 231B calculates the position (coordinates) (Hz1, Vt1) (see FIG. 5) of the detection target DT on the image IM.
In step S11D, the target position setting unit 231B sets the position (Hz1, Vt1) as the target position of the detection target DT on the image IM captured by the visual sensor 14 when the movable part 11B (hand) of the robot 1 starts moving in the X direction when the robot control unit 232 executes visual feedback.
In step S11E, the memory unit 22 stores a first correspondence relationship between the movement start position X1 in the X-direction of the movable part 11B of the robot 1 and the target position (Hz1, Vt1) on the image IM where the detection target DT should be located when the movable part 11B of the robot 1 is located at the movement start position X1 (i.e., in the state shown in Figures 4 and 5).
That is, in step S11, the target position (Hz1, Vt1) of the detection target DT on the image IM when the movable part 11B (hand) of the robot 1 is located at the movement start position X1 is taught.

Next, in step S12, robot teaching is executed to register the position and posture of the robot 1 at the movement end position X2 of the movable part 11B (hand) of the robot 1 in the X direction.
In detail, for example, the position (actual coordinate value) in the X direction of the movable part 11B (hand) of the robot 1 is changed by an operator from X1 to X2.

In step S12A, the storage unit 22 stores information indicating the positions of the rotation axes of the actuators 13 (servo motors) detected by the actuator sensors 13A. That is, the position and posture of the robot 1 at the movement end position X2 in the X direction of the movable part 11B (hand) of the robot 1 are registered.
In step S12B, the image acquisition unit 231A acquires data of an image IM (see FIG. 7) including the detection target DT on the workpiece W1 captured by the visual sensor 14, and the memory unit 22 stores the data of the image IM.
In step S12C, the target position setting unit 231B calculates the position (coordinates) (Hz2, Vt2) (see FIG. 7) of the detection target DT on the image IM.
In step S12D, the target position setting unit 231B sets the position (Hz2, Vt2) as the target position of the detection target DT on the image IM captured by the visual sensor 14 at the end of the X-directional movement of the movable part 11B (hand) of the robot 1 when visual feedback is executed by the robot control unit 232.
In step S12E, the memory unit 22 stores a second correspondence relationship between the end position X2 of the movement of the movable part 11B of the robot 1 in the X-direction and the target position (Hz2, Vt2) on the image IM where the detection target DT should be located when the movable part 11B of the robot 1 is located at the end position X2 (i.e., in the state shown in Figures 6 and 7).
That is, in step S12, the target position (Hz2, Vt2) of the detection target DT on the image IM when the movable part 11B (hand) of the robot 1 is located at the movement end position X2 is taught.

Next, in step S13, the robot control unit 232 executes control (visual feedback) of the position of the movable part 11B (hand) of the robot 1.
In detail, for example, an operator sets the position of the movable part 11B (hand) of the robot 1 in the X direction to a movement start position X1, and the control unit 232B starts control (visual feedback) of the Y direction position of the movable part 11B (hand) of the robot 1. During the control (visual feedback) of the Y direction position of the movable part 11B (hand) of the robot 1, the robot control unit 232 executes control (different from visual feedback) to change the position of the movable part 11B (hand) of the robot 1 in the X direction from the movement start position X1 to the movement end position X2.

In step S13A, the position calculation unit 232A calculates the X-, Y-, and Z-directional positions (X, Y, Z) of the movable part 11B (hand) of the robot 1 connected to the visual sensor 14 via the connecting member 15, based on the positions of the rotation axes of each of the multiple actuators 13 (servo motors) detected by each of the multiple actuator sensors 13A (encoders).
In step S13B, the image acquisition unit 231A acquires data of an image IM including the detection target DT on the workpiece W1 captured by the visual sensor 14.
In step S13C, the target position setting unit 231B sets a target position (Hz(X), Vt(X)) on the image IM where the detection target DT should be located, based on the first correspondence relationship, the second correspondence relationship, and the position X in the X-direction of the movable part 11B of the robot 1.
In detail, the target position setting unit 231B calculates the horizontal target position Hz(X) of the detection target DT on the image IM and the vertical target position Vt(X) of the detection target DT on the image IM based on the above-mentioned formula.
In the example shown in FIG. 10, a process of calculating the target position (Hz(X), Vt(X)) of the detection target DT on the image IM is performed during the execution of visual feedback. Meanwhile, in another example, a process of calculating the target position (Hz(X), Vt(X)) of the detection target DT on the image IM may be performed in pre-processing. In this example, a map is created that shows the correspondence between the position X in the X-direction of the movable part 11B (hand) of the robot 1 and the target position (Hz(X), Vt(X)) of the detection target DT on the image IM. Furthermore, by using the map during the execution of visual feedback, the target position (Hz(X), Vt(X)) of the detection target DT on the image IM that corresponds to the position X in the X-direction of the movable part 11B (hand) of the robot 1 is set.

In the example shown in FIG. 10, in step S13D, the robot control unit 232 controls the Y-direction position (visual feedback) of the movable part 11B (hand) of the robot 1 based on the position (Hz, Vt) of the detection target DT on the image IM indicated by the data acquired in step S13B and the target position (Hz(X), Vt(X)) on the image IM where the detection target DT should be located, which is set in step S13C.
In detail, the robot control unit 232 executes control (visual feedback) of the Y-direction position of the movable part 11B (hand) of the robot 1 so that the position (Hz, Vt) of the detection target DT on the image IM indicated by the data acquired in step S13B coincides with the target position (Hz(X), Vt(X)) on the image IM where the detection target DT should be located, which was set in step S13C. As a result, no matter where the detection target DT is located on the image IM, the movable part 11B (hand) of the robot 1 can be made to follow the workpiece W1 being transported in the transport direction (Y direction) while moving the X-direction position of the movable part 11B (hand) of the robot 1.
That is, in step S13, while the movable part 11B (hand) of the robot 1 is moved in the X direction, control (visual feedback) of the position of the movable part 11B (hand) of the robot 1 in the Y direction is executed. Furthermore, during execution of the visual feedback, the target position (Hz(X), Vt(X)) of the detection target DT on the image IM is changed according to the position of the movable part 11B (hand) of the robot 1 in the X direction.

Second Embodiment
The robot control device 2 of the second embodiment is configured similarly to the robot control device 2 of the above-described first embodiment, except for the points that will be described later.

FIG. 11 is a diagram showing an example of a robot control device 2 according to the second embodiment.
In the example shown in FIG. 11, the robot control device 2 includes an image processing section 2-1 and a robot control section 2-2.
The image processing section 2-1 is configured by a computer having a communication section 21-1, a storage section 22-1, and a processing section 23-1.
The communication unit 21-1 is a communication interface and has an interface circuit for connecting the image processing unit 2-1 to the visual sensor 14 of the robot 1, the robot control unit 2-2, etc., via, for example, a signal line. The storage unit 22-1 is a memory such as a ROM or RAM, or a storage such as a HDD or SDD. The storage unit 22-1 stores programs and various data used in the processing executed by the processing unit 23-1. The processing unit 23-1 is a processor and has a function as an image acquisition unit 231A-1 and a function as a target position setting unit 231B-1. The image acquisition unit 231A-1 has the same function as the image acquisition unit 231A shown in FIG. 2. The target position setting unit 231B-1 has the same function as the target position setting unit 231B shown in FIG. 2.

The robot control unit 2-2 is composed of a computer having a communication unit 21-2, a storage unit 22-2, and a processing unit 23-2.
The communication unit 21-2 is a communication interface, and has an interface circuit for connecting the robot control unit 2-2 to the actuator 13 of the robot 1, the image processing unit 2-1, etc., via, for example, a signal line. The storage unit 22-2 is a memory such as a ROM or RAM, or a storage such as a HDD or SDD. The storage unit 22-2 stores programs and various data used in the processing executed by the processing unit 23-2. The processing unit 23-2 is a processor, and has a function as a position calculation unit 232A-2 and a function as a control unit 232B-2. The position calculation unit 232A-2 has the same function as the position calculation unit 232A shown in FIG. 2. The control unit 232B-2 has the same function as the control unit 232B shown in FIG. 2.

Third Embodiment
The robot system 100 to which the robot control device 2 of the third embodiment is applied is configured similarly to the robot system 100 to which the robot control device 2 of the first embodiment described above is applied, except for the points described below.

As described above, in the robot system 100 to which the robot control device 2 of the first embodiment is applied, the actuator 13 is configured by a servo motor, and has an encoder as the actuator sensor 13A.
On the other hand, in a robot system 100 to which the robot control device 2 of the third embodiment is applied, the actuator 13 is configured by a stepping motor and does not have an actuator sensor.

In a robot system 100 to which the robot control device 2 of the third embodiment is applied, the position and speed of the rotation axis of the stepping motor serving as the actuator 13 are estimated based on a control signal (information indicating a drive pulse) transmitted from the robot control device 2 to the stepping motor serving as the actuator 13.
That is, in the robot system 100 to which the robot control device 2 of the third embodiment is applied, the position calculation unit 232A calculates the position (more specifically, the position in the X-direction, Y-direction, and Z-direction (X, Y, Z)) of the movable part 11B (hand) of the robot 1 connected to the visual sensor 14 via the connecting member 15, based on the positions of the rotation axes of each of the multiple actuators 13 (stepping motors) estimated based on the control signals (information indicating drive pulses) transmitted to the actuators 13 (stepping motors).

Fourth Embodiment
The robot system 100 to which the robot control device 2 of the fourth embodiment is applied is configured similarly to the robot system 100 to which the robot control device 2 of the first embodiment described above is applied, except for the points described below.

As described above, the robot 1 included in the robot system 100 to which the robot control device 2 of the first embodiment is applied is a six-axis vertical articulated robot.
On the other hand, the robot 1 included in the robot system 100 to which the robot control device 2 of the fourth embodiment is applied is a robot other than a six-axis vertical articulated robot (e.g., a horizontal articulated robot, a Cartesian coordinate robot, a parallel link robot, etc.).

According to the robot control device 2 of the first to fourth embodiments, even if the appearance of the detection target DT on the workpiece W1 moving in the transport direction contained in the image IM generated by the visual sensor 14 changes as the visual sensor 14, which is connected to the movable part 11B (hand) of the robot 1 and moves integrally with the movable part 11B (hand) of the robot 1, moves in the optical axis direction of the visual sensor 14, the position of the movable part 11B (hand) of the robot 1 can be appropriately controlled to follow the workpiece W1 moving in the transport direction based on the image IM generated by the visual sensor 14.

Although the embodiments of the robot control device, robot control method, and program of the present disclosure have been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible to these embodiments without departing from the gist of the invention, or without departing from the idea and intent of the present invention derived from the contents described in the claims and their equivalents. For example, in the above-mentioned embodiments, the order of each operation and the order of each process are shown as examples, and are not limited to these. The same applies when numerical values or formulas are used in the explanation of the above-mentioned embodiments. Furthermore, appropriate combinations of several of the above-mentioned embodiments are included in the scope of the present disclosure.

The following supplementary notes are further disclosed regarding the above-described embodiments and modifications.
(Appendix 1)
an image acquisition unit (231A; 231A-1) that acquires data of an image (IM) including a detection target (DT) on a workpiece (W1) moving in a conveying direction, the image (IM) being generated by a visual sensor (14) provided on a movable unit (11B) of the robot (1);
a storage unit (22; 22-1) that stores at least a first correspondence relationship that is a correspondence relationship between a first position (X1) of the movable part (11B) in an optical axis direction of the visual sensor (14), which is a direction intersecting the conveying direction, and a first detection target position (Hz1, Vt1) that is a target position on the image (IM) where the detection target (DT) should be located when the movable part (11B) is located at the first position (X1), and a second correspondence relationship that is a correspondence relationship between a second position (X2) of the movable part (11B) in the optical axis direction, and a second detection target position (Hz2, Vt2) that is a target position on the image (IM) where the detection target (DT) should be located when the movable part (11B) is located at the second position (X2);
a target position setting unit (231B; 231B-1) that sets a target position (Hz(X), Vt(X)) on the image (IM) where the detection target (DT) should be located, based on the first correspondence relationship, the second correspondence relationship, and a position (X) of the movable part (11B) in the optical axis direction.
(Appendix 2)
The robot control device (2) according to claim 1, wherein the optical axis direction is a direction substantially perpendicular to the conveying direction.
(Appendix 3)
The robot control device (2) according to appendix 1 or 2, further comprising a robot control unit (231; 2-2) that executes control of a position (X) of the movable part (11B) based on a position (Hz, Vt) of the detection target (DT) on the image (IM) and a target position (Hz(X), Vt(X)) on the image (IM) where the detection target (DT) should be located, which is set by the target position setting unit (231B; 231B-1).
(Appendix 4)
The robot control unit (2) described in Appendix 3 controls the position of the movable part (11B) only in the transport direction based on the position (Hz, Vt) of the detection target (DT) on the image (IM) and the target position (Hz(X), Vt(X)) on the image (IM) where the detection target (DT) should be located, which is set by the target position setting unit (231B; 231B-1), when the movable part (11B) is brought closer to the workpiece (W1) in the optical axis direction.
(Appendix 5)
The robot control unit (232; 2-2) controls the position (X) of the movable part (11B) in the optical axis direction based on information indicating the first position (X1) and information indicating the second position (X2) stored in the memory unit (22).
(Appendix 6)
The first position (X1) is a movement start position of the movable part (11B) in the optical axis direction,
The second position (X2) is a movement end position of the movable part (11B) in the optical axis direction,
the first correspondence relationship is a correspondence relationship between the movement start position (X1) and the first detection target position (Hz1, Vt1) which is a target position on the image (IM) where the detection target (DT) should be located when the movable part (11B) is located at the movement start position (X1);
the second correspondence relationship is a correspondence relationship between the movement end position (X2) and the second detection target position (Hz2, Vt2) which is a target position on the image (IM) where the detection target (DT) should be located when the movable part (11B) is located at the movement end position (X2),
The robot control device (2) according to any one of Appendices 1 to 5, wherein the target position setting unit (231B; 231B-1) sets a target position (Hz(X), Vt(X)) on the image (IM) where the detection target (DT) should be located when the position (X) of the movable part (11B) in the optical axis direction is neither the movement start position (X1) nor the movement end position (X2), based on the first correspondence relationship, the second correspondence relationship, and the position (X) of the movable part (11B) in the optical axis direction.
(Appendix 7)
The robot control device (2) described in any one of Appendices 1 to 6, wherein the first position (X1) of the movable part (11B) in the optical axis direction and the second position (X2) of the movable part (11B) in the optical axis direction are positions where the position and posture of the robot (1) are registered in robot teaching.
(Appendix 8)
The target position (Hz(X), Vt(X)) on the image (IM) where the detection target (DT) is to be positioned, which is set by the target position setting unit (231B; 231B-1), is represented by a horizontal coordinate value (Hz(X)) and a vertical coordinate value (Vt(X));
the horizontal coordinate value (Hz(X)) changes in proportion to a change in the position (X) of the movable part (11B) in the optical axis direction;
The robot control device (2) according to any one of appendices 1 to 7, wherein the vertical coordinate value (Vt(X)) changes in proportion to a change in the position (X) of the movable part (11B) in the optical axis direction.
(Appendix 9)
The robot control device (2) described in any of Appendices 1 to 8, wherein the target position setting unit (231B; 231B-1) sets a target position (Hz(X), Vt(X)) on the image (IM) where the detection target (DT) is to be located, based on the first correspondence relationship, the second correspondence relationship, and the position (X) of the movable part (11B) in the optical axis direction, when the position (X) of the movable part (11B) in the optical axis direction is neither the first position (X1) nor the second position (X2).
(Appendix 10)
an image acquisition step of acquiring data of an image (IM) including a detection target (DT) on a workpiece (W1) moving in a conveying direction, the image (IM) being generated by a visual sensor (14) provided on a movable part (11B) of the robot (1);
a storage step of storing at least a first correspondence relationship which is a correspondence relationship between a first position (X1) of the movable part (11B) in an optical axis direction of the visual sensor (14), which is a direction intersecting the conveying direction, and a first detection target position (Hz1, Vt1) which is a target position on the image (IM) where the detection target (DT) should be located when the movable part (11B) is located at the first position (X1), and a second correspondence relationship which is a correspondence relationship between a second position (X2) of the movable part (11B) in the optical axis direction, and a second detection target position (Hz2, Vt2) which is a target position on the image (IM) where the detection target (DT) should be located when the movable part (11B) is located at the second position (X2);
a target position setting step of setting a target position (Hz(X), Vt(X)) on the image (IM) where the detection target (DT) is to be located, based on the first correspondence relationship, the second correspondence relationship, and a position (X) of the movable part (11B) in the optical axis direction.
(Appendix 11)
On the computer,
an image acquisition step of acquiring data of an image (IM) including a detection target (DT) on a workpiece (W1) moving in a conveying direction, the image (IM) being generated by a visual sensor (14) provided on a movable part (11B) of the robot (1);
a storage step of storing at least a first correspondence relationship which is a correspondence relationship between a first position (X1) of the movable part (11B) in an optical axis direction of the visual sensor (14), which is a direction intersecting the conveying direction, and a first detection target position (Hz1, Vt1) which is a target position on the image (IM) where the detection target (DT) should be located when the movable part (11B) is located at the first position (X1), and a second correspondence relationship which is a correspondence relationship between a second position (X2) of the movable part (11B) in the optical axis direction, and a second detection target position (Hz2, Vt2) which is a target position on the image (IM) where the detection target (DT) should be located when the movable part (11B) is located at the second position (X2);
a target position setting step of setting a target position (Hz(X), Vt(X)) on the image (IM) where the detection target (DT) is to be located, based on the first correspondence relationship, the second correspondence relationship, and a position (X) of the movable part (11B) in the optical axis direction.

REFERENCE SIGNS LIST 100 Robot system 1 Robot 11 Link section 11A Base section 11B Movable section 12 Joint section 13 Actuator 13A Actuator sensor 14 Visual sensor 15 Connection member 2 Robot control device 21 Communication section 22 Memory section 23 Processing section 231 Image processing section 231A Image acquisition section 231B Target position setting section 232 Robot control section 232A Position calculation section 232B Control section C Conveyor W1 Work DT Detection target W2 Work IM Image

Claims (11)

an image acquisition unit that acquires image data including a detection target on a workpiece moving in a conveying direction, the image data being generated by a visual sensor provided on a movable part of the robot;
a storage unit that stores at least a first correspondence relationship that is a correspondence relationship between a first position of the movable part in an optical axis direction of the visual sensor that is a direction intersecting the transport direction and a first detection target position that is a target position on the image where the detection target should be located when the movable part is located at the first position, and a second correspondence relationship that is a correspondence relationship between a second position of the movable part in the optical axis direction and a second detection target position that is a target position on the image where the detection target should be located when the movable part is located at the second position;
a target position setting unit that sets a target position on the image where the detection object should be located based on the first correspondence relationship, the second correspondence relationship and the position of the movable part in the optical axis direction. The robot control device according to claim 1, wherein the optical axis direction is substantially perpendicular to the conveying direction. The robot control device according to claim 1 or 2, further comprising a robot control unit that controls the position of the movable part based on the position of the detection target on the image and the target position on the image where the detection target should be located, which is set by the target position setting unit. The robot control device according to claim 3, wherein when the movable part is brought closer to the workpiece in the optical axis direction, the robot control part controls the position of the movable part only in the transport direction based on the position of the detection target on the image and the target position on the image where the detection target should be located, which is set by the target position setting part. The robot control device according to claim 4, wherein the robot control unit controls the position of the movable part in the optical axis direction based on the information indicating the first position and the information indicating the second position stored in the memory unit. the first position is a movement start position of the movable part in the optical axis direction,
the second position is a movement end position of the movable part in the optical axis direction,
the first correspondence relationship is a correspondence relationship between the movement start position and the first detection target position which is a target position on the image where the detection target should be located when the movable part is located at the movement start position,
the second correspondence relationship is a correspondence relationship between the movement end position and the second detection target position which is a target position on the image where the detection target should be located when the movable part is located at the movement end position,
3. The robot control device according to claim 1 or 2, wherein the target position setting unit sets a target position on the image where the detection object should be located when the position of the movable part in the optical axis direction is neither the movement start position nor the movement end position, based on the first correspondence relationship, the second correspondence relationship, and the position of the movable part in the optical axis direction. The robot control device according to claim 1 or 2, wherein the first position of the movable part in the optical axis direction and the second position of the movable part in the optical axis direction are positions where the position and posture of the robot are registered in robot teaching. the target position on the image where the detection target is to be positioned, which is set by the target position setting unit, is represented by a horizontal coordinate value and a vertical coordinate value;
the horizontal coordinate value changes in proportion to a change in the position of the movable part in the optical axis direction;
The robot control device according to claim 1 , wherein the vertical coordinate value changes in proportion to a change in the position of the movable part in the optical axis direction. The robot control device according to claim 1 or 2, wherein the target position setting unit sets a target position on the image where the detection target should be located based on the first correspondence relationship, the second correspondence relationship, and the position of the movable part in the optical axis direction when the position of the movable part in the optical axis direction is neither the first position nor the second position. an image acquiring step of acquiring image data including a detection target on a workpiece moving in a conveying direction, the image data being generated by a visual sensor provided on a movable part of the robot;
a storage step of storing at least a first correspondence relationship which is a correspondence relationship between a first position of the movable part in an optical axis direction of the visual sensor which is a direction intersecting the transport direction and a first detection target position which is a target position on the image where the detection target should be located when the movable part is located at the first position, and a second correspondence relationship which is a correspondence relationship between a second position of the movable part in the optical axis direction and a second detection target position which is a target position on the image where the detection target should be located when the movable part is located at the second position;
a target position setting step of setting a target position on the image where the detection object should be located based on the first correspondence relationship, the second correspondence relationship and the position of the movable part in the optical axis direction. On the computer,
an image acquiring step of acquiring image data including a detection target on a workpiece moving in a conveying direction, the image data being generated by a visual sensor provided on a movable part of the robot;
a storage step of storing at least a first correspondence relationship which is a correspondence relationship between a first position of the movable part in an optical axis direction of the visual sensor which is a direction intersecting the transport direction and a first detection target position which is a target position on the image where the detection target should be located when the movable part is located at the first position, and a second correspondence relationship which is a correspondence relationship between a second position of the movable part in the optical axis direction and a second detection target position which is a target position on the image where the detection target should be located when the movable part is located at the second position;
a target position setting step of setting a target position on the image where the detection object should be located based on the first correspondence relationship, the second correspondence relationship and the position of the movable part in the optical axis direction.

Images