WO2024247135A1 - Control device, robot system, and robot control method - Google Patents

Abstract

A main control device 3 has a robot arm 12 and a hand 14 attached to the robot arm 12, and controls a robot 1 that performs pick-and-place processing of a workpiece. The main control device 3 comprises a searcher 41 that searches a plurality of candidates for the relative position and posture of the hand 14 with respect to a workpiece placed at a placement position candidate, and a determiner 42 that determines the relative position and posture of the hand 14 with respect to a workpiece W to be picked from a plurality of candidates for the relative position and posture of the hand at the time of placement.

Description

CONTROL DEVICE, ROBOT SYSTEM, AND ROBOT CONTROL METHOD

The technology disclosed herein relates to a control device, a robot system, and a method for controlling a robot.

Control devices that control robots that perform picking are known from the past. For example, the control device described in Patent Document 1 controls the robot arm and hand to pick up a workpiece and transport and place it at another location. This control device changes the gripping mode depending on the state of the workpiece to be picked, thereby realizing gripping of the workpiece without the robot arm and hand interfering with other objects.

JP 2004-230513 A

When picking a workpiece, a single target workpiece is often selected from multiple candidates, so interference between the robot arm and hand and other objects tends to be taken into consideration when selecting the target workpiece.

However, if only interference during picking is considered, it may be difficult to place the workpiece appropriately.

The technology disclosed here has been developed in light of these points, and its purpose is to properly pick and place workpieces.

The control device disclosed herein is a control device that controls a robot that has a robot arm and a hand attached to the robot arm and performs pick-and-place processing of a workpiece, and is equipped with a searcher that searches for multiple candidates for the relative position and orientation of the hand with respect to the workpiece to be placed in a candidate placement position, and a determiner that determines the relative position and orientation of the hand with respect to the workpiece to be picked from multiple candidates for the relative position and orientation of the hand at the time of placement.

The robot system disclosed herein comprises a robot having a robot arm and a hand attached to the robot arm, and the control device.

The robot control method disclosed herein is a method for controlling a robot that has a robot arm and a hand attached to the robot arm and performs pick-and-place processing of a workpiece, and includes searching for multiple candidates for the relative position and orientation of the hand with respect to a workpiece to be placed at a candidate placement position, and determining the relative position and orientation of the hand with respect to the workpiece to be picked from the multiple candidates for the relative position and orientation of the hand at the time of placement.

The control device allows the workpiece to be picked and placed appropriately.

The robot system can properly pick and place workpieces.

The robot control method described above allows the workpiece to be picked and placed appropriately.

FIG. 1 is a schematic diagram showing the configuration of a robot system. FIG. 2 is a diagram showing a schematic hardware configuration of the robot control device and the main control device 3. As shown in FIG. FIG. 3 is a block diagram showing the configuration of a control system of the processor. FIG. 4 is a view of the inside of the first container as seen from above. FIG. 5 is an exemplary perspective view showing the inside of the second container. FIG. 6 is a schematic plan view of the inside of the second container for explaining the search for position candidates. FIG. 7 is an explanatory diagram showing the hand when picking up the target workpiece. FIG. 8 is a schematic diagram showing the relative position and orientation of the hand when viewed in the direction of the reference axis. FIG. 9 is a flow chart of location planning. FIG. 10 is a flow chart of the place location planning subroutine. FIG. 11 is a flow chart of the pickup position planning subroutine. FIG. 12 is a timing chart of the placement position plan, the pickup position plan, and the robot operation. FIG. 13 is a flowchart of a position plan according to a modified example. FIG. 14 is a flow chart of a subroutine for correcting a place position. FIG. 15 is a timing chart of the placement position plan, the pickup position plan, and the robot operation according to the modified example.

Below, an exemplary embodiment will be described in detail with reference to the drawings. Figure 1 is a schematic diagram showing the configuration of a robot system 100.

The robot system 100 includes a robot 1 and a main control device 3 that controls the robot 1. The main control device 3 controls the robot 1 to cause the robot 1 to execute a process. In this example, the process performed by the robot 1 is a pick-and-place process. For example, the robot 1 picks up a target workpiece from a first container 91 in which multiple workpieces are piled up randomly, and transfers it into a second container 92. The robot 1 arranges the target workpieces in a regular pattern in the second container 92. The robot system 100 arranges the multiple workpieces in the first container 91 into the second container 92 by repeating the pick-and-place process. The target workpiece has, for example, a roughly rectangular parallelepiped shape.

The robot system 100 may include a camera 51 that captures an image of the workpiece. The camera 51 may include a first camera 51A that captures an image inside the first container 91 and a second camera 51B that captures an image inside the second container 92. The first camera 51A is fixedly disposed above the first container 91. The first camera 51A captures an image inside the first container 91 from above. The first camera 51A captures an image including a plurality of workpieces inside the first container 91. The second camera 51B is fixedly disposed above the second container 92. The second camera 51B captures an image inside the second container 92 from above. The second camera 51B captures an image including a plurality of workpieces inside the second container 92. Hereinafter, when the first camera 51A and the second camera 51B are not distinguished from each other, they will be simply referred to as cameras 51.

Here, the image is a two-dimensional image or a three-dimensional image. The three-dimensional image can be point cloud data, an RGB-D image, an RGB image, a depth image, or a voxel, etc. In other words, the camera 51 can be a three-dimensional camera, i.e., an RGB-D camera that outputs an RGB-D image, a stereo camera that acquires an RGB image, or a three-dimensional vision sensor that acquires point cloud data, etc.

-robot-
The robot 1 has a robot arm 12 and a hand 14 attached to the robot arm 12. In this example, the robot 1 is an industrial robot. The hand 14 is a so-called end effector. A base coordinate system of three orthogonal axes is defined in a space in which the robot 1 is disposed.

The robot arm 12 is configured to operate in a three-dimensional manner. Specifically, the robot arm 12 is configured to perform operations including translation with at least three degrees of freedom. In this example, the robot arm 12 is a vertical multi-joint robot arm. The robot arm 12 is supported by the base 10. The robot arm 12 has multiple links, multiple joints that connect the multiple links, and a servo motor that rotates the multiple joints.

In detail, the robot arm 12 has a first link 12a connected to the base 10, a second link 12b connected to the first link 12a, a third link 12c connected to the second link 12b, a fourth link 12d connected to the third link 12c, and a fifth link 12e connected to the fourth link 12d.

In detail, the base 10 and the first link 12a are connected to each other via a first joint 13a that can rotate around an axis extending in the vertical direction. The first link 12a and the second link 12b are connected to each other via a second joint 13b that can rotate around an axis extending in the horizontal direction. The second link 12b and the third link 12c are connected to each other via a third joint 13c that can rotate around an axis extending in the horizontal direction. The third link 12c and the fourth link 12d are connected to each other via a fourth joint 13d that can rotate around the axis of the fourth link 12d (i.e., the direction in which the fourth link 12d extends). The fourth link 12d and the fifth link 12e are connected to each other via a fifth joint 13e that can rotate around an axis perpendicular to the axis of the fourth link 12d.

The robot arm 12 has servo motors 16 (see Figure 2) that rotate each joint. Each servo motor 16 has an encoder 16a (see Figure 2).

The robot arm 12 thus configured is configured to perform translational movements in the directions of each of the three orthogonal axes, and rotational movements around each of the three orthogonal axes.

The hand 14 includes a suction device 15 that is positioned at a position offset in the radial direction around a predetermined reference axis T of the robot arm 12. The suction device 15 is positioned eccentrically with respect to the reference axis T. For example, the hand 14 has a base 14a that is attached to the robot arm 12. The base 14a extends in the radial direction around the reference axis T. The suction device 15 is attached to the end of the base 14a that is farther from the reference axis T via a plate, block, or the like.

Furthermore, the suction device 15 is configured to be able to change its position relative to the robot arm 12. For example, the hand 14 is attached to the robot arm 12 so as to be rotatable around the reference axis T. Specifically, the hand 14 is connected to the tip of the robot arm 12, i.e., the fifth link 12e. The fifth link 12e and the hand 14 are connected to each other via the sixth joint 13f so as to be rotatable around the reference axis T. The suction device 15 changes its relative position with respect to the robot arm 12 by rotating the hand 14 around the reference axis T.

The hand 14 sucks the workpiece with a suction device 15. The suction device 15 has a suction pad. An air hose that is connected to a negative pressure generator is connected to the suction device 15. The air hose is provided with a solenoid valve that serves as an actuator 15a (see Figure 2). By controlling the solenoid valve, the suction device 15 can switch between suction and release. The suction device 15 sucks the surface of the target workpiece W. The air hose is provided with a pressure sensor 15b (see Figure 2). The pressure sensor 15b detects the magnitude of the negative pressure in the air hose.

FIG. 2 is a diagram showing the general hardware configuration of the robot controller 2 and the main controller 3. The main controller 3 transmits and receives signals and information to and from the robot controller 2. The main controller 3 controls the robot 1 via the robot controller 2. The main controller 3 outputs commands to the robot controller 2. The robot controller 2 controls the servo motor 16 of the robot arm 12 in response to commands from the main controller 3.

Images from the first camera 51A and the second camera 51B are input to the main control device 3. The main control device 3 detects the target workpiece W based on the images. In addition, the main control device 3 generates a target path for the robot 1 based on the detected target workpiece W, and outputs a command corresponding to the generated target path to the robot control device 2. An actuator 15a and a pressure sensor 15b are connected to the main control device 3. The main control device 3 controls suction by the hand 14 via the actuator 15a and the pressure sensor 15b. The main control device 3 is an example of a control device.

The robot control device 2 has a processor 21, a memory 22, and a memory 23.

The processor 21 controls the entire robot control device 2. The processor 21 performs various types of arithmetic processing. For example, the processor 21 is formed of a processor such as a CPU (Central Processing Unit). The processor 21 may also be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, etc.

The memory 22 stores the programs executed by the processor 21 and various data. The memory 22 is formed of a non-volatile memory, a hard disc drive (HDD), a solid state drive (SSD), etc. The memory 23 temporarily stores data, etc. For example, the memory 23 is formed of a volatile memory.

The processor 21 drives the servo motor 16 based on a command from the main control device 3. At this time, the robot control device 2 feedback controls the current supplied to the servo motor 16 based on the detection result of the encoder 16a.

The main control device 3 has a processor 31, a memory 32, and a memory 33.

The processor 31 controls the entire main control device 3. The processor 31 performs various types of arithmetic processing. For example, the processor 31 is formed of a processor such as a CPU (Central Processing Unit). The processor 31 may also be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, etc.

The memory 32 stores the programs and various data executed by the processor 31. The memory 32 is formed of a non-volatile memory, a HDD (Hard Disc Drive), an SSD (Solid State Drive), etc.

For example, the memory 32 stores an object detection program that detects objects, a position and orientation determination program that determines the position and orientation of the hand 14, and a path generation program that generates a target path for the robot 1. The various programs cause the main control device 3 to realize various functions.

Memory 33 temporarily stores data, etc. For example, memory 33 is formed of a volatile memory. Memory 33 stores images from camera 51.

The processor 31 causes the robot 1 to execute a picking operation in which the hand 14 moves to the target workpiece W and causes the suction device 15 to suction the target workpiece W, and a placing operation in which the target workpiece W adsorbed by the hand 14 moves to a target placement position and causes the suction device 15 to release the suction of the target workpiece W. The processor 31 causes the robot 1 to execute a placing operation after the picking operation.

Prior to a picking operation, the processor 31 determines the relative position and orientation of the hand 14 with respect to the workpiece at the time of picking. Prior to a placing operation, the processor 31 determines the target placement position of the target workpiece W at the time of placement. Hereinafter, the relative position and orientation of the hand 14 with respect to the workpiece will be referred to simply as the "relative position and orientation of the hand 14."

FIG. 3 is a block diagram showing the configuration of the control system of the processor 31. The processor 31 realizes various functions by reading out a program from the storage device 32 to the memory 33 and expanding it. In detail, the processor 31 functions as a searcher 41 that searches for candidates for the relative position and orientation of the hand 14 with respect to the workpiece to be placed at the candidate placement position in the storage space, and a determiner 42 that determines the relative position and orientation of the hand 14 with respect to the target workpiece to be picked. Furthermore, the processor 31 functions as an operation controller 43 that causes the robot arm 12 and the hand 14 to execute robot operations including a picking operation for picking the target workpiece W and a placing operation for placing the target workpiece W at the placement position. The processor 31 can also function as a photography controller 44 that causes the camera 51 to acquire an image, a workpiece detector 45 that detects the workpiece based on the image, a height detector 46 that detects height information of the storage space based on the image, and a path generator 47 that generates a target path for the robot arm 12 and the hand 14.

The photography controller 44 controls the first camera 51A and the second camera 51B to capture images from each of the first camera 51A and the second camera 51B. The photography controller 44 causes the first camera 51A to perform picking photography to capture images inside the first container 91. The photography controller 44 causes the second camera 51B to perform place photography to capture images inside the second container 92. The photography controller 44 stores the images from the cameras 51 in the memory 33.

The work detector 45 detects a work from within an image by image recognition. The image may include multiple workpieces. The work detector 45 performs plane detection, for example, by RANSAC (Random Sample Consensus) to detect the surface of the workpiece (hereinafter referred to as the "target surface S"). In other words, the work detector 45 detects a workpiece by detecting a plane of a predetermined shape included in the image. FIG. 4 is a view of the inside of the first container 91 from above. The work detector 45 can detect multiple target surfaces S, i.e., multiple workpieces. When detecting the target surface S, the work detector 45 determines at least the center position of the target surface S, the normal direction of the target surface S, the vertical and horizontal orientations of the target surface S, and the size of the target surface S (e.g., the vertical and horizontal sizes). The size of the target surface S may be stored in advance in the memory 32.

The height detector 46 detects height information of the storage space in which the target workpiece W is to be stored. In this example, the storage space is inside the second container 92. The height information is information about the height of each position in the storage space, specifically, the height information of each position inside the second container 92. In areas inside the second container 92 where the bottom surface is exposed, the height information is the height of each position on the bottom surface. In areas inside the second container 92 where the workpiece is present, the height information is the height of each position on the top surface of the workpiece. The height detector 46 detects the height information of the storage space based on an image inside the second container 92. The height detector 46 obtains the height information of the storage space, for example, by performing preprocessing such as discretization or smoothing from the point cloud data of the three-dimensional image. The height detector 46 outputs the height information of the storage space as a height map.

The path generator 47 generates a target path for the robot arm 12 and the hand 14. The target path is an ordered set of target positions and target postures of the robot arm 12 and the hand 14. That is, the target path is a change in the target position and target posture of the robot arm 12 and the hand 14 over time. The target path may be an ordered set of target rotation angles of each joint of the robot arm 12 (including the sixth joint 13f) that realizes the change in the target position and target posture of the robot arm 12 and the hand 14 over time. The target path uniquely defines the position and posture of the robot arm 12 and the hand 14. The target path also defines the target rotation position of the suction device 15 around the reference axis T. For example, the path generator 47 generates a target path for the robot arm 12 and the hand 14 for a picking operation. The path generator 47 generates a target path for the robot arm 12 and the hand 14 for a placing operation.

For example, the path generator 47 executes path planning using the Probabilistic Roadmap Method (PRM) or the Rapidly-Exploring Random Tree (RRT), etc. In these methods, the path generator 47 also executes an interference check, i.e., checks whether the robot arm 12 and the hand 14 interfere with other objects. The interference check may be, for example, an interference check using point cloud data or CAD data.

The searcher 41 searches for candidates for the relative position and orientation of the hand 14 with respect to the workpiece to be placed at a candidate place position in the storage space. The searcher 41 searches for multiple candidates for the relative position and orientation of the hand 14. The place position candidates corresponding to the multiple candidates for the relative position and orientation of the hand 14 may be the same as each other, or may be different from each other. In other words, the searcher 41 may search for multiple different candidates for the relative position and orientation of the hand 14 with respect to one candidate position. Alternatively, the searcher 41 may search for multiple candidates for the relative position and orientation of the hand 14 with respect to different candidate positions. In this case, the candidates for the relative position and orientation of the hand 14 may be different from each other, or may be the same as each other.

Specifically, the searcher 41 searches for multiple place candidates, which are combinations of position candidates and candidates for the relative position and orientation of the hand 14 with respect to the workpiece placed at the position candidates. Each of the multiple place candidates has a different combination of position candidates and candidates for the relative position and orientation of the hand 14. At least some of the position candidates included in the multiple place candidates may be the same as each other. At least some of the candidates for the relative position and orientation of the hand 14 included in the multiple place candidates may be the same as each other.

The searcher 41 may search not only for candidates for the relative position and orientation of the hand 14, but also for candidates for the placement position. For example, the searcher 41 detects a space in the storage space where a work can be placed (hereinafter referred to as a "placeable space"). The searcher 41 detects a plane P in the storage space where a work can be placed (hereinafter referred to as a "placeable surface") based on the height map, and sets the space above the placeable surface P as the placeable space. Figure 5 is an exemplary perspective view showing the inside of the second container 92. In Figure 5, the surface marked with dots is the placeable surface P. The placeable surface P is a plane exposed upward and is an area of uniform height in the height map. The placeable surface P is an exposed portion of the bottom surface of the second container 92. If the top surface of the existing work W0 is also exposed, it can become the placeable surface P. In addition, when multiple existing workpieces W0 are lined up, the upper surfaces of the existing workpieces W0 are exposed upward, and the heights of the upper surfaces of the existing workpieces W0 are uniform, the upper surfaces of the existing workpieces W0 become a single placeable surface P. In the example of FIG. 5, the upper surfaces of the two existing workpieces W0 are each treated as a separate placeable surface P.

The searcher 41 determines whether or not the work can be placed at each position while changing the position of the work on the placement surface P. FIG. 6 is a schematic plan view of the inside of the second container to explain the search for position candidates. The positions determined to be possible for placement become position candidates. Whether or not the work can be placed is determined based on whether or not the work can be placed without interfering with other objects. At this time, the searcher 41 sets the size of the work for search (see dashed line) to the actual size of the work (see dashed line) plus a predetermined margin. Specifically, the searcher 41 enlarges the bottom surface of the work by the amount of the predetermined margin. The size of the work may be stored in the memory 32 in advance. Alternatively, the size of the work may be identified when the work detector 45 detects the work.

In this way, the searcher 41 searches for positions in the storage space where the work can be placed as candidate placement positions.

The searcher 41 finds candidates for the relative position and orientation of the hand 14 with respect to the workpiece to be placed at the candidate position.

In this example, the relative position and orientation of the hand 14 is determined by the position of the suction device 15 with respect to the workpiece, the position of the reference axis T, and the orientation of the reference axis T. FIG. 7 is an explanatory diagram showing the hand 14 when suctioning the target workpiece W. Since the suction device 15 is eccentric with respect to the reference axis T, even if the position of the suction device 15 with respect to the workpiece, i.e., the suction position, is determined, the reference axis T can be positioned at any position within 360 degrees around the suction device 15. In this example, the position of the suction device 15 and the orientation of the reference axis T among the relative position and orientation of the hand 14 are determined. The suction position of the suction device 15 is the center of the target surface S. The reference axis T is approximately perpendicular to the target surface S. The position of the reference axis T among the relative position and orientation of the hand 14 can be set to any angle position within 360 degrees around the suction device 15.

In this way, since the position of the reference axis T can be set arbitrarily, the hand 14 or robot arm 12 may interfere with other objects when placing the workpiece. In particular, when the workpiece is placed in a space that includes a corner within the storage space, there is a risk that the hand 14 or robot arm 12 may interfere with the side wall of the second container 92 that forms the corner or the side surface of the existing workpiece W0. When the workpiece is placed in such a candidate position, the appropriate relative position and orientation of the hand 14 is one in which the reference axis T is away from the side wall of the second container 92, etc.

For example, the memory 32 stores the relative positions and orientations of multiple hands 14 with different angular positions of the reference axis T centered on the suction device 15. Figure 8 is a schematic diagram showing the relative positions and orientations of the hand 14 when viewed in the direction of the reference axis T. For example, as shown in Figure 8, the memory 32 may store the relative positions and orientations of 12 types of hands 14 with different angular positions of the reference axis T centered on the suction device 15 at 30 degree intervals. The searcher 41 creates a place candidate by combining one of the multiple relative positions and orientations of the hand 14 with one of multiple position candidates.

The searcher 41 creates, as a place candidate, a combination of a position candidate and a relative position and orientation of the hand 14, where the hand 14 and robot arm 12 can hold a workpiece without interfering with other objects. Specifically, the searcher 41 determines whether or not a target path for the hand 14 and robot arm 12 corresponding to the place candidate can be generated. In other words, the searcher 41 sets, as a place candidate, a combination of a position candidate and a candidate relative position and orientation of the hand 14, where a target path for the hand 14 and robot arm 12 that does not interfere with other objects can be generated.

Specifically, the path generator 47 generates a target path for the hand 14 and robot arm 12 to move the hand 14 to a state corresponding to the position candidate and the candidate for the relative position and orientation of the hand 14. In the placing operation, an intermediate position through which the robot arm 12 and the hand 14 pass is set, so the path generator 47 generates a target path from the intermediate position. At this time, the path generator 47 performs an interference check of the target path to confirm the presence or absence of interference. In other words, the path generator 47 generates a target path without interference. If interference is present, the path generator 47 does not generate a target path. When a target path is generated, the searcher 41 sets a combination of the position candidate at that time and the candidate for the relative position and orientation of the hand 14 as a placing candidate.

The searcher 41 registers the place candidates. That is, the searcher 41 stores the combination of the position candidate and the relative position and orientation of the hand 14 in the memory 33 or the storage device 32. The searcher 41 further associates the target route with the place candidate and stores it in the memory 33 or the storage device 32.

In this way, the searcher 41 searches for multiple place candidates.

Furthermore, the searcher 41 assigns priorities to multiple place candidates. Specifically, the searcher 41 assigns priorities to position candidates included in the place candidates. For example, the searcher 41 assigns priorities to multiple position candidates from the viewpoint of work storage efficiency. The searcher 41 may assign a high priority to a position candidate corresponding to a corner space in the storage space from among the multiple position candidates. When storing the work in the second container 92, it is preferable from the viewpoint of work storage efficiency to pack the work into a space including a corner. In addition, when storing the work in the second container 92, it is also preferable from the viewpoint of work storage efficiency to have the side of the work, for example, the side surface, come into contact with the side wall of the second container 92 or the side surface of the existing work W0. Therefore, the searcher 41 may assign a high priority to a position candidate where the side of the work comes into contact with another object from among the multiple position candidates.

The searcher 41 may assign priorities to multiple position candidates from the viewpoint of storage stability. For example, the searcher 41 may assign a higher priority to a position candidate that is lower in height from the bottom surface of the second container 92. Placing the work on the bottom surface of the second container 92 allows the work to be placed more stably than placing the work on top of the existing work W0.

In this way, the searcher 41 searches for multiple place candidates, which are combinations of candidate placement positions and corresponding candidates for the relative position and orientation of the hand 14, and assigns priorities to the multiple place candidates.

The determiner 42 determines the relative position and orientation of the hand 14 with respect to the target workpiece to be picked from among multiple candidates for the relative position and orientation of the hand 14 at the time of placing. Specifically, the determiner 42 determines the relative position and orientation of the hand 14 at which the hand 14 and robot arm 12 can hold the target workpiece without interfering with other objects. Specifically, the determiner 42 judges whether or not the hand 14 can generate a target path for the hand 14 and robot arm 12 to pick up the target workpiece in the relative position and orientation. In other words, the determiner 42 determines the relative position and orientation of the hand 14 that can generate a target path for the hand 14 and robot arm 12 not interfering with other objects from among multiple candidates for the relative position and orientation of the hand 14 included in multiple place candidates.

In detail, the determiner 42 provisionally selects one relative position and posture of the hand 14 from among the relative positions and postures of the multiple hands 14. The path generator 47 generates a target path for the hand 14 and the robot arm 12 to pick up the target workpiece at the selected relative position and posture of the hand 14. In the picking operation, an intermediate position through which the robot arm 12 and the hand 14 pass is set, so the path generator 47 generates a target path from the intermediate position and posture. At this time, the path generator 47 performs an interference check of the target path to confirm the presence or absence of interference. In other words, the path generator 47 generates a target path without interference. If interference is present, the path generator 47 does not generate a target path. If the target path is generated, the determiner 42 determines the provisional relative position and posture of the hand 14 as the relative position and posture of the hand 14 at the time of picking. If the target path is not generated, the determiner 42 selects another relative position and posture of the hand 14, and the path generator 47 attempts to generate a target path for the selected relative position and posture of the hand 14.

When there are multiple options for the target workpiece, i.e., when selecting a target workpiece from multiple workpieces, the determiner 42 also determines the target workpiece from the multiple works when determining the relative position and posture of the hand 14. When determining the relative position and posture of the hand 14 that can pick up the target workpiece from the relative positions and postures of multiple hands 14, the more options there are for the target workpiece, the wider the range of choices for the relative position and posture of the hand 14.

By determining the relative position and orientation of the hand 14 during picking, a candidate position defined by a place candidate corresponding to the determined relative position and orientation of the hand 14 is determined as a place target position. In other words, the determiner 42 determines the relative position and orientation of the hand 14 during picking, and also determines the place target position. Furthermore, the determiner 42 also generates a target path for picking, as described above.

The motion controller 43 causes the robot arm 12 and hand 14 to execute robot motions including a picking motion for picking up the target workpiece W and a placing motion for placing the target workpiece W at a target placement position. The motion controller 43 outputs a command according to the target path to the robot control device 2. Specifically, the motion controller 43 outputs a command according to the target rotation angle of each joint of the robot arm 12 to the robot control device 2. As described above, the robot control device 2 drives the servo motor 16 based on the command, causing the robot arm 12 and hand 14 to move along the target path.

The operation controller 43 switches between suction by the suction device 15 and release of the suction by controlling the actuator 15a of the hand 14. The operation controller 43 controls the actuator 15a to generate negative pressure in the suction device 15, thereby causing the suction device 15 to suction the workpiece. The operation controller 43 controls the actuator 15a to release the negative pressure in the suction device 15, thereby causing the suction device 15 to release the suction of the workpiece. The operation controller 43 determines whether suction of the target workpiece W by the suction device 15 has been completed based on the detection result of the pressure sensor 15b. The operation controller 43 determines whether suction has been completed when the negative pressure detected by the pressure sensor 15b is equal to or greater than a predetermined threshold value.

Next, the operation of the robot system 100 configured in this manner will be described. In the pick-and-place process of the robot system 100, position planning and robot operation are performed in parallel. Position planning includes a place position plan and a pickup position plan. Robot operation includes a picking operation and a placing operation. Position planning will be described with reference to the flowchart in Figure 9. Figure 9 is a flowchart of position planning.

First, in step S101, the photography controller 44 causes the second camera 51B to perform place photography. The place photography is photography of the inside of the second container 92. As a result, an image of the inside of the second container 92 is obtained.

Next, in step S102, the searcher 41 executes a place position plan. The searcher 41 searches for multiple place candidates. At this time, the searcher 41 also generates a target path for the robot arm 12 and hand 14 for each of the place candidates.

Meanwhile, in parallel with steps S101 and S102, in step S103, the photography controller 44 causes the first camera 51A to perform picking photography. The picking photography is photography of the inside of the first container 91. As a result, an image of the inside of the first container 91 is acquired. Note that step S103 may be performed before or after step S101. However, if the placement position planning is performed using the size of the work W identified by the work detector 45, steps S103 and S104 are performed before step S102.

Next, in step S104, the work detector 45 detects a work from within the image of the first container 91. The detected work is simply referred to as the "detected work."

Next, in step S105, the determiner 42 executes a pickup position plan. The determiner 42 determines the target work W to be picked, and determines the relative position and orientation of the hand 14 with respect to the target work W. By determining the relative position and orientation of the hand 14, a position candidate specified by the corresponding place candidate is determined as the place target position. Furthermore, in the pickup position plan, a target path of the robot arm 12 and the hand 14 for picking up the target work W is also generated.

In this way, the position planning determines the target workpiece W, the relative position and orientation of the hand 14, the target placement position, the target path for picking, and the target path for placing.

The motion controller 43 performs picking and placing operations by moving the robot arm 12 and hand 14 along the target path.

Specifically, the motion controller 43 controls the robot arm 12 and the hand 14 so as to adsorb the target workpiece W in the relative position and posture of the hand 14. Specifically, the motion controller 43 moves the robot arm 12 and the hand 14 along the target picking path, and moves the suction device 15 to the target workpiece W. The motion controller 43 causes the suction device 15 to adsorb the target workpiece W.

Then, the motion controller 43 controls the robot arm 12 and hand 14 so that the target workpiece W is placed at the target placement position. Specifically, the motion controller 43 moves the robot arm 12 and hand 14 along the target placement path, and moves the target workpiece W to the target placement position. When the target workpiece W reaches the target placement position, the motion controller 43 releases the suction by the suction device 15. In this way, the placement of the target workpiece W at the target placement position is completed. This completes the pick-and-place process.

When the main control device 3 completes the pick-and-place process for one target workpiece W, it executes the pick-and-place process for another target workpiece W. By repeating this type of control, the main control device 3 sequentially transfers the target workpieces W in the first container 91 into the second container 92.

=Place location plan=
Next, the place position planning will be described in detail with reference to Fig. 10, which is a flow chart of the place position planning subroutine.

In step S201, the height detector 46 creates a height map of the storage space, i.e., the interior of the second container 92.

Next, in step S202, the searcher 41 searches for position candidates for the place position. In detail, the searcher 41 creates multiple position candidates for the workpiece within the second container 92.

Next, in step S203, the searcher 41 assigns priorities to the location candidates. As described above, the searcher 41 assigns priorities to each of the multiple location candidates, for example, from the perspective of accommodation efficiency or accommodation stability.

In step S204, the searcher 41 reads out one relative position and orientation from the multiple relative positions and orientations of the hand 14 stored in the memory 32. In step S205, the searcher 41 reads out the position candidate with the highest priority from the multiple position candidates.

In step S206, the path generator 47 generates a target path for placing the workpiece at the read-out position candidate with the hand 14 in the read-out relative position and orientation adsorbing the workpiece. At this time, the path generator 47 checks for interference with the target path and generates a target path without interference. If interference is present, the path generator 47 does not generate a target path.

In step S207, the searcher 41 determines whether or not a target path has been generated. If a target path has been generated, the searcher 41 sets the current relative position and orientation as a candidate for the relative position and orientation of the hand 14. In step S208, the searcher 41 registers a combination of the position candidate and the candidate for the relative position and orientation of the hand 14 as a place candidate.

On the other hand, if a target path has not been generated, it means that the workpiece cannot be placed appropriately with the combination of the current position candidate and relative position and orientation. Therefore, if a target path has not been generated, the searcher 41 determines in step S209 whether or not it has attempted to generate a target path with the current relative position and orientation for all position candidates. If it has not attempted to generate a target path for all position candidates, the searcher 41 returns to step S205 and reads out the position candidate with the next highest priority from the multiple position candidates. The path generator 47 then attempts to generate a target path for the new position candidate (step S206), and the searcher 41 determines whether or not a target path has been generated (step S207).

In this way, the searcher 41 searches for position candidates in which the workpiece can be appropriately placed in the current relative position and orientation, in descending order of priority. When the searcher 41 finds a position candidate in which the workpiece can be appropriately placed, it associates the position candidate with the current relative position and orientation.

However, if target path generation has already been attempted for all position candidates in step S209, the multiple position candidates do not include a position where the workpiece can be appropriately placed at the current relative position and orientation. In that case, no place candidate is registered for the current relative position and orientation. The searcher 41 returns to step S204 and searches for a place candidate for another relative position and orientation. Specifically, the searcher 41 reads out another relative position and orientation from the multiple relative positions and orientations of the hand 14 stored in the memory 32, and repeats the processing from step S205 onwards.

After the place candidates are registered in step S208, the searcher 41 determines in step S210 whether the search for place candidates has been completed for all relative positions and orientations. If the search for place candidates has not been completed for all relative positions and orientations, the searcher 41 returns to step S204 and searches for place candidates for another relative position and orientation.

When the search for place candidates has been completed for all relative positions and orientations, the searcher 41 ends the place position planning.

By this type of place position planning, multiple place candidates are created, which are combinations of position candidates and candidates for the relative position and orientation of the hand 14. In other words, a set of place candidates is created by the place position planning.

=Pickup position planning=
Next, the pickup position planning will be described in detail with reference to a flowchart of a subroutine for the pickup position planning.

In step S301, the determiner 42 reads out one relative position/posture candidate from among the multiple relative position/posture candidates of the hand 14 included in the multiple place candidates. Specifically, the determiner 42 reads out the relative position/posture candidate of the hand 14 that is defined by the place candidate with the highest priority among the multiple place candidates.

Next, in step S302, the determiner 42 associates the candidates for relative position and orientation with each detected work. Specifically, the determiner 42 determines the absolute or controlled position and orientation of the hand 14 when the hand 14 is placed with respect to each detected work at the candidate relative position and orientation. For example, this hand position and orientation is not a relative position and orientation with respect to the work, but the position and orientation of the hand 14 in a coordinate system (e.g., a robot coordinate system, etc.) used to control the robot 1. In step S302, combinations of the detected work and the position and orientation of the hand 14 (this combination is simply referred to as a "combination of detected work and hand 14") are created according to the number of detected works.

Next, in step S303, the determiner 42 assigns a priority to each combination of the detected workpiece and the hand 14. For example, the determiner 42 assigns a high priority to a combination that includes a position and orientation of the hand 14 that has a small amount of movement from the current position of the hand 14.

In step S304, the determiner 42 reads out one combination from among multiple combinations of the detection workpiece and the hand 14. At this time, the determiner 42 reads out the combination with the highest priority.

Then, in step S305, the path generator 47 generates a target path for the robot arm 12 to the position and posture of the hand 14, which is defined by the combination read from the current position of the hand 14. At this time, the path generator 47 performs an interference check of the target path, and generates a target path without interference. Depending on the state of surrounding objects, the path generator 47 may not be able to generate a target path without interference.

In step S306, the determiner 42 determines whether an interference-free target path has been generated. If an interference-free target path has not been generated, in step S307, the determiner 42 determines whether an attempt has been made to generate target paths for all of the multiple combinations of the detected workpiece and the hand 14. If target path generation has not been attempted for all combinations, the determiner 42 returns to step S304 and reads out the combination with the next highest priority from the multiple combinations. The path generator 47 then attempts to generate a target path for the new combination (step S305), and the determiner 42 determines whether a target path has been generated (step S306).

In this way, the determiner 42 searches for detected workpieces for which an appropriate target path can be generated from the candidates for the relative position and posture of the hand 14 set in step 301, in descending order of priority. When an appropriate target path is generated, the determiner 42 determines the target path as the target path at the time of picking in step S308. In other words, the determiner 42 determines the detected workpiece at the time the appropriate target path was generated as the target workpiece W, and determines the candidate for the relative position and posture of the hand 14 at that time as the relative position and posture of the hand 14 at the time of picking. When the relative position and posture of the target workpiece W and the hand 14 for picking are determined, the determiner 42 ends the suction position planning. Furthermore, by determining the relative position and posture of the hand 14 for picking, the position candidate specified by the corresponding place candidate is determined as the place target position.

However, if the generation of the target path has already been tried for all combinations in step S307, there is no detection work that can be appropriately picked with the candidate relative position and orientation of the hand 14 set this time. In that case, in step S309, the determiner 42 determines whether the search for detection work that can be appropriately picked with respect to the candidates for the relative position and orientation of the hand 14 defined in all place candidates has been completed. If the search for detection work with respect to the candidates for the relative position and orientation of the hand 14 defined in all place candidates has not been completed, the determiner 42 returns to step S301 and reads out the candidate for the relative position and orientation of the hand 14 defined in the place candidate with the next highest priority among the multiple place candidates. Then, the processing from step S302 onwards is repeated for the newly read candidate for the relative position and orientation. In other words, the determiner 42 searches for detection work that can generate an appropriate target path with the candidate for the relative position and orientation of the changed hand 14.

On the other hand, if the search for the detected workpiece has been completed for the relative positions and orientations of the hand 14 defined in all placement candidates, the determiner 42 ends the pick-and-place process in step S310.

In this way, the determiner 42 determines the relative position and orientation of the hand 14 with respect to the target workpiece W to be picked from among the candidates for the relative position and orientation of the hand 14 included in the place candidates. Furthermore, the determiner 42 determines the target workpiece W from among multiple works.

In addition, the target placement position of the target work W is also determined by determining the relative position and posture of the hand 14 during picking. The target paths for each placement candidate have already been created in the placement position plan. Therefore, once the target placement position is determined, the target path for the placement operation is also determined accordingly.

Next, the parallel processing of the place position plan, the pick-up position plan, and the robot operation will be explained using FIG. 12. FIG. 12 is a timing chart of the place position plan, the pick-up position plan, and the robot operation.

For place position planning, first, place photography is performed (S101). This results in an image of the inside of the second container 92 being acquired. Next, the searcher 41 performs place position planning based on the image of the inside of the second container 92 (S102). This results in multiple place candidates being obtained.

After the placement position planning is completed, the gripping position planning is executed (S105). Prior to the gripping position planning, picking photography (S103) and workpiece detection (S104) are executed. In detail, an image of the inside of the first container 91 is acquired by picking photography. A workpiece is detected based on the image of the inside of the first container 91. The gripping position planning is executed based on the image of the inside of the first container 91, the detected workpiece, and the placement candidate. The gripping position planning determines the target workpiece W and the relative position and orientation of the hand 14 with respect to the target workpiece W.

After the gripping position plan is completed, the picking operation is executed. The operation controller 43 causes the robot 1 to pick up the target workpiece W in the relative position and posture of the hand 14 determined by the gripping position plan.

After the picking operation, a placing operation is executed. The operation controller 43 operates the robot 1 according to the target path determined by the placing position plan and the gripping position plan. As a result, the target work W is placed at the placing target position.

In this way, one placement position planning, gripping position planning, picking operation, and placement operation are completed.

In such pick-and-place processing, some processes are performed in parallel with other processes. For example, place photography can be performed in parallel with pick photography, workpiece detection, gripping position planning, or picking operation. Place position planning can be performed in parallel with pick photography, workpiece detection, gripping position planning, picking operation, or placing operation. Pick photography can be performed in parallel with place photography, place position planning, or placing operation. Grip position planning can be performed in parallel with place photography, picking operation, or placing operation.

In this example, the place position plan and grip position plan for the next pick-and-place process are executed in parallel with the robot operation. Specifically, the place photography is executed in parallel with the picking operation in the previous pick-and-place process. Since the situation inside the second container 92 does not change during the picking operation, the next place photography can be executed during the picking operation. The pick photography is executed in parallel with the place operation in the previous pick-and-place process. Since the situation inside the first container 91 does not change during the placing operation, the pick photography can be executed.

Each of the place position plan and the grip position plan is executed in parallel with the picking operation and the placing operation in the previous pick-and-place process. Since each of the place position plan and the grip position plan is executed based on an image, each of the place position plan and the grip position plan can be executed during the robot operation.

In this way, several processes can be performed in parallel, reducing the cycle time of the pick-and-place process.

However, the placement position planning is performed based on an image captured before the completion of the placement operation of the previous pick-and-place process. In other words, the placement position planning is performed based on an image of the state in which the workpiece has not yet been placed in the previous placement operation. Therefore, in the placement position planning, the placement target position of the previous pick-and-place process is excluded and placement candidates are found.

<Modification>
Next, a modified example of the pick-and-place process will be described with reference to FIG 13, which is a flowchart of position planning according to the modified example.

As described above, the place position planning is performed based on the image captured by the place shooting before the completion of the previous placing operation, so in the place position planning, the place candidate is found excluding the previous placing target position. However, there is a possibility that the workpiece actually placed in the previous placing operation may be displaced from the placing target position. Alternatively, there is a possibility that the existing workpiece may move due to the previous placing operation. Therefore, the nth place shooting is performed after the n-2th (n is a natural number of 3 or more) placing operation and before the n-1th placing operation, and based on a comparison between the image captured by the nth place shooting and the image captured by the n-1th place shooting, it is determined whether or not the place target position of the n-1th placing operation needs to be corrected. At the time of the nth place shooting, the n-1th placing operation has not yet been performed, so the place target position of the n-1th placing operation can be corrected.

Note that the processes in steps S101, S102, S103, S104, and S105 in FIG. 13 are similar to the processes in steps S101, S102, S103, S104, and S105 in FIG. 9.

In more detail, first, in step S101, the shooting controller 44 causes the second camera 51B to perform place shooting.

Next, in step S401, the searcher 41 determines the difference between the image of the current, i.e., the nth place photograph and the image of the previous, i.e., the n-1th place photograph. Specifically, the image difference is determined based on the height map created by the height detector 46. The searcher 41 detects the difference between the image of the nth place photograph and the image of the n-1th place photograph by the difference between the height map of the image of the nth place photograph and the height map of the image of the n-1th place photograph. The n-1th place photograph is performed before the n-2th place operation. The nth place photograph is performed after the n-2th place operation and before the n-1th place operation. Therefore, the difference in the height map corresponds to the change in the situation inside the second container 92 due to the n-2th place operation.

Next, in step S402, the searcher 41 determines the need to correct the placement target position of the previous, i.e., the n-1th, placing operation based on the image difference. Note that the n-1th placing operation has not yet been performed. The position of the workpiece placed in the n-2th placing operation can be determined from the difference between the image taken in the nth placing operation and the image taken in the n-1th placing operation. Furthermore, the movement of other workpieces due to the influence of the n-2th placing operation can also be determined from the difference between the image taken in the nth placing operation and the image taken in the n-1th placing operation.

For example, if the deviation between the target position of the n-2th placing operation and the position of the workpiece actually placed by the n-2th placing operation exceeds a predetermined range, the searcher 41 determines that the target position of the n-1th placing operation needs to be corrected. Alternatively, if another workpiece has entered the target position of the n-1th placing operation, the searcher 41 determines that the target position of the n-1th placing operation needs to be corrected.

If the searcher 41 determines that the (n-1)th place target position does not need to be corrected, it proceeds to step S102 and executes the current, i.e., nth, place position plan. The place position plan is as described above.

On the other hand, if the searcher 41 determines that the n-1th place target position needs to be corrected, then in step S403, it executes a place position correction to correct the n-1th place target position. The operation controller 43 executes the n-1th place operation based on the corrected place target position. After executing the place position correction, the searcher 41 executes the nth place position plan in step S102.

Note that, since the placement position correction involves correcting the (n-1)th placement target position in response to changes in the second container 92 resulting from the (n-2)th placement operation, steps S401, S402, and S403 are not executed until the second position planning. Step S401 is executed from the third position planning onwards.

Next, we will explain the place position correction in detail. Figure 14 is a flowchart of the place position correction subroutine.

First, in step S501, the searcher 41 searches for position candidates for the place position. In detail, the searcher 41 creates multiple position candidates for the workpiece in the second container 92. The searcher 41 searches for the position candidates using the height map created in step S401. Next, in step S502, the searcher 41 assigns priorities to the position candidates.

These steps S501 and S502 are the same processes as steps S202 and S203, respectively. However, these processes are for correcting the previous, i.e., n-1th, place target position, but are based on the image captured during the nth place shot.

Then, in step S503, the searcher 41 reads out the location candidate with the highest priority from among the multiple location candidates. This process is the same as step S205.

Then, in step S504, the path generator 47 generates a target path for placing the workpiece at the read-out position candidate. When this placement position correction is being performed, a picking operation is being performed, and the relative position and orientation of the hand 14 cannot be changed. Therefore, the path generator 47 generates a target path for placing the workpiece at the position candidate at the already determined relative position and orientation of the hand 14. At this time, the path generator 47 performs an interference check of the target path to confirm whether or not there is interference. In other words, the path generator 47 generates a target path without interference. If there is interference, the path generator 47 does not generate a target path.

In step S505, the searcher 41 determines whether or not a target route has been generated. This process is the same as in step S207. If a target route has been generated, in step S506, the searcher 41 determines the position candidate as the previous, i.e., the (n-1)th place target position, and sets the generated target route as the previous, i.e., the (n-1)th place operation target route.

On the other hand, if a target route has not been generated, the searcher 41 determines in step S507 whether or not an attempt has been made to generate a target route for all location candidates. This process is the same as step S209. If an attempt has not been made to generate a target route for all location candidates, the searcher 41 returns to step S503 and reads out the location candidate with the next highest priority from the multiple location candidates. The route generator 47 then attempts to generate a target route for the new location candidate (step S504), and the searcher 41 determines whether or not a target route has been generated (step S505).

If the searcher 41 has attempted to generate target paths for all position candidates, the searcher 41 ends the pick-and-place process in step S508.

In this way, the searcher 41 searches for a new placement target position from among the position candidates in descending order of priority without changing the relative position and orientation that has already been determined. When the searcher 41 finds a position candidate where the workpiece can be appropriately placed, it corrects the placement target position of the n-1th robot operation, i.e., the placement operation in the currently ongoing robot operation, to the new position candidate.

This place position correction is completed by the time the place operation in the currently ongoing robot operation is started. After the place position correction, the place position plan (S102) for the next robot operation of the currently ongoing robot operation is executed.

Such parallel processing of position planning will be described with reference to FIG. 15. FIG. 15 is a timing chart of the place position planning, pickup position planning, and robot operation in the modified example.

In the example of FIG. 15, a third place photograph is taken in parallel with the second picking operation. After the third place photograph, the searcher 41 determines the difference between the image taken by the third place photograph and the image taken by the second place photograph (S401), and determines whether or not the place target position of the second place operation needs to be corrected (S402). The second place photograph is taken before the first place operation, and the third place photograph is taken after the first place operation. Therefore, by comparing the image taken by the third place photograph with the image taken by the second place photograph, the change in the situation inside the second container 92 caused by the first place operation can be determined.

When the searcher 41 determines that the second place target position needs to be corrected, it executes the place position correction (S403). The second place target position is corrected by the place position correction, and a target path is generated accordingly. The place position correction is completed before the second place operation. The operation controller 43 executes the second place operation according to the corrected place target position and target path.

After completing the place position correction, the searcher 41 executes the third place position plan.

In this way, when position planning and robot operation are performed in parallel, the next place position plan can be executed based on the image of the storage space before the placing operation is completed. Therefore, the nth place photograph is executed before the n-1th placing operation, and the need to correct the n-1th placing target position is determined using the image from the nth place photograph. Then, if correction of the placing target position is necessary, the placing target position is corrected to a position where the work can be appropriately placed with the current relative position and orientation of the hand 14. This allows the work to be appropriately placed while shortening the cycle time of the pick-and-place process.

In such a pick-and-place process, multiple candidates for the relative position and orientation of the hand 14 corresponding to candidate positions for placing the workpiece are created, and the relative position and orientation of the hand 14 at the time of picking is determined from the multiple candidates for the relative position and orientation of the hand 14. This allows the workpiece to be picked at a relative position and orientation of the hand 14 that allows the workpiece to be placed at the candidate position. As a result, it is possible to determine how to pick the workpiece after taking into consideration the placement of the workpiece in advance. Furthermore, multiple candidates for the relative position and orientation of the hand 14 corresponding to candidate positions for placing are created, but the relative position and orientation of the hand 14 at the time of picking is determined from among them, so there is no need to calculate multiple relative positions and orientations of the hand 14 at the time of picking. In other words, the amount of calculation required to determine the relative position and orientation of the hand 14 at the time of picking can be reduced.

In addition, when there are multiple position candidates, priorities are assigned to the multiple position candidates. Then, it is determined whether or not the workpiece can be picked up, starting from the relative position and posture of the hand 14 that corresponds to the position candidate with the highest priority. Therefore, the relative position and posture of the hand 14 at the time of picking can be made to correspond to the position candidate with the highest priority as much as possible. As a result, it is possible to place the workpiece at a position candidate with a relatively high priority.

Furthermore, when determining the relative position and orientation of the hand 14 during picking, the target work W to be picked is determined from among multiple workpieces at the same time. In other words, it is possible to test multiple workpieces to see whether one of multiple candidates for the relative position and orientation of the hand 14 during placing can be used to properly pick the workpiece. In other words, it is possible to increase the options available when determining whether a workpiece can be picked using a candidate for the relative position and orientation of the hand 14.

Other Embodiments
As described above, the above embodiment has been described as an example of the technology disclosed in this application. However, the technology in this disclosure is not limited to this, and can be applied to embodiments in which modifications, replacements, additions, omissions, etc. are appropriately performed. In addition, it is also possible to combine the components described in the above embodiment to form a new embodiment. In addition, among the components described in the attached drawings and detailed description, not only components essential for solving the problem but also components that are not essential for solving the problem in order to exemplify the technology may be included. Therefore, the fact that these non-essential components are described in the attached drawings and detailed description should not immediately be taken to mean that these non-essential components are essential.

For example, the robot system 100 incorporating the main control device 3 is not limited to one that transfers the target work W from the first container 91 to the second container 92. For example, the robot 1 may perform debunking. The storage location of the target work W is not limited to the second container 92. For example, the storage location of the target work W may be a conveyor, a shelf, a pallet, or the like.

The robot arm 12 is not limited to a vertical multi-joint type robot arm. The robot arm 12 may be a horizontal multi-joint type, a parallel link type, a Cartesian coordinate type, or a polar coordinate type robot arm, etc.

The suction device 15 does not have to rotate around the reference axis T relative to the robot arm 12. Even if the suction device 15 cannot rotate around the reference axis T, it only needs to be positioned at a position offset in the radial direction around the reference axis T of the robot arm 12, that is, positioned eccentrically with respect to the reference axis T. The suction device 15 may attract the workpiece by magnetic force or the like instead of negative pressure.

In addition, the holding of the workpiece by the hand 14 is not limited to suction, but may also be gripping, etc.

The camera 51 does not have to include a first camera 51A and a second camera 51B. A single camera 51 may capture an image of the inside of the first container 91 and an image of the inside of the second container 92. The camera 51 does not have to be fixedly disposed, but may be attached to the robot arm 12 and moved by the robot arm 12.

The robot system 100 may not be equipped with a camera 51. In that case, an image may be input to the main control device 3 from outside. The method of acquiring the image is not important. The image input from outside may be stored in the memory 33 or in the storage device 32.

The target work to be picked may not be piled up randomly, but may be aligned. The target work is not limited to a rectangular parallelepiped work. For example, the target work may have a shape of a roughly triangular prism. The target work may be a bag filled with powder or granular material such as fertilizer, lime, gravel, etc.

The processing by the operation controller 43, the photography controller 44, and the work detector 45 is merely an example. For example, the work detector 45 can detect a work from an image using various methods.

The processing of the searcher 41 and the determiner 42 is merely an example. For example, the searcher 41 can search for place location candidates using any method. Alternatively, the searcher 41 may receive place location candidates from outside.

The searcher 41 assigns a position candidate to each of the candidates for the relative position and orientation of the hand 14, but is not limited to this. For example, the searcher 41 may assign a candidate for the relative position and orientation of the hand 14 to each of the position candidates.

The searcher 41 may assign multiple candidates for the relative position and orientation of the hand 14 to a single candidate position to create multiple place candidates. Alternatively, the searcher 41 may assign multiple position candidates to a single candidate for the relative position and orientation of the hand 14 to create multiple place candidates.

When creating a place candidate, the searcher 41 determines whether the hand 14 and robot arm 12 can hold the workpiece without interfering with other objects depending on whether a target path can be generated, but is not limited to this. For example, the searcher 41 may only determine whether the hand 14 and robot arm 12 will interfere with other objects when the hand 14 holds the workpiece placed in the position candidate.

When determining the relative position and orientation of the hand 14 with respect to the target workpiece W, the determiner 42 determines whether the hand 14 and robot arm 12 can hold the workpiece without interfering with other objects depending on whether a target path can be generated, but is not limited to this. For example, the determiner 42 may only determine whether the hand 14 and robot arm 12 will interfere with other objects when the target workpiece is held by the hand 14.

When the determiner 42 determines the relative position and orientation of the hand 14, the target workpiece W may already be determined.

The control device does not have to be a single device such as the main control device 3. The control device may include multiple separate devices. For example, the work detector 45, the determiner 42, and the operation controller 43 may each be realized by separate devices.

The flowchart is merely an example. Steps in the flowchart may be changed, replaced, added, omitted, etc. as appropriate. The order of steps in the flowchart may also be changed, and serial processing may be performed in parallel. For example, the placement photography in step S101 in FIG. 9 may be performed consecutively with the picking photography in S103 or the workpiece detection in step S104.

Also, position planning and robot operation may be performed sequentially rather than in parallel.

The functions provided by the components described herein may be implemented in circuitry or processing circuitry, including general purpose processors, application specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and/or combinations thereof, programmed to provide the described functions. A processor includes transistors and other circuits and is considered a circuit or processing circuit. A processor may be a programmable processor that executes a program stored in a memory.

In this specification, a circuit, unit, or means is hardware that is programmed to realize or executes the described functions. The hardware may be any hardware disclosed in this specification or any hardware known to be programmed to realize or execute the described functions.

If the hardware is a processor that is considered to be a type of circuitry, the circuit, means, or unit is a combination of hardware and software used to configure the hardware and/or processor.

The technology disclosed here can be summarized as follows:

[1] The main control device 3 (control device) is a control device that controls the robot 1 that has a robot arm 12 and a hand 14 attached to the robot arm 12 and performs pick-and-place processing of a workpiece, and is equipped with a searcher 41 that searches for multiple candidates for the relative position and orientation of the hand 14 with respect to the workpiece to be placed at a candidate place position, and a determiner 42 that determines the relative position and orientation of the hand 14 with respect to the workpiece W to be picked from multiple candidates for the relative position and orientation of the hand 14 at the time of placement.

With this configuration, first, multiple candidates for the relative position and orientation of the hand 14 corresponding to the candidate placement positions are created, and then the relative position and orientation of the hand 14 with respect to the workpiece W to be picked is determined from the multiple candidates for the relative position and orientation of the hand 14. Since multiple candidates are provided when determining the relative position and orientation of the hand 14 during picking, it becomes easier to achieve appropriate holding of the workpiece during picking. Furthermore, since the multiple candidates at this time correspond to the candidate placement positions, the workpiece picked in the relative position and orientation of the hand 14 determined from the multiple candidates can be appropriately placed in the candidate position. As a result, both picking and placing of the workpiece can be performed appropriately.

[2] In the main control device 3 described in [1], the searcher 41 searches for a plurality of the position candidates and also searches for a plurality of candidates for the relative position and orientation of the hand 14 corresponding to the plurality of the position candidates, and the determiner 42 determines the position candidate corresponding to the candidate for the relative position and orientation of the hand 14 at the time of placement, which is determined as the relative position and orientation of the hand 14 with respect to the target workpiece W, as the placement target position of the workpiece.

With this configuration, the placement target position is also determined by determining the relative position and orientation of the hand 14 during picking. In other words, the relative position and orientation of the hand 14 during picking and the placement target position are determined in relation to each other. As a result, both picking and placing of the work can be performed appropriately.

[3] In the main control device 3 described in [1] or [2], the searcher 41 assigns priorities to multiple position candidates, and the determiner 42 determines whether multiple candidates for the relative position and orientation of the hand 14 at the time of placement are the relative position and orientation of the hand 14 with respect to the target workpiece W, in order of the priority of the corresponding position candidates.

With this configuration, multiple candidates for the relative position and orientation of the hand 14 corresponding to multiple position candidates are created, and priorities are assigned to the multiple position candidates. Then, whether or not the multiple candidates for the relative position and orientation of the hand 14 become the relative position and orientation of the hand 14 with respect to the target workpiece W is determined in the order of priority of the corresponding position candidates. This makes it easier for a candidate for the relative position and orientation of the hand 14 corresponding to a position candidate with a relatively high priority to be determined as the relative position and orientation of the hand 14 at the time of picking. As a result, the placement of the workpiece in a position candidate with a relatively high priority is realized.

[4] In the main control device 3 described in any one of [1] to [3], the determiner 42 determines the target work W from among a plurality of works when determining the relative position and posture of the hand 14 with respect to the target work W to be picked from among a plurality of candidates for the relative position and posture of the hand 14 at the time of placement.

With this configuration, when there are multiple options for the target workpiece, the determiner 42 also determines the target workpiece from among the multiple works when determining the relative position and orientation of the hand 14. Since the options for the target workpiece increase, the range of choices for determining the relative position and orientation of the hand 14 capable of picking up the target workpiece is expanded.

[5] The main control device 3 described in any one of [1] to [4] further includes an operation controller 43 that causes the robot arm 12 and the hand 14 to execute robot operations including a picking operation for picking the target work W and a placing operation for placing the target work W, and the searcher 41 and the determiner 42 search for candidates for the relative position and posture of the hand 14 at the time of placing for the next picking operation and the placing operation, and determine the relative position and posture of the hand 14 with respect to the target work W for picking, in parallel with the robot operation by the operation controller 43.

With this configuration, the robot operation, including the picking operation and the placing operation, and the search for candidates for the relative position and orientation of the hand 14 during placing and the determination of the relative position and orientation of the hand 14 during picking are performed in parallel. As a result, the cycle time of the pick-and-place process can be shortened.

[6] The main control device 3 described in any one of [1] to [5] further includes an imaging controller 44 that causes the first camera 51A to perform picking photography to obtain an image of the work before picking and causes the second camera 51B to perform place photography to obtain an image of the space in which the work is to be placed, the searcher 41 searches for the position candidates and the relative position and posture candidates of the hand 14 based on the image obtained by the place photography, the determiner 42 determines the relative position and posture of the hand 14 with respect to the target work W based on the image obtained by the pick photography, the imaging controller 44 causes the second camera 51B to perform the n-th place photography after the n-2th place operation (n is a natural number of 3 or more) and before the n-1th place operation, and the searcher 41 corrects the place target position of the n-1th place operation based on the image obtained by the nth place photography.

According to this configuration, the searcher 41 searches for position candidates and candidates for the relative position and orientation of the hand 14 based on the image of the place photograph. The determiner 42 determines the relative position and orientation of the hand 14 with respect to the target work W based on the image of the pick photograph. The nth place photograph is performed after the n-2th place operation and before the n-1th place operation. In other words, the n-1th place photograph is performed before the n-2th place operation. The place target position of the n-1th place operation is basically determined based on the image before the n-2th place operation. Therefore, the situation of the space where the placement is scheduled to take place may have changed by the completion of the n-2th place operation. Since the nth place photograph is performed after the n-2th place operation, an image of the space where the placement is scheduled to take place after the completion of the n-2th place operation is obtained. Furthermore, the nth place operation is performed before the start of the n-1th place operation. Therefore, the searcher 41 corrects the placement target position of the (n-1)th placement operation based on the image captured during the nth placement shot. This allows the (n-1)th placement operation to be executed more appropriately.

[7] The robot system 100 includes a robot 1 having a robot arm 12 and a hand 14 attached to the robot arm 12, and a main control device 3 described in any one of [1] to [6].

With this configuration, first, multiple candidates for the relative position and orientation of the hand 14 corresponding to the candidate placement positions are created, and then the relative position and orientation of the hand 14 with respect to the workpiece W to be picked is determined from the multiple candidates for the relative position and orientation of the hand 14. Since multiple candidates are provided when determining the relative position and orientation of the hand 14 during picking, it becomes easier to achieve appropriate holding of the workpiece during picking. Furthermore, since the multiple candidates at this time correspond to the candidate placement positions, the workpiece picked in the relative position and orientation of the hand 14 determined from the multiple candidates can be appropriately placed in the candidate position. As a result, both picking and placing of the workpiece can be performed appropriately.

[8] A control method for a robot 1 having a robot arm 12 and a hand 14 attached to the robot arm 12 and performing pick-and-place processing of a workpiece includes searching for multiple candidates for the relative position and orientation of the hand 14 with respect to a workpiece to be placed at a candidate placement position, and determining the relative position and orientation of the hand 14 with respect to a workpiece W to be picked from multiple candidates for the relative position and orientation of the hand 14 at the time of placement.

With this configuration, first, multiple candidates for the relative position and orientation of the hand 14 corresponding to the candidate placement positions are created, and then the relative position and orientation of the hand 14 with respect to the workpiece W to be picked is determined from the multiple candidates for the relative position and orientation of the hand 14. Since multiple candidates are provided when determining the relative position and orientation of the hand 14 during picking, it becomes easier to achieve appropriate holding of the workpiece during picking. Furthermore, since the multiple candidates at this time correspond to the candidate placement positions, the workpiece picked in the relative position and orientation of the hand 14 determined from the multiple candidates can be appropriately placed in the candidate position. As a result, both picking and placing of the workpiece can be performed appropriately.

100 Robot system 1 Robot 12 Robot arm 14 Hand 3 Main control device (control device)
41 Searcher 42 Deciding device 43 Operation controller 44 Photography controller 51A First camera 51B Second camera W Target work

Claims (8)

A control device for controlling a robot that has a robot arm and a hand attached to the robot arm and performs a pick-and-place process for a workpiece,
a searcher that searches for a plurality of candidates for a relative position and orientation of the hand with respect to a workpiece to be placed at a candidate placement position;
A control device comprising: a determiner that determines the relative position and orientation of the hand with respect to the workpiece to be picked from among a plurality of candidates for the relative position and orientation of the hand at the time of placing. 2. The control device according to claim 1,
the searcher searches a plurality of the position candidates and searches a plurality of candidates for relative positions and orientations of the hand corresponding to the plurality of the position candidates;
The determiner is a control device that determines the position candidate corresponding to the candidate relative position and orientation of the hand at the time of placement, which is determined as the relative position and orientation of the hand with respect to the target workpiece, as the placement target position of the workpiece. 3. The control device according to claim 2,
The searcher prioritizes a plurality of location candidates;
The determiner is a control device that determines whether multiple candidates for the relative position and posture of the hand at the time of placement are the relative position and posture of the hand with respect to the target work in order of the priority of the corresponding position candidates. 2. The control device according to claim 1,
The determiner is a control device that determines the target work from among multiple works when determining the relative position and posture of the hand with respect to the work to be picked from among multiple candidates for the relative position and posture of the hand at the time of placing. 2. The control device according to claim 1,
and an operation controller that causes the robot arm and the hand to execute robot operations including a picking operation for picking the target workpiece and a placing operation for placing the target workpiece,
The searcher and the determiner are control devices that, in parallel with the robot operation by the operation controller, search for candidates for the relative position and posture of the hand at the time of placing for the next picking operation and placing operation, and determine the relative position and posture of the hand with respect to the target work to be picked. 6. The control device according to claim 5,
The system further includes an imaging controller that causes the first camera to perform picking imaging to acquire an image of the work before it is picked, and causes the second camera to perform place imaging to acquire an image of the space in which the work is to be placed,
The searcher searches for the position candidates and the relative position and posture candidates of the hand based on the image captured by the place photographing,
The determiner determines a relative position and orientation of the hand with respect to the target work based on the image obtained by the picking photography,
the shooting controller causes the second camera to execute the n-th place shooting after the n-2-th (n is a natural number equal to or greater than 3) place operation and before the n-1-th place operation;
The searcher is a control device that corrects a place target position of the (n-1)th placing operation based on an image captured by the nth placing operation. a robot having a robot arm and a hand attached to the robot arm;
A robot system comprising the control device according to any one of claims 1 to 6. A method for controlling a robot having a robot arm and a hand attached to the robot arm, the robot arm performing a pick-and-place process for a workpiece, comprising the steps of:
Searching for a plurality of candidates for a position and an orientation of the hand relative to a workpiece to be placed at a candidate placement position;
and determining the relative position and orientation of the hand with respect to the workpiece to be picked from among a plurality of candidates for the relative position and orientation of the hand at the time of placing.

Images