WO2025225572A1 - Control device, control method, and program - Google Patents

Abstract

This control device (100)comprises: a motion generation unit (110) that generates a control command for causing a reverse direction motion for changing the motion state of a robot device (200) from a target state to an arbitrary state different from the target state; an acquisition unit (120) that acquires measurement information obtained by using a sensor (300) for performing measurement pertaining to the motion state of the robot device (200); and a data collection unit (130) that repeatedly collects data including a set of the control command and the measurement information during execution of the reverse direction motion in accordance with the control command. The sensor (300) includes at least one among a visual sensor (310) and a force sensor (320).

Description

Control device, control method, and program CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Japanese Patent Application No. 2024-072901 (filed April 26, 2024), the entire contents of which are incorporated herein by reference.

This disclosure relates to a control device, control method, and program for controlling a robot device.

Control devices for controlling robotic devices have been widely used in the past. A known technology for such control devices is to measure the operating state of the robotic device using a visual sensor and a force sensor, and to control the operating state of the robotic device to achieve a target state (see, for example, Patent Document 1).

In addition, a conventional method for learning the movements of a robotic device is to teach it by manually operating the robotic device and recording the series of movements it performs.

Patent No. 7295344

With conventional technology, any change in the work environment and/or target object requires new teaching. As a result, with conventional technology, it is difficult to efficiently collect data that can be used to control the operation of the robot device.

This disclosure provides a control device, control method, and program that enable more appropriate control of a robotic device.

A control device according to a first aspect of the present disclosure includes a motion generation unit that generates a control command for executing a reverse motion that changes the motion state of a robotic device from a target state to an arbitrary state different from the target state, an acquisition unit that acquires measurement information obtained using a sensor that measures the motion state of the robotic device, and a data collection unit that repeatedly collects data including a set of the control command and the measurement information while the reverse motion is being executed in accordance with the control command. The sensor includes at least one of a visual sensor and a force sensor.

A control method according to a second aspect of the present disclosure includes generating a control command for executing a reverse motion that changes the motion state of a robotic device from a target state to an arbitrary state different from the target state, acquiring measurement information using a sensor that measures the motion state of the robotic device, and repeatedly collecting data including a set of the control command and the measurement information while the reverse motion is being executed in accordance with the control command. The sensor includes at least one of a visual sensor and a force sensor.

A program according to a third aspect of the present disclosure causes a control device to generate a control command for executing a reverse operation that changes the operating state of a robotic device from a target state to an arbitrary state different from the target state, acquire measurement information using a sensor that measures the operating state of the robotic device, and repeatedly collect data including a set of the control command and the measurement information while the reverse operation is being executed in accordance with the control command. The sensor includes at least one of a visual sensor and a force sensor.

1 is a diagram illustrating an example of a system configuration of a control system including a control device according to an embodiment. FIG. 2 is a block diagram showing a functional block configuration of the control device according to the first embodiment. FIG. 3 is a diagram for explaining an example of data collection by the control device according to the first embodiment. FIG. 3 is a diagram for explaining an example of data collection by the control device according to the first embodiment. FIG. 4 is a flow chart for explaining an example of data collection by the control device according to the first embodiment. FIG. 2 is a diagram illustrating an example of a setting library stored in a control device according to the embodiment. FIG. 10 is a block diagram showing a functional block configuration of a control device according to a second embodiment. FIG. 10 is a flowchart illustrating an example of a control flow by a control device according to a second embodiment. 10A and 10B are diagrams for explaining a specific example of control during the operation of "mounting components on a board" as control by the control device according to the second embodiment. FIG. 11 is a block diagram showing a functional block configuration of a control device according to a third embodiment. 11A and 11B are diagrams for explaining a specific example of control during the operation of "mounting components on a board" as control by the control device according to the third embodiment.

Embodiments will be described with reference to the drawings. In the drawings, identical or similar parts are designated by the same or similar reference numerals.

(1) Overview of Embodiments A control device according to an embodiment includes: a motion generating unit that generates a control command for executing a reverse motion that changes the motion state of a robotic device from a target state to an arbitrary state different from the target state; an acquiring unit that acquires measurement information obtained using a sensor that measures the motion state of the robotic device; and a data collecting unit that repeatedly collects data including a set of the control command and the measurement information while the reverse motion is being executed in accordance with the control command. The sensor includes at least one of a visual sensor and a force sensor.

In this way, by repeatedly collecting data including a set of control commands and measurement information while performing reverse movement in accordance with the control commands, it becomes possible to efficiently collect data that can be used to control the movement of the robot device during actual work (i.e., forward movement). For example, during actual work using the robot device, by generating control commands from measurement information obtained during work based on previously collected data, it becomes possible to appropriately control the movement of the robot device from an arbitrary state to a target state.

Note that the term "robot device" refers to any device that can operate in accordance with control commands output by a control device, and any robot device is applicable. For example, the robot device may be an industrial robot such as a manipulator, or may include a mobile body that can move automatically. Industrial robots include, for example, vertical articulated robots, SCARA robots, parallel link robots, Cartesian robots, and collaborative robots. Mobile bodies that can move automatically include, for example, drones, vehicles configured to be self-driving, automated guided vehicles, or mobile robots, as well as combinations with the above industrial robots. In the embodiments described below, an example in which the robot device is a manipulator will be mainly described.

"Control commands" are related to the control of the operation of a robotic device, and are, for example, target control amounts, operation amounts, etc. "Outputting control commands" may mean directly controlling the robotic device based on the control commands, or, if the robotic device is equipped with a controller, outputting control commands to the controller to have the controller control the operation of the robotic device.

The "operating state of a robotic device" refers to the state related to the operation of a portion of the configuration of the robotic device (e.g., an end effector) and/or the state related to the object of work performed using the robotic device. Furthermore, "measurement related to the operating state of a robotic device" refers to the measurement of the state related to the operation of a portion of the configuration of the robotic device (e.g., an end effector) and/or the measurement of the state related to the object of work performed using the robotic device. An "object" is an object that may be related to the operation of the robotic device, such as a workpiece. A portion of the configuration of a robotic device (e.g., an end effector) may also be considered an object.

"Task" refers to a job that is performed by a robotic device and may include multiple processes. Examples of tasks include transporting parts, fitting parts, screwing, and processing. Tasks may also be simple tasks such as gripping and releasing a workpiece. Tasks may be assigned in advance or specified by the operator.

The "target state" is the state at which the purpose of a task (or process) is achieved. The target state can vary depending on the task being performed. For example, if a robotic device is tasked with attaching an assembly part carried by an end effector to a part being assembled, the target state could be a state in which the assembly part has been attached to the part being assembled.

An "arbitrary state" may be any state that is different from the target state, but may also be, for example, the state at the start or midway of a task performed using the robot device.

"Measurement information" is not limited to the sensor measurement data itself, but may also be feature quantities calculated from the measurement data. For example, "measurement information" may be information measured and/or calculated using an encoder and/or servo motor that are present as basic components of the robot.

(2) First Embodiment In the first embodiment, data collection for collecting data to be used for control during actual work before the work will be mainly described. Details of control during actual work will be described in the second and third embodiments.

(2.1) System Configuration Fig. 1 is a diagram showing an example of the system configuration of a control system including a control device 100 according to the first embodiment. Here, the description will be focused on the hardware configuration of the control system.

In the illustrated example, the robot device 200 is a manipulator. Specifically, the robot device 200 (manipulator) is a six-axis vertical articulated industrial robot, and has a base 221 and six joints 211 to 216. Each of the joints 211 to 216 has a built-in servo motor (not shown) and is configured to be rotatable around each axis.

The first joint unit 211 is connected to the base unit 221 and rotates its tip portion around the axis of the base. A moving mechanism capable of automatic movement (self-propelled) may be provided instead of the base unit 221. The second joint unit 212 is connected to the first joint unit 211 and rotates its tip portion back and forth. The third joint unit 213 is connected to the second joint unit 212 via a link 222 and rotates its tip portion up and down. The fourth joint unit 214 is connected to the third joint unit 213 via a link 223 and rotates its tip portion around the axis of the link 223. The fifth joint unit 215 is connected to the fourth joint unit 214 via a link 224 and rotates its tip portion up and down. The sixth joint unit 216 is connected to the fifth joint unit 215 via a link 225 and rotates its tip portion around the axis of the link 225. A force sensor 320 and a gripper 226 are attached to the tip side of the sixth joint 216. The gripper 226 is an example of an end effector.

The visual sensor 310 is a sensor that performs image measurement, and is positioned to observe each object (gripper 226, workpiece W1, workpiece W2) present in the environment (work space) in which the robot device 200 operates. In the example shown, the visual sensor 310 is attached to the link 225, and the visual sensor 310 is provided integrally with the robot device 200. However, the visual sensor 310 may also be fixed to equipment in the work space, and provided separately from the robot device 200. For example, a camera such as a digital camera or video camera may be used as the visual sensor 310. The measurement data (i.e., image data) of the visual sensor 310 is an example of visual measurement information.

The force sensor 320 is a sensor that measures forces and is configured to measure the forces and moments acting on the robot device 200 (specifically, the gripper 226). The force sensor 320 may be, for example, a six-axis force sensor that measures forces acting on the gripper 226 in the three axial directions of the X-axis, Y-axis, and Z-axis, and moments about the X-axis, Y-axis, and Z-axis. In other words, the force sensor 320 can measure forces and moments generated by contact between an assembly part supported by the robot device 200 or the gripper 226 and an object. The measurement data from the force sensor 320 may be used to adjust the gripping force of the gripper 226 or to detect whether an abnormal force is acting on the gripper 226. In the following explanation, the term "force" is used to include the meaning of "moment."

The force sensor 320 may be realized, for example, by measuring the current value of a motor (not shown) built into each of the joints 211 to 216, thereby measuring forces in the three axial directions of the X, Y, and Z axes, and moments around the X, Y, and Z axes. The force sensor 320 may be, for example, a pressure sensor provided on the surface of the robot device 200, and/or a sensor that utilizes changes in the state of a jacket provided on the surface of the robot device 200. The force sensor 320 may also detect changes in the flow rate of air and/or liquid, and/or changes in capacitance.

Note that each of the joints 211 to 216 may have an encoder (not shown) built in. The encoder is an example of a sensor. The encoder is configured to be able to measure the angle (control amount) of each of the joints 211 to 216. The measurement data from the encoder may be used to control the angle of each of the joints 211 to 216.

The control system may have a transport device 510 that transports the workpiece W2. The robot device 200 can perform work using a gripper 226 (end effector) attached to the tip of its arm. The end effector is an external device that can be replaced depending on the application, and a welding gun, tool, etc. may be attached instead of the gripper 226. The robot device 200 can perform work using a force sensor 320 while tracking the workpiece W2 traveling on the transport device 510 using a visual sensor 310. In the illustrated example, the robot device 200 performs work by fitting the workpiece W1, which is an assembly part grasped by the robot device 200, into a hole in the workpiece W2 (a part to be assembled, such as a circuit board) traveling on the transport device 510.

The control device 100 has a processor 101, memory 102, and an external interface (I/F) 103. The processor 101 is configured to include a CPU (Central Processing Unit). Furthermore, the processor 101 may include at least one of a microprocessor, an FPGA (field-programmable gate array), and a DSP (digital signal processor). The memory 102 is configured to include RAM (Random Access Memory), ROM (Read Only Memory), and an auxiliary storage device (e.g., a hard disk drive, solid state drive). The processor 101 and memory 102 constitute a computer. The control device 100 may be configured from multiple computers. The control device 100 is not limited to an information processing device designed specifically for the service provided, but may also be a general-purpose information processing device such as a PC (Personal Computer), or a controller such as a PLC (Programmable Logic Controller).

Memory 102 stores programs executed by processor 101. By executing the programs stored in memory 102, processor 101, together with memory 102, realizes the functions of each functional block described below. As will be described in detail below, memory 102 may store, for example, a recognition library containing trained models used for image recognition to recognize objects, and a setting library containing data collected by pre-task data collection and setting information obtained based on the collected data. The recognition library and setting library may be provided for each type of task that the robot device 200 can perform. The setting information included in the setting library may include trained models generated based on the collected data.

The external I/F 103 is, for example, a USB (Universal Serial Bus) port or a dedicated port, and is an interface for connecting to an external device so as to be able to communicate with the external device. The external I/F 103 may be connected to the external device (including the robot device 200) via a wired or wireless connection. The type and number of external I/Fs 103 may be selected appropriately depending on the type and number of external devices to be connected. In the illustrated example, the control device 100 is connected to the robot device 200, a visual sensor 310, and a user interface (I/F) 400 via the external I/F 103. As shown in FIG. 2, the user I/F 400 includes a display device 410 and an operation device 420. In the illustrated example, the user I/F 400 is provided separately from the control device 100, but the user I/F 400 may also be provided integrally with the control device 100. The display device 410 may be a liquid crystal display or an organic EL (Electro-Luminescence) display. The display device 410 may also be a display equipped with a speaker. The operation device 420 is a device for inputting operations, such as a keyboard, mouse, or touch panel. The display device 410 and operation device 420 may be integrated into a touch panel display. The operator can use the display device 410 and operation device 420 to check the status of the control device 100 and operate the control device 100.

(2.2) Functional Block Configuration of Control Device FIG. 2 is a block diagram showing the functional block configuration of the control device 100 according to the first embodiment. In the first embodiment, the functional block configuration related to data collection, which collects data to be used for control during work before the work is performed, will be mainly described. Such data collection may be performed before a user (including an operator) starts using the control device 100 (and the robot device 200). For example, data collection may be performed in advance before the control device 100 (and the robot device 200) is shipped.

The control device 100 has an action generation unit 110, an acquisition unit 120, a data collection unit 130, a data storage unit 140, a setting acquisition unit 150, and a library storage unit 160. In this embodiment, the control device 100 may or may not have a control unit 170.

The motion generation unit 110 generates a control command for executing a forward motion that changes the motion state of the robot device 200 from an arbitrary state to a target state. The motion generation unit 110 also generates a control command for executing a reverse motion that changes the motion state of the robot device 200 from a target state to an arbitrary state different from the target state. The motion generation unit 110 outputs the generated control command to the drive unit 210 of the robot device 200. The drive unit 210 includes servo motors provided in each of the joints 211 to 216 of the robot device 200. The drive unit 210 may include a controller on the robot device 200 side. The drive unit 210 drives the servo motors in accordance with the control command to operate the robot device 200.

Here, the target state refers to the state that is achieved when the purpose of the work (or process) is achieved, and/or an intermediate state of the work. In the first embodiment, a series of operations is assumed in which the robot device 200 attaches an assembly part (work W1) carried by the end effector (gripper 226) to a part to be assembled (work W2), and therefore the target state is a state in which the assembly part has been attached to the part to be assembled, and/or an intermediate state before attachment. Specifically, the target state is a state in which the work W1 has been fitted into a hole in the work W2, and/or an intermediate state in which the work W1 is in contact with the work W2 at a position where it can be fitted into the hole.

The arbitrary state is a state different from the target state, for example, a state in which workpiece W1 is located away from the hole in workpiece W2. The arbitrary state may be set by operation input (user input) via the operation device 420. For example, the position of the arbitrary state may be set by operation input (user input) using the position of the target state as a reference. Note that the term "position" may not only mean "coordinates" but may also mean "posture."

The acquisition unit 120 acquires measurement information obtained using a sensor 300 for measuring the operating state of the robot device 200. In the illustrated example, the sensor 300 includes a visual sensor 310 and a force sensor 320. The sensor 300 may further include other sensors 330 such as an encoder. The acquisition unit 120 may also include an image recognition unit 121 that performs image recognition on the measurement data (i.e., image data) output by the visual sensor 310. The image recognition unit 121 may recognize objects (e.g., gripper 226, workpiece W1, workpiece W2) through image recognition such as feature extraction, and acquire the position of the objects (e.g., coordinates of characteristic features). Such position information is an example of visual measurement information.

The data collection unit 130 repeatedly collects data including a set of a control command output by the motion generation unit 110 and measurement information acquired by the acquisition unit 120 while performing a reverse motion in accordance with the control command. This makes it possible to efficiently collect data that can be used to control the motion (forward motion) of the robot device 200 during actual work. For example, during actual work using the robot device 200, it becomes possible to appropriately control the motion of the robot device from an arbitrary state to a target state by generating a control command from measurement information obtained during work based on already collected data. Note that the data collection unit 130 may repeatedly collect data including a set of a control command and measurement information acquired by the acquisition unit 120 while performing a forward motion in accordance with the control command output by the motion generation unit 110.

The motion generation unit 110 may generate control commands for executing multiple patterns of reverse motion that change the motion state of the robot device 200 from a target state to multiple arbitrary states that are different from one another. The data collection unit 130 may repeatedly collect data for each of the multiple patterns of reverse motion. This makes it possible to collect multiple patterns of data corresponding to multiple movement paths, thereby providing versatility to the motion control of the robot device 200 during actual work. For example, even if the workpiece W2 is not fixed or moves during work, it becomes easy to attach the workpiece W1 to the workpiece W2.

The action generation unit 110 may generate control commands that collect more data in areas closer to the target state, and generate control commands that collect less data in areas farther from the target state. The closer an area is to the target state, the more precise and accurate control is required during actual work. By increasing the amount of data collected in areas closer to the target state, it becomes possible to collect a sufficient amount of data to perform precise and accurate control. On the other hand, in areas farther from the target state, precise and accurate control is not as necessary. Therefore, by generating control commands that collect less data in areas farther from the target state, unnecessary data collection can be suppressed, enabling efficient data collection.

The acquisition unit 120 may shorten the acquisition cycle (e.g., sampling frequency) so that the amount of measurement information acquired increases in areas closer to the target state, and may extend the acquisition cycle so that the amount of measurement information acquired increases in areas farther from the target state. This type of processing also allows the amount of data collected to increase in areas closer to the target state, and decrease in areas farther from the target state.

The acquisition unit 120 may acquire, as relative measurement information, the difference between target measurement information obtained in the target state and current measurement information obtained during the execution of the reverse operation. The data collection unit 130 may repeatedly collect data including a set of the control command and relative measurement information while the reverse operation is being executed in accordance with the control command. For example, the acquisition unit 120 may acquire, as relative position information, the difference between the visual measurement information (target position) of the object obtained by the visual sensor 310 in the target state and the current measurement information (current position) of the object obtained during the execution of the reverse operation. In the environment of Figure 1, the acquisition unit 120 may set the position of the hole in the workpiece W2 as the target position, and each position on the movement path of the workpiece W1 as the current position, and acquire the difference between the target position and each current position as the relative position. This allows a control command to be associated with each relative positional relationship between the target position and the current position of the object. Therefore, during actual work, appropriate control commands can be generated based on the relative positional relationship between the target position and the current position of the object. Furthermore, control using relative positional relationships can be applied even when the object moves during work.

The acquisition unit 120 may acquire measurement information including a velocity-related value related to the robot device 200 in response to the output of the sensor 300 or a control command. The velocity-related value related to the robot device 200 may be at least one of the velocity, acceleration, and jerk of the object. The acquisition unit 120 may derive the velocity, acceleration, and jerk of the object from visual measurement information (position information of the object) obtained by the visual sensor 310. The acquisition unit 120 may derive the velocity, acceleration, and jerk of the object from a control command output to the drive unit 210. The acquisition unit 120 may derive the velocity, acceleration, and jerk of the object from measurement information obtained by another sensor 330 (e.g., an encoder). The data collection unit 130 may repeatedly collect data including a set of the control command and measurement information including the velocity-related value while performing a reverse movement in accordance with the control command. The velocity-related value collected in this manner can be used to set a velocity-related target value in control during actual work. Control using the velocity-related target value will be described in the third embodiment.

The data storage unit 140 stores the data collected by the data collection unit 130. The data collected by the data collection unit 130 includes multiple sets of measurement information (relative measurement information) and control commands. Each set may include measurement information (relative measurement information), a speed-related value, and a control command. The data collected by the data collection unit 130 may include more data sets the closer the area is to the target state.

The setting acquisition unit 150 acquires setting information for performing work using the robot device 200 based on data collected by the data collection unit 130 (specifically, data stored in the data storage unit 140). Setting information for performing a certain work is referred to as the "setting library" for that work. For example, the setting acquisition unit 150 acquires the setting library after data collection by the data collection unit 130 is complete. The library storage unit 160 stores the setting library acquired by the setting acquisition unit 150. The control unit 170 controls the robot device 200 using the setting library during actual work.

The setting library includes operation parameters for each work process, and information on the transition destination and transition conditions for each process (normal values, timeout values, error values, etc. of the transition conditions). The operation parameters may include correspondence information that associates measurement information (relative measurement information) with control commands. The operation parameters may include correspondence information that associates measurement information (relative measurement information), speed-related values, and control commands. The operation parameters may include a control ratio between visual control, which is robot control based on visual measurement information obtained by the visual sensor 310, and force control, which is robot control based on force measurement information obtained by the force sensor 320. The operation parameters may include information on the type of object (metal, resin, screw, connector, etc.). The operation parameters may include information on features on the object (holes, edge surfaces, connectors, etc.). The setting acquisition unit 150 may acquire the setting library based at least in part on operation input performed via the operation device 420.

The setting acquisition unit 150 may use the data collected by the data collection unit 130 as learning data to acquire a trained model for deriving control commands from measurement information through machine learning. In this case, the control unit 170 uses a setting library including the trained model during actual work to control the robot device 200 based on measurement information obtained during work. The type of trained model (trained model) is not particularly limited as long as it is capable of acquiring the ability to make inferences for generating control commands through machine learning. The type of machine learning is not particularly limited, but is typically supervised learning or reinforcement learning. The training model may be configured, for example, by a neural network such as a deep neural network (DNN). The training model may be configured, for example, by a value function such as a state value function or an action value function. In this way, the setting acquisition unit 150 uses machine learning to determine calculations for inferring optimal values, enable inferences, and generate optimal value variables for control commands.

The setting acquisition unit 150 may acquire a setting library that includes the control ratio between visual control and force sense control, for example, for each process. In this case, the control unit 170 uses the setting library during actual work to dynamically or gradually change the control ratio depending on the status of the work. Details of this type of control will be explained in the second embodiment.

The setting acquisition unit 150 may acquire a setting library containing speed-related values related to the robot device 200, for example, for each process. In this case, the control unit 170 may derive the speed-related values related to the robot device 200 from measurement information or control commands obtained during work. The control unit 170 may then repeatedly control the robot device 200 to reduce the difference between the speed-related value and the speed-related target value, based on the setting library and the measurement information obtained during work. Details of such control will be described in the third embodiment.

The motion generation unit 110 may generate a control command for executing a reverse motion for each type of task performed by the robot device 200. Assuming the environment of Figure 1, the task type is "mounting components on a circuit board." Other examples of task types include "packing food into boxes," "screw tightening," "pick and place," and "AGV (Automated Guided Vehicle) control." The data collection unit 130 may collect data for each task type by repeatedly collecting data for each task type.

In this case, the setting acquisition unit 150 can acquire a setting library for each type of work based on the data collected by the data collection unit 130 for each type of work. The control unit 170 then controls the robot device 200 to perform the work using the setting library corresponding to the type of work that is actually being performed. For example, the control unit 170 may display a list of setting libraries stored in the library storage unit 160 on the display device 410, and control the robot device 200 using a setting library selected from the list using the operation device 420. Note that if the library storage unit 160 also stores a recognition library associated with the setting library, image recognition for performing the work may be performed using the recognition library corresponding to the type of work that is actually being performed.

(2.3) Example of Data Collection FIGS. 3 and 4 are diagrams for explaining an example of data collection by the control device 100 according to the first embodiment.

As described above, the motion generation unit 110 generates control commands for executing multiple patterns of reverse motion that change the motion state of the robot device 200 from a target state to multiple mutually different arbitrary states. The data collection unit 130 repeatedly collects data for each of the multiple patterns of reverse motion. As shown in FIG. 3, the motion generation unit 110 generates control commands that collect more data in areas closer to the target state, and generates control commands that collect less data in areas farther from the target state. In FIG. 3, the x-axis and y-axis indicate directions that are perpendicular to each other in the horizontal plane, and the z-axis indicates the vertical direction. Also, in FIG. 3, the amount of collected data is indicated by shading. Specifically, the greater the amount of collected data, the darker the color, and the less the amount of collected data, the lighter the color.

In the example of Figure 4, the motion generation unit 110 sets a state in which the workpiece W1 (component) is in the hole in the workpiece W2 (circuit board) as the target state, and causes the robot device 200 to perform reverse motion to any of eight patterns P1 to P8. While the movement path for each pattern P1 to P8 is illustrated as linear, each movement path does not have to be linear. For example, the movement path may be such that the workpiece W1 (component) is pulled upward from the hole in the workpiece W2 (circuit board), and then moved horizontally or diagonally upward. Furthermore, of the patterns P1 to P8, the movement of P2, P4, and P8 ends in region R, which is close to the target state. As a result, the closer the point is to the position of the target state, the greater the amount of data collected.

(2.4) Specific Example of Operation of the Control Device FIG. 5 is a flow chart for explaining an example of data collection by the control device 100 according to the first embodiment.

In step S101, a user (e.g., a worker) moves an object (e.g., a workpiece W1 grasped or supported by an end effector) to a target state. The motion generation unit 110 also acquires control command sequence information corresponding to one of a plurality of reverse motion patterns (i.e., any arbitrary state) and/or position information for that arbitrary state. Here, the acquisition unit 120 acquires the position of the object in the target state as the target position from visual measurement information obtained using the visual sensor 310. The acquisition unit 120 acquires the force (reaction force) applied to the object in the target state as the target force (target reaction force) from force measurement information obtained using the force sensor 320.

In step S102, the motion generation unit 110 outputs a control command to the robot device 200 (drive unit 210) based on the information acquired in step S101, thereby moving the target object a predetermined distance.

In step S103, the acquisition unit 120 acquires current measurement information. The acquisition unit 120 acquires the current position of the object from visual measurement information obtained using the visual sensor 310. The acquisition unit 120 may acquire the difference between the current position of the object and the target position as a relative position (amount of position change). The acquisition unit 120 also acquires the current force (reaction force) from force measurement information obtained using the force sensor 320. The acquisition unit 120 may acquire the difference between the current force (reaction force) and the target force (reaction force) as a relative force (amount of force change). Furthermore, the acquisition unit 120 may acquire velocity-related values (velocity, acceleration, and jerk) of the object from the current visual measurement information or the control command of step S102.

In step S104, the data collection unit 130 collects a set of the control command from step S102 and the information acquired in step S103 (position change amount, force change amount, velocity-related value), and stores the set in the data storage unit 140.

If the movement of the object to the arbitrary state corresponding to the current reverse movement pattern has not been completed (step S105: NO), the process returns to step S102, and the movement generation unit 110 outputs a control command to the robot device 200 (drive unit 210) to move the object a predetermined distance. Then, in step S103, the acquisition unit 120 acquires information (position change amount, force change amount, velocity-related value). In step S104, the data collection unit 130 collects a set of the control command from step S102 and the information acquired in step S103 (position change amount, force change amount, velocity-related value), and stores them in the data storage unit 140. This process is repeated until the movement to the arbitrary state corresponding to the current reverse movement pattern is completed.

On the other hand, if the movement of the object to an arbitrary state corresponding to the current reverse movement pattern is complete (step S105: YES), the movement generation unit 110 checks whether the movements have been completed for all of the multiple reverse movement patterns. If the movements have not been completed for all patterns (step S106: NO), the process moves to the next reverse movement pattern (step S107) and resumes processing from step S101.

If processing has been completed for all patterns (step S106: YES), in step S108, the setting acquisition unit 150 determines the operating conditions (operating parameters) for performing work using the robot device 200 based on the data collected by the data collection unit 130 (specifically, the data stored in the data storage unit 140). Then, in step S109, the setting acquisition unit 150 stores setting information including the operating conditions (operating parameters) determined in step S108 in the library storage unit 160 as a setting library.

In addition, by executing the flow shown in Figure 5 for each type of work, a setting library for each type of work may be stored in the library storage unit 160.

(2.5) Example of Setting Library FIG. 6 is a diagram for explaining an example of a setting library stored in the control device 100 according to the embodiment.

In the example shown, the types of work include "mounting components on a circuit board," "packing food into boxes," "screw tightening," "pick and place" (also known as "picking"), and "AGV control." Each work involves multiple processes.

For example, in a task type called "mounting components on a board," the steps are performed in the following order: Step 1 "Recognize component → Move" → Step 2 "Grab component" → Step 3 "Move component toward board" → Step 4 "Approach" → Step 5 "Recognize holes on board" → Step 6 "Approach hole" → Step 7 "Hole tracing operation" → Step 8 "Insert hole" → Step 9 "Hole insertion completed" → Step 10 "Release grip." On the other hand, when collecting data, the control device 100 may control the operation of the robot device 200 in the reverse order. Note that in the case of "AGV control," an obstacle avoidance step may be further included. The obstacle avoidance step may be included in Step 4 "Move object toward target object."

The setting library includes operation parameters for each work process, and information on the transition destination and transition conditions for each process (normal values, timeout values, error values, etc. for transition conditions). The operation parameters may include correspondence information that associates measurement information (relative measurement information) with control commands, or may include correspondence information that associates measurement information (relative measurement information), speed-related values, and control commands. The operation parameters may include the control ratio between visual control and force control. The operation parameters may include information on the type of object (metal, resin, screws, connectors, etc.). The operation parameters may include information on features on the object (holes, edges, connectors, etc.).

(3) Second Embodiment The second embodiment will be described mainly focusing on the differences from the first embodiment. The system configuration of the second embodiment is the same as that of the first embodiment (see FIG. 1). Note that the second embodiment will be described mainly as an example in which the second embodiment is based on the first embodiment, but the second embodiment does not necessarily have to be based on at least a part of the first embodiment.

(3.1) Functional Block Configuration of Control Device FIG. 7 is a block diagram showing the functional block configuration of the control device 100 according to the second embodiment.

The control device 100 according to the second embodiment includes an acquisition unit 120 that acquires measurement information obtained using a sensor 300 for measuring the operating state of the robot device 200, and a control unit 170 that controls the operation of the robot device 200 based on the measurement information. The sensor 300 includes a visual sensor 310 and a force sensor 320. The control unit 170 dynamically or stepwise changes the control ratio between visual control, which is control based on visual measurement information obtained by the visual sensor 310, and force control, which is control based on force measurement information obtained by the force sensor 320, depending on the status of the work using the robot device 200. This makes it possible to take advantage of the benefits of both visual control and force control, and appropriately control the operation of the robot device 200 by using both visual control and force control.

In the second embodiment, the control unit 170 includes a visual control unit 171A that references visual measurement information to generate first information indicating the control content of the robot device 200, a force-sense control unit 171B that references force-sense measurement information to generate second information indicating the control content of the robot device 200, a command generation unit 172 that generates control commands for the robot device 200 based on the first information and the second information, and a weighting unit 173 that changes the control ratio by performing weighting processing between the visual measurement information and the force-sense measurement information, or between the first information and the second information, depending on the status of the work using the robot device 200. This makes it possible to appropriately change the control ratio through weighting processing.

The visual control performed by the visual control unit 171A may be performed based on, for example, operational parameters (such as correspondence information) in the setting library. The visual control unit 171A identifies the relative positional relationship between the current position and the target position of the object based on visual measurement information obtained using the visual sensor 310. The visual control unit 171A may then generate a visual control command based on the identified positional relationship using the correspondence information to reduce the difference between the current position and the target position (i.e., move the object closer to the target position), and output this visual control command as the first information. At least a portion of this visual control may be performed using a trained model.

Similarly, the force sense control performed by the force sense control unit 171B may be performed based on, for example, operation parameters (e.g., correspondence information) in a setting library. The force sense control unit 171B may identify the difference between the current force (current reaction force) of the object and the target force (target reaction force) based on force sense measurement information obtained using the force sensor 320, and may generate a force sense control command from the identified difference to reduce the difference using correspondence information, and output this force sense control command as second information. At least a portion of this force sense control may be performed using a trained model.

The control unit 170 (weighting unit 173) may dynamically or gradually change the control ratio as the work using the robotic device 200 progresses. As described above, the work using the robotic device 200 may include multiple predetermined steps. The control unit 170 may change the control ratio for each step. This allows the operation of the robotic device 200 to be controlled with an appropriate control ratio for each step.

The library storage unit 160 may store a setting library (setting information) that includes settings related to the control ratios for each of multiple processes. The control unit 170 (weighting unit 173) may change the control ratio for each process based on the setting library. This allows an appropriate control ratio to be set for each process.

When transitioning from one process to the next in a task, the control unit 170 (weighting unit 173) gradually changes the control ratio from the control ratio for that process to the control ratio for the next process. This prevents sudden fluctuations in the control ratio, enabling precise operation control and reducing the occurrence of operation errors.

For example, if the control ratio (visual control: force control) for process A is set to "80:20" and the control ratio for process B, which follows process A, is set to "50:50," when the transition condition from process A to process B is satisfied, the control unit 170 (weighting unit 173) changes the control ratio in stages, such as "80:20" → "75:25" → "70:30" → "65:35" → "60:40" → "55:45" → "50:50." Alternatively, the control unit 170 (weighting unit 173) may change the control ratio continuously, such as "80:20" → "79:21" → "78:22" → "77:23" → ... Such changes in the control ratio may be made in control cycle units, or in time units consisting of multiple control cycles.

The control unit 170 (weighting unit 173) may determine whether a transition condition from one process to the next process in a task is satisfied based on measurement information obtained using the sensor 300. The transition condition may be included as one of the operation parameters in the setting library. When the transition condition is satisfied, the control unit 170 (weighting unit 173) may transition from the one process to the next process and change the control ratio corresponding to the next process.

If the control in the next process does not converge after switching to the next process, the control unit 170 may return to the previous process and change the control ratio corresponding to that process. In this way, if the control does not converge, by returning to the previous process and restarting the operation, it is possible to expect that the control will converge when transitioning to the next process. Note that "if the control does not converge" may mean that at least one of the following conditions is met: the measurement information does not satisfy the normal value of the transition condition, a timeout has occurred, or the measurement information has become an error value.

The control unit 170 (weighting unit 173) may switch the control of the robot device 200 between a first control state (also referred to as "visual-based") in which control is performed with priority given to visual control, a second control state (also referred to as "visual+force-based") in which control is performed using visual control and force-sense control in cooperation, and a third control state (also referred to as "force-sense-based") in which control is performed with priority given to force-sense control. "Control is performed with priority given to visual control" may mean, for example, that the control ratio accounted for by visual control is approximately 65% to 100%. "Control is performed using visual control and force-sense control in cooperation" may mean, for example, that the visual control:force-sense control ratio is approximately 50:50. "Control is performed with priority given to force-sense control" may mean, for example, that the control ratio accounted for by force-sense control is approximately 65% to 100%. The control unit 170 may determine whether to switch control between the first control state, the second control state, and the third control state based on measurement information obtained using the sensor 300. If, in the first control state, the difference between the measurement information and the target value for the measurement information does not become equal to or less than a predetermined value (this may be the case when the transition condition is not satisfied), the control unit 170 (weighting unit 173) may switch to the second control state or the third control state and refer to the force measurement information.

When an external instruction is received, the control unit 170 (weighting unit 173) may change the control ratio in accordance with the instruction. For example, when an instruction to change the control ratio is received via the operation device 420, the control unit 170 (weighting unit 173) may change the control ratio in accordance with the instruction.

The control unit 170 may further include a prediction unit 174A that predicts the control results for subsequent control operations performed on the robotic device 200 based on the control operations performed on the robotic device 200 and the control results (sensor information) for those control operations, and a correction unit 174B that corrects the subsequent control operations based on the prediction. For example, if the robotic device 200 does not perform the operation specified in the control command due to external factors such as an error factor on the robotic device 200 side or an error factor on the sensor 300 side, the prediction unit 174A identifies the error (i.e., the difference between the control command content and the control result) and predicts the control results for the subsequent control operations. The correction unit 174B may then correct the control command generated by the command generation unit 172 based on the prediction result of the prediction unit 174A and output it to the drive unit 210. For example, the correction unit 174B may correct the control command to cancel out the identified error. This enables more accurate robot control even when there are external error factors.

The library storage unit 160 may store multiple setting libraries prepared for each type of work. Each of the multiple setting libraries may include setting information (operation parameters) for control ratios. Each of the multiple setting libraries includes setting information for each series of processes. The control unit 170 controls the robot device 200 using a setting library selected from the multiple setting libraries according to the type of work actually to be performed. The selection may be made by operation input via the operation device 420. Prior to such operation input, the acquisition unit 120 (image recognition unit 121) may estimate the actual work to be performed based on visual measurement information obtained by the visual sensor 310 observing the work site or object. The control unit 170 may suggest the setting library corresponding to the estimated work to the user (operator) by displaying it on the display device 410.

The library storage unit 160 may further store multiple recognition libraries prepared for each type of work. Each of the multiple recognition libraries may contain a trained model used for image recognition processing of an object. The control unit 170 performs image recognition processing using a recognition library selected from the multiple recognition libraries according to the type of work actually to be performed. The selection may be made by operation input via the operation device 420. Prior to such operation input, the acquisition unit 120 (image recognition unit 121) may estimate the actual work to be performed based on visual measurement information obtained by the visual sensor 310 observing the work site or the object. The control unit 170 may suggest the recognition library corresponding to the estimated work to the user (operator) by displaying it on the display device 410.

(3.2) Operation Flow of the Control Device FIG. 8 is a flow diagram for explaining an example of a control flow by the control device 100 according to the second embodiment.

In step S201, the control unit 170 starts process n. At the start of work, the value of n is "1." At that time, the control unit 170 applies the setting information (operation parameters including control ratios) corresponding to process n.

In step S202, the control unit 170 applies the setting information (operation parameters including control ratios) corresponding to process n to control the operation of the robot device 200.

In step S203, the control unit 170 determines whether the transition condition from process n to the next process is satisfied (i.e., whether process n has been completed). If process n has been completed (step S203: YES), processing proceeds to step S204. On the other hand, if process n has not been completed (step S203: NO), in step S205, the control unit 170 determines whether the control of process n has converged. If it is determined that the control of process n has converged (step S205: YES), processing returns to step S202.

If it is determined that the control of process n has not converged (step S205: NO), the control unit 170 returns to the process (n-1) prior to process n. At that time, the control unit 170 may gradually change the control ratio from the control ratio of process n to the control ratio of process (n-1). Alternatively, instead of returning to process (n-1), the control unit 170 may change at least some of the operating parameters of process n in accordance with a predetermined rule and then resume process n. For example, if it is estimated that the speed-related target value is too high, the speed-related target value may be lowered by one level and process n may be resumed.

In step S204, the control unit 170 determines whether all steps of the work have been completed. If all steps have been completed, this flow ends. If all steps have not been completed (step S204: NO), in step S207, the control unit 170 proceeds to step (n+1), which is the step after step n. At this time, the control unit 170 may gradually change the control ratio from the control ratio for step n to the control ratio for step (n+1).

(3.3) Specific Example of Operation of the Control Device FIG. 9 is a diagram for explaining a specific example of control performed by the control device 100 according to the second embodiment during the operation of "mounting components on a board."

In the task type "mounting components on a board," the process is performed in the following order: Process 1 "Recognize component → Move" → Process 2 "Grab component" → Process 3 "Move component toward board" → Process 4 "Approach" → Process 5 "Recognize board holes" → Process 6 "Approach hole" → Process 7 "Hole tracing operation" → Process 8 "Insert hole" → Process 9 "Hole insertion completed" → Process 10 "Release gripping," and the operation parameters for each of these processes are included in the settings library.

Of these processes, basically, force-based control is set for processes involving contact between objects (end effector (gripper 226), component (workpiece W1), substrate (workpiece W2)), visual-based control is set for processes in which objects (end effector, component) are moved at a certain speed or higher, and visual and force-based control is set for processes in which objects are moved at a low speed toward contact between them. However, it is preferable that the specific control ratio for each process be set based on data collected in the data collection according to the first embodiment.

In the example shown, vision-based control is applied to process 1, "Part recognition → movement." However, if the position of the part recognized by image recognition is set as the target position, vision + force-based control may be applied when the current position of the end effector approaches the vicinity of the target position. Force-based control is applied to process 2, "Part grip."

Vision-based control is applied to process 3, "Moving the component toward the board." Vision-based control is applied to process 4, "Approach." However, if the position of the board recognized by image recognition is set as the target position, vision and force-based control may be applied when the current position of the component approaches the vicinity of the target position. Vision-based control is applied to process 5, "Recognizing holes in the board."

In step 6, "Approaching the hole," vision and force-based control is applied. From step 7, "Following the hole," to step 10, "Releasing the grip," force-based control is applied.

(4) Third Embodiment The third embodiment will be described mainly focusing on the differences from the first and second embodiments. The system configuration of the third embodiment is the same as that of the first embodiment (see FIG. 1). Note that the third embodiment will be described mainly as an example in which the third embodiment is based on the first and second embodiments, but the third embodiment does not necessarily have to be based on at least part of the first and second embodiments.

(4.1) Functional Block Configuration of Control Device FIG. 10 is a block diagram showing the functional block configuration of a control device 100 according to the third embodiment.

The control device 100 according to the third embodiment includes an acquisition unit 120 that acquires measurement information obtained using a sensor 300 for measuring the operating state of the robot device 200, and a control unit 170 that repeatedly generates control commands for the robot device 200 for each control period based on the measurement information. The control unit 170 derives a velocity-related value for the robot device 200 from the measurement information or the control command, and generates a control command to reduce the difference between the velocity-related value and a velocity-related target value that is variable for each control period depending on the operating state of the robot device 200. This enables control that prevents operational errors when working with the robot device 200. Furthermore, by varying the velocity-related target value for each control period depending on the operating state (work situation) of the robot device 200, it becomes possible to handle precise work involving nonlinear movements. As described above, the velocity-related value includes at least one of velocity, acceleration, and jerk.

In the third embodiment, the control unit 170 has a derivation unit 175 that derives a speed-related value from measurement information or a control command for each control cycle, a target setting unit 176 that variably sets a speed-related target value for each control cycle according to the operating state (work situation) of the robot device 200, and a command generation unit 172 that generates a control command for each control cycle so as to reduce the difference between the speed-related value and the speed-related target value. The target setting unit 176 may variably set the speed-related target value according to a speed-related value included in an operating parameter in the setting library.

The command generation unit 172 may incorporate the functions of the visual control unit 171A, the haptic control unit 171B, and the weighting unit 173 described in the second embodiment. Alternatively, the control unit 170 may be provided with the visual control unit 171A, the haptic control unit 171B, and the weighting unit 173 described in the second embodiment, separate from the command generation unit 172.

The control unit 170 (command generation unit 172) generates a control command to reduce the difference between the measurement information and the target measurement information (target value of the measurement information), and to reduce the difference between the speed-related value and the speed-related target value.

For example, the control unit 170 (command generation unit 172) may generate a control command to reduce the difference between the visual measurement information (current position) obtained using the visual sensor 310 and the target visual measurement information (target position), and to reduce the difference between the speed-related value and the speed-related target value. Here, the control to reduce the difference between the visual measurement information (current position) and the target visual measurement information (target position) is the same as the visual control described in the second embodiment.

The control unit 170 (command generation unit 172) may generate a control command to reduce the difference between the force measurement information (current force (reaction force)) obtained using the force sensor 320 and the target force measurement information (target force (reaction force)), and to reduce the difference between the velocity-related value and the velocity-related target value. Here, the control to reduce the difference between the force measurement information (current force (reaction force)) and the target force measurement information (target force (reaction force)) is the same as the force control described in the second embodiment.

The control unit 170 (target setting unit 176) may change the speed-related target value according to the difference between the current measurement information and the target measurement information (current relative measurement information). For example, the control unit 170 (target setting unit 176) may use association information included in the setting library that associates measurement information (relative measurement information) with speed-related values to set the speed-related value that corresponds to the current relative measurement information as the speed-related target value.

For example, the control unit 170 (target setting unit 176) may change the speed-related target value in accordance with the difference between the current visual measurement information obtained using the visual sensor 310 and the target visual measurement information. The control unit 170 (target setting unit 176) may change the speed-related target value in accordance with the difference between the current force measurement information obtained using the force sensor 320 and the target force measurement information.

The control unit 170 (derivation unit 175 and target setting unit 176) may select one or more values from speed, acceleration, and jerk to be used as the speed-related value (and speed-related target value) depending on the difference between the measurement information and the target measurement information (current relative measurement information). For example, the control unit 170 (derivation unit 175 and target setting unit 176) may select speed as the speed-related value (and speed-related target value) during visual-based control. On the other hand, during force-based control, the control unit 170 (derivation unit 175 and target setting unit 176) may select jerk as the speed-related value (and speed-related target value). During visual-and force-based control, the control unit 170 (derivation unit 175 and target setting unit 176) may select acceleration as the speed-related value (and speed-related target value).

Similar to the second embodiment, the control unit 170 may have a prediction unit 174A that predicts the control results of subsequent control operations on the robot device 200 based on the control operations performed on the robot device 200 and the control results of those control operations, and a correction unit 174B that corrects the subsequent control operations based on the prediction.

The library storage unit 160 may store multiple setting libraries prepared for each type of work. Each of the multiple setting libraries may include setting information for speed-related values (speed-related target values). The control unit 170 controls the robot device 200 using a setting library selected from the multiple setting libraries according to the type of work actually to be performed. The selection may be made by operation input via the operation device 420. Prior to such operation input, the acquisition unit 120 (image recognition unit 121) may estimate the work to be actually performed based on visual measurement information obtained by the visual sensor 310 observing the work site or object. The control unit 170 may suggest the setting library corresponding to the estimated work to the user (operator) by displaying it on the display device 410.

The library storage unit 160 may further store multiple recognition libraries prepared for each type of work. Each of the multiple recognition libraries may contain a trained model used for image recognition processing of an object. The control unit 170 performs image recognition processing using a recognition library selected from the multiple recognition libraries according to the type of work actually to be performed. The selection may be made by operation input via the operation device 420. Prior to such operation input, the acquisition unit 120 (image recognition unit 121) may estimate the actual work to be performed based on visual measurement information obtained by the visual sensor 310 observing the work site or the object. The control unit 170 may suggest the recognition library corresponding to the estimated work to the user (operator) by displaying it on the display device 410.

(4.2) Specific Example of Operation of the Control Device FIG. 11 is a diagram for explaining a specific example of control performed by the control device 100 according to the third embodiment during the operation of "mounting components on a board."

As mentioned above, the task type "mounting components on a board" is carried out in the following order: Process 1 "Recognize component → move" → Process 2 "Grab component" → Process 3 "Move component toward board" → Process 4 "Approach" → Process 5 "Recognize holes on board" → Process 6 "Approach hole" → Process 7 "Hole tracing operation" → Process 8 "Insert hole" → Process 9 "Hole insertion completed" → Process 10 "Release gripping", and the operation parameters for each of these processes are included in the settings library.

In each step, the control unit 170 (command generation unit 172) generates a control command to reduce the difference between the measurement information obtained using the sensor 300 and the target measurement information (target value of the measurement information), and to reduce the difference between the speed-related value and the speed-related target value.

In the illustrated example, for step 1 "Part recognition → movement", "part position" can be applied as the target measurement information, and "speed" can be applied as the speed-related target value. In this case, the control unit 170 (command generation unit 172) can generate a control command to reduce the difference between the current position of the end effector obtained using the visual sensor 310 and the position of the part, and to reduce the difference between the end effector speed and the speed target value. Here, the control unit 170 (target setting unit 176) can change the speed target value for each control cycle depending on the difference between the current position of the end effector obtained using the visual sensor 310 and the position of the part.

In step 2 "Part gripping", "target reaction force" can be applied as the target measurement information, and "speed" can be applied as the speed-related value. In this case, the control unit 170 (command generation unit 172) can generate a control command to reduce the difference between the target reaction force and the current reaction force acting on the end effector obtained using the force sensor 320, and also to reduce the difference between the end effector speed and the speed target value. Here, the control unit 170 (target setting unit 176) can change the speed target value for each control cycle depending on the difference between the target reaction force and the reaction force acting on the end effector obtained using the force sensor 320.

In step 3, "Move the component toward the board," "board position" may be applied as the target measurement information, and "jerk" may be applied as the velocity-related value. In this case, the control unit 170 (command generation unit 172) may generate a control command to reduce the difference between the current position of the end effector (or component) obtained using the visual sensor 310 and the position of the board, and to reduce the difference between the jerk of the end effector (or component) and the target jerk value. Here, the control unit 170 (target setting unit 176) may change the target jerk value for each control cycle depending on the difference between the current position of the end effector (or component) obtained using the visual sensor 310 and the position of the board.

For step 4 "Approach," the "difference in position between the end effector and the board" can be applied as the target measurement information, and "acceleration" can be applied as the speed-related value. In this case, the control unit 170 (command generation unit 172) can generate a control command to reduce the difference between the current position of the end effector (or component) obtained using the visual sensor 310 and the position of the board, and to reduce the difference between the acceleration of the end effector and the target acceleration value. Here, the control unit 170 (target setting unit 176) can change the target acceleration value for each control cycle depending on the difference between the current position of the end effector (or component) obtained using the visual sensor 310 and the position of the board.

In step 5, "Board Hole Recognition," "Board Hole" can be applied as the target measurement information. In this case, the control unit 170 (command generation unit 172) recognizes the board hole using the visual sensor 310.

In step 6 "Hole Approach", "Board hole position" and "target reaction force" can be applied as target measurement information, and "acceleration" can be applied as a speed-related value. In this case, the control unit 170 (command generation unit 172) can generate a control command to reduce the difference between the current position of the component obtained using the visual sensor 310 and the position of the hole on the board, and to reduce the difference between the acceleration of the end effector and the target acceleration value. The control unit 170 (command generation unit 172) can also generate a control command to reduce the difference between the target reaction force and the current reaction force acting on the end effector obtained using the force sensor 320, and to reduce the difference between the acceleration of the end effector and the target acceleration value.

For each step from step 7 "Hole Copying Operation" to step 10 "Grip Release", a "target reaction force" for each step can be applied as target measurement information, and a "jerk" can be applied as a speed-related value. In this case, the control unit 170 (command generation unit 172) may generate a control command to reduce the difference between the current reaction force acting on the end effector obtained using the force sensor 320 and the target reaction force for each step, and to reduce the difference between the jerk of the end effector and the target jerk value for each step. Here, the control unit 170 (target setting unit 176) may change the target jerk value for each control cycle depending on the difference between the reaction force acting on the end effector obtained using the force sensor 320 and the target reaction force.

(5) Other Embodiments The operational flows and operational examples in the above-described embodiments do not necessarily have to be executed in chronological order according to the order shown in the flow diagrams. For example, steps in the operations may be executed in an order different from that shown in the flow diagrams, or may be executed in parallel. Furthermore, some steps in the operations may be deleted, or additional steps may be added to the processing.

A program may be provided that causes a computer to execute the operations of the above-described embodiments. The program may be recorded on a computer-readable medium. Using the computer-readable medium, the program can be installed on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory storage medium. The non-transitory storage medium is not particularly limited, but may be, for example, a storage medium such as a CD-ROM or DVD-ROM.

The functions achieved by the devices according to the above-described embodiments may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and/or combinations thereof, programmed to achieve the described functions. A processor includes transistors or other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in memory. In this disclosure, circuitry, units, and means refer to hardware that is programmed to achieve the described functions or that executes them. The hardware may be any hardware disclosed herein or any hardware known to be programmed or capable of performing the described functions. If the hardware is a processor, which is considered a type of circuitry, the circuitry, means, or unit is a combination of hardware and software used to configure the hardware and/or processor.

As used in this disclosure, the terms "based on" and "depending on" do not mean "based only on" or "depending only on," unless expressly stated otherwise. The term "based on" means both "based only on" and "based at least in part on." Similarly, the term "depending on" means both "depending only on" and "depending at least in part on." Furthermore, the terms "include," "comprise," and variations thereof do not mean including only the listed items, but may mean including only the listed items or including additional items in addition to the listed items. Furthermore, the term "or" as used in this disclosure is not intended to mean an exclusive or. In this disclosure, when articles are added by translation, such as a, an, and the in English, these articles are intended to include the plural unless the context clearly dictates otherwise.

The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made without departing from the spirit of the invention.

(6) Supplementary Notes The following are additional notes regarding the features of the above-described embodiment.

・Appendix 1
a motion generating unit that generates a control command for executing a reverse motion that changes the motion state of the robot apparatus from a target state to an arbitrary state different from the target state;
an acquisition unit that acquires measurement information obtained using a sensor for measuring the operating state of the robot device;
a data collection unit that repeatedly collects data including a set of the control command and the measurement information during execution of the reverse direction operation in accordance with the control command;
The control device, wherein the sensor includes at least one of a visual sensor and a force sensor.

・Appendix 2
the motion generation unit generates the control command for executing a plurality of patterns of reverse motions that change the motion state of the robot apparatus from the target state to a plurality of arbitrary states different from each other;
The control device according to claim 1, wherein the data collection unit repeatedly collects the data for each of the plurality of patterns of reverse direction motion.

・Appendix 3
The action generation unit
generating the control command such that the amount of data collected increases in an area closer to the target state;
The control device according to claim 1 or 2, wherein the control command is generated such that the amount of data collected decreases as the area becomes farther from the target state.

・Appendix 4
The acquisition unit
shortening the acquisition cycle so that the amount of acquired measurement information increases in an area closer to the target state;
The control device according to any one of appendices 1 to 3, wherein an acquisition cycle is extended so that an amount of the measurement information acquired increases in an area farther from the target state.

Appendix 5
the acquisition unit acquires, as relative measurement information, a difference between target measurement information acquired in the target state and current measurement information acquired during execution of the reverse direction motion;
The control device according to any one of appendices 1 to 4, wherein the data collection unit repeatedly collects the data including a set of the control command and the relative measurement information while the reverse direction operation is being performed in accordance with the control command.

・Appendix 6
the acquisition unit acquires the measurement information including a velocity-related value related to the robot apparatus in response to the output of the sensor or the control command;
The control device according to any one of appendices 1 to 5, wherein the data collection unit repeatedly collects the data including a set of the control command and the measurement information including the speed-related value during execution of the reverse direction operation in accordance with the control command.

Appendix 7
a setting acquisition unit that acquires setting information for performing a task using the robot device based on the data collected by the data collection unit;
7. The control device according to any one of claims 1 to 6, further comprising: a control unit that uses the setting information to control the robot device to perform the work.

Appendix 8
the setting acquisition unit acquires the setting information including a trained model for deriving the control command from the measurement information by machine learning using the data collected by the data collection unit as learning data;
The control device according to claim 7, wherein the control unit performs the control based on the measurement information obtained during the work, using the setting information including the trained model.

Appendix 9
the sensors include the visual sensor and the force sensor;
the setting acquisition unit acquires the setting information including information for setting a control ratio between visual control, which is the control based on visual measurement information obtained using the visual sensor, and force control, which is the control based on force measurement information obtained using the force sensor; and
The control device according to claim 7 or 8, wherein the control unit uses the setting information to dynamically or stepwise change the control ratio depending on a status of a task using the robot device.

Appendix 10
The control unit
deriving a velocity-related value for the robot device from the measurement information or control commands obtained during the work;
10. The control device according to any one of appendices 7 to 9, wherein the control device repeatedly performs control on the robot device to reduce a difference between the speed-related value and a speed-related target value based on the setting information and the measurement information.

Appendix 11
the operation generation unit generates the control command for executing the reverse operation for each type of work using the robot device;
The control device according to any one of appendixes 1 to 10, wherein the data collection unit collects the data for each of the types by repeatedly collecting the data for each of the types.

Appendix 12
a setting acquisition unit that acquires setting information for performing work using the robot device for each type based on the data collected for each type by the data collection unit;
12. The control device according to claim 11, further comprising: a control unit that uses the setting information corresponding to a type of work to actually be performed to control the robot device to perform the work.

Appendix 13
generating a control command for executing a reverse movement that changes the motion state of the robot device from a target state to an arbitrary state different from the target state;
acquiring measurement information obtained using a sensor for measuring an operating state of the robot device;
repeatedly collecting data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command;
The sensor includes at least one of a visual sensor and a force sensor.

Appendix 14
The control device
generating a control command for executing a reverse movement that changes the motion state of the robot device from a target state to an arbitrary state different from the target state;
acquiring measurement information obtained using a sensor for measuring an operating state of the robot device;
repeatedly collecting data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command;
The sensor includes at least one of a visual sensor and a force sensor.

Claims (14)

a motion generating unit that generates a control command for executing a reverse motion that changes the motion state of the robot apparatus from a target state to an arbitrary state different from the target state;
an acquisition unit that acquires measurement information obtained using a sensor for measuring the operating state of the robot device;
a data collection unit that repeatedly collects data including a set of the control command and the measurement information during execution of the reverse direction operation in accordance with the control command;
The control device, wherein the sensor includes at least one of a visual sensor and a force sensor. the motion generation unit generates the control command for executing a plurality of patterns of reverse motions that change the motion state of the robot apparatus from the target state to a plurality of arbitrary states different from each other;
The control device according to claim 1 , wherein the data collection unit repeatedly collects the data for each of the plurality of patterns of reverse direction motion. The action generation unit
generating the control command such that the amount of data collected increases in an area closer to the target state;
The control device according to claim 1 , wherein the control command is generated so that the amount of data collected decreases in an area farther from the target state. The acquisition unit
shortening the acquisition cycle so that the amount of acquired measurement information increases in an area closer to the target state;
The control device according to claim 1 , wherein the acquisition period is extended so that the amount of acquired measurement information increases in an area farther from the target state. the acquisition unit acquires, as relative measurement information, a difference between target measurement information acquired in the target state and current measurement information acquired during execution of the reverse direction motion;
The control device according to claim 1 , wherein the data collection unit repeatedly collects the data including a set of the control command and the relative measurement information during the execution of the reverse direction operation in accordance with the control command. the acquisition unit acquires the measurement information including a velocity-related value related to the robot apparatus in response to the output of the sensor or the control command;
The control device according to claim 1 , wherein the data collection unit repeatedly collects the data including a set of the control command and the measurement information including the speed-related value while the reverse direction operation is being performed in accordance with the control command. a setting acquisition unit that acquires setting information for performing a task using the robot device based on the data collected by the data collection unit;
The control device according to claim 1 , further comprising: a control unit that uses the setting information to control the robot device to perform the work. the setting acquisition unit acquires the setting information including a trained model for deriving the control command from the measurement information by machine learning using the data collected by the data collection unit as learning data;
The control device according to claim 7 , wherein the control unit performs the control based on the measurement information obtained during the work, using the setting information including the trained model. the sensors include the visual sensor and the force sensor;
the setting acquisition unit acquires the setting information including information for setting a control ratio between visual control, which is the control based on visual measurement information obtained using the visual sensor, and force control, which is the control based on force measurement information obtained using the force sensor; and
The control device according to claim 7 , wherein the control unit uses the setting information to dynamically or stepwise change the control ratio depending on the status of work using the robot device. The control unit
deriving a velocity-related value for the robot device from the measurement information or control commands obtained during the work;
The control device according to claim 7 , wherein the control device repeatedly controls the robot device to reduce the difference between the speed-related value and the speed-related target value based on the setting information and the measurement information. the operation generation unit generates the control command for executing the reverse operation for each type of work using the robot device;
The control device according to claim 1 , wherein the data collection unit collects the data for each of the types by repeatedly collecting the data for each of the types. a setting acquisition unit that acquires setting information for performing work using the robot device for each type based on the data collected for each type by the data collection unit;
The control device according to claim 11 , further comprising: a control unit that uses the setting information corresponding to a type of work to be actually performed to control the robot device to perform the work. generating a control command for executing a reverse movement that changes the motion state of the robot device from a target state to an arbitrary state different from the target state;
acquiring measurement information obtained using a sensor for measuring an operating state of the robot device;
repeatedly collecting data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command;
The sensor includes at least one of a visual sensor and a force sensor. The control device
generating a control command for executing a reverse movement that changes the motion state of the robot device from a target state to an arbitrary state different from the target state;
acquiring measurement information obtained using a sensor for measuring an operating state of the robot device;
repeatedly collecting data including a set of the control command and the measurement information during the execution of the reverse direction operation in accordance with the control command;
The sensor includes at least one of a visual sensor and a force sensor.