WO2024180656A1 - Learning device, control device, control system, learning method, and storage medium - Google Patents

Abstract

In the present invention, a level set function learning unit learns a level set function in which an operation parameter stipulating an operation objective and operation environment of a robot, and a control parameter of the robot are input, and in which an evaluation value relating to the achievability of the operation objective based on the operation parameter and the control parameter is output. A high-level controller learning unit learns a high-level controller that determines, for the robot, a control parameter for realizing the objective operation based on the operation parameter, on the basis of the precision of prediction of the control parameter and the level set function.

Description

Learning device, control device, control system, learning method, and storage medium

This application relates to a learning device, a control device, a control system, a learning method, and a storage medium.

Patent Document 1 describes an information processing device that generates one or more virtual models that serve as moving machines, and virtually operates the moving machines in a virtual environment in which the generated virtual models are placed. When there are multiple types of moving machines specified by parameters, the information processing device performs simulations for each type of moving machine. In addition, the information processing device uses the detection results of sensors in the virtual environment to operate the moving machine to be learned with arbitrary control content, and determines whether a preset operation result is obtained.

JP 2019-153246 A

The information processing device described in Patent Document 1 merely determines whether a preset operation result can be obtained by the operating machine that is the learning target. Depending on the operating environment, it may be desirable to adjust the parameters used during learning. Since the operation result for the adjusted parameters cannot be known in advance, it is not possible to determine whether the operation will be successful.

The present application aims to provide a learning device, a control device, a control system, a learning method, and a storage medium that solve the above problems.

According to a first aspect of the present application, a learning device includes a level set function learning unit that learns a level set function that receives as input an operation parameter that defines an operation goal and an operation environment of a robot and a control parameter of the robot, and outputs an evaluation value regarding the reachability of the operation goal based on the operation parameter and the control parameter;
The robot is equipped with a high-level controller learning unit that learns a high-level controller that determines control parameters for realizing a target motion based on the motion parameters, based on the level set function and the prediction accuracy of the control parameters.

According to a second aspect of the present application, the learning method includes a level set function learning step in which a learning device learns a level set function that receives as input operation parameters that define the robot's operation target and operation environment and the robot's control parameters, and outputs an evaluation value related to the attainment of the operation target based on the operation parameters and the control parameters, and a high-level controller learning step in which a high-level controller that determines control parameters for realizing the robot's target operation based on the operation parameters is learned based on the level set function and the prediction accuracy of the control parameters.

In a third aspect of the present application, the control device comprises a high-level control unit that determines control parameters for realizing a target motion based on motion parameters that define the motion objectives and motion environment of the robot, and a motion planning unit that calculates an evaluation value indicating the feasibility of the target motion based on the motion parameters and the control parameters using a level set function, and when it is determined that the target motion is feasible based on the evaluation value, uses the control parameters for motion control of the robot, and when it is determined that the target motion is not feasible based on the evaluation value, searches for control parameters that will make the target motion feasible based on the evaluation value.

According to one aspect of the present application, even if the control parameters used to learn the high-level controller are modified, it is possible to determine whether the target state can be reached.

FIG. 1 is a diagram illustrating an example of the configuration of a control system according to a first embodiment. FIG. 11 is a diagram showing an example of known task parameters. FIG. 13 is a diagram illustrating an example of unknown task parameters. FIG. 2 is a diagram illustrating an example of a hardware configuration of the learning device according to the first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of a robot controller according to the first embodiment. FIG. 1 is a diagram illustrating a robot according to a first embodiment. FIG. 1 illustrates an example system state represented in an abstract space. FIG. 2 is a diagram illustrating an example of the configuration of a control system related to skill execution in the first embodiment. FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device related to updating of a skill database in the first embodiment. FIG. 2 is a diagram illustrating an example of the configuration of a skill learning unit according to the first embodiment. 4 is a data flow diagram illustrating a data flow in a skill learning unit according to the first embodiment. FIG. 4 is a flowchart illustrating a learning process according to the first embodiment. 1 is a flowchart illustrating an operation plan according to the first embodiment. FIG. 11 is a data flow diagram illustrating a data flow in a skill learning unit according to the second embodiment. 10 is a flowchart illustrating a learning process according to a second embodiment. A data flow diagram illustrating a data flow in a skill learning unit according to the third embodiment. FIG. 13 is a schematic block diagram illustrating an example of a functional configuration of a system model learning unit according to the third embodiment. 13 is a flowchart illustrating a learning process according to the third embodiment. FIG. 2 is a schematic block diagram illustrating an example of a minimum configuration of a learning device according to an embodiment of the present application. FIG. 2 is a schematic block diagram illustrating an example of a minimum configuration of a control device according to an embodiment of the present application.

Hereinafter, the embodiments of the present application will be described with reference to the drawings. The following description is not intended to limit the scope of the claims. In addition, not all combinations of technical features described in the embodiments are necessarily essential to solving the problem. In other words, even if some of the combinations are omitted, other parts may lead to the solution of the problem. In addition, for convenience, a character with an arbitrary symbol "x" added above an arbitrary letter "A" may be expressed as "A ^x " in this application. For example, the notation y^ may indicate a character formed by combining the symbol ^ directly above the letter y.

First Embodiment
(1) System Configuration An example of the system configuration of the control system 100 according to the first embodiment will be described.
1 is a diagram showing an example of the configuration of a control system 100 according to the first embodiment. The control system 100 includes a learning device 1, a storage device 2, a robot controller 3, a measuring device 4, and a robot 5. The learning device 1 is connected to the storage device 2 wirelessly or via a wire so as to be able to input and output various types of data. The robot controller 3 is connected to each of the storage device 2, the measuring device 4, and the robot 5 wirelessly or via a wire so as to be able to input and output various types of data. The connection between the learning device 1 and the storage device 2, and the connection between the robot controller 3 and each of the storage device 2, the measuring device 4, and the robot 5 may be made directly or via a communication network.

The learning device 1 learns the actions of the robot 5 to execute a given task. In learning the actions, a machine learning method such as self-supervised learning (SSL) is used. The learning device 1 also learns a set of system states that enable the execution of the action to be learned.

In this application, the actions and system states to be learned by the learning device 1 are not limited. The learning device 1 can control actions and system information that are controllable and that enable learning of that control. The processes to be controlled are not limited to actions that involve changes in position or form. For example, the actions of the robot 5 to be controlled may include obtaining measurement data using a sensor.

The system state refers to the state of the system to be controlled, including the robot 5 and the operating environment of the robot 5. The robot 5 and the operating environment of the robot 5 are sometimes collectively referred to as the target system, or simply the system. For tasks that involve handling an object, such as a task to grasp an object, the object may also be included in the target system.

The state of a target system is sometimes called the system state or simply the state. The system state upon completion of a task is sometimes called the goal state of the task or simply the goal state. A set of goal states is sometimes called a goal state set. Reaching the goal state of a task is sometimes called accomplishing the task or succeeding in the task.
When a task is accomplished by executing a skill, the state at the end of skill execution corresponds to the goal state.
The state of the system when a task begins is sometimes called the initial state of that task.

The learning device 1 performs learning on the skills of the robot 5. Each skill is formed by modularizing one or more specific actions of the robot 5. In this application, we will mainly assume a task that can be achieved by executing one skill for one task at a time, and explain the case where the learning device 1 learns the skills to achieve that task.

However, the robot controller 3 may be capable of executing a task that is a combination of multiple skills. For example, the robot controller 3 may plan the execution of a task that is made up of multiple subtasks by combining skills for executing the individual subtasks.

The learning device 1 may, in learning a skill, learn a set of states that makes it possible to execute the skill. The learning device 1 stores information about the skill obtained by learning in the storage device 2, and forms a skill database. Information registered in the skill database is sometimes called a skill tuple. A skill tuple includes various information required to execute the operation that constitutes the skill through modularization. The learning device 1 generates a skill tuple based on detailed system model information, low-level controller information, and target parameter information stored in the storage device 2.

The storage device 2 stores various information that can be referenced by the learning device 1 and the robot controller 3. The storage device 2 stores, for example, detailed system model information, low-level controller information, target parameter information, and a skill database. The storage device 2 does not necessarily have to be configured separately from the learning device 1 or the robot controller 3. The storage device 2 may be built into the learning device 1 or the robot controller 3. The storage device 2 may be configured to include an external storage device such as a hard disk directly connected to or built into the learning device 1 or the robot controller 3, or a storage medium such as a flash memory. The storage device 2 may be a server device that can execute data communication with the learning device 1 and the robot controller 3. The storage device 2 may also be configured to include multiple storage media, and each storage medium may be distributed in the control system 100.

Detailed system model information is information that represents a model of a target system in real space. In this application, the model of a target system in real space may also be referred to as a "detailed system model." The detailed system model may be referred to as a "detailed" system model to distinguish it from a more abstract "abstract" system model.
The detailed system model information may be represented by a differential equation or a difference equation that represents the detailed system model. The detailed system model information may be configured as a simulator program that simulates the operation of the robot 5.

The low-level controller information is information related to the low-level controller. The low-level controller generates a control input for controlling the actual movement of the robot 5 based on parameter values output from the high-level controller. For example, when the high-level controller generates a trajectory for the robot 5, the low-level controller generates a control input that follows the movement of the robot 5 according to the trajectory. The low-level controller may also control the movement of the robot 5 using servo control with PID (Proportional Integral Differential) based on parameters output from the high-level controller.

Target parameter information is provided for each skill to be learned by the learning device 1. Target parameter information includes, for example, initial state information, target state/known task parameter information, unknown task parameter information, execution time information, and general constraint information. Here, the variable parts of a task are sometimes referred to as task parameters.

Among the task parameters, those that are expressed by numerical values are sometimes referred to as known task parameters. Examples of known task parameters include the size of an object to be grasped in a case where the task is a task of grasping an object, the size of an object in the task, and the trajectory of the robot 5 for executing the task.
The known task parameters may also be treated as parameters of a skill.

FIG. 2 is a diagram showing examples of known task parameters. FIG. 2 shows an example of a case where the robot 5 executes a task of grasping an object having a cylindrical shape. In this case, the radius and height of the object, which is the cylinder, correspond to examples of known task parameters.

On the other hand, task parameters that are difficult to express numerically are sometimes called unknown task parameters. Examples of unknown task parameters include the shape of the object in the task, such as the shape of the object to be grasped when the task is to grasp an object, and the type of movement of the robot 5 to execute the task, such as the skills required to execute the task.

FIG. 3 is a diagram showing examples of unknown task parameters. FIG. 3 shows an example in which the robot 5 executes a task of grasping objects of various shapes. In this case, the shapes of the objects correspond to an example of unknown parameters.

Furthermore, in this application, it is assumed that the control system 100 handles quantified system states, and the target state is expressed as a numerical value. For example, in the case of a task in which the robot 5 performs pick-and-place, the target state is expressed as the coordinates of the object to be operated being within a predetermined range.

The initial state information is information indicating a set of states in which the target skill can be executed. The state at the start of execution of a skill is also referred to as the initial state of that skill, or simply as the initial state. The set of initial states is also referred to as an initial state set. The parameters of the initial state are also referred to as initial state parameters. For example, when the task of the robot 5 is pick-and-place, the initial hand position and posture correspond to the initial state parameters. The initial hand position corresponds to the position of the end effector of the robot 5 when performing an action. The initial posture corresponds to the overall shape of the robot 5 before performing an action. When the robot 5 is configured to include multiple joints, the angle between each pair of two joints that are connected to each other is an element of the initial posture.
In the present application, an initial state or an initial state parameter may be expressed as _xs or _xsi , where "i" is a positive integer representing an identification number for identifying each initial state. In addition, the time when the skill execution starts may be set to 0, and the initial state may be expressed as _x0 .

The goal state/known task parameter information is information indicating a set of combinations of possible values of a goal state, which is a state that can be reached by executing a target skill, and possible values of known task parameters treated as dynamic parameters in the target skill. For example, in a skill in which the robot 5 grasps an object, a target gripping position and posture may be included as an element of the goal state. The target gripping position refers to the position of the end effector that grasps the object to be grasped, and may be expressed as a relative target value based on the position of the object. The target posture refers to a target value of the shape of the robot 5 at the start of the gripping operation of the object. The goal state may include information on stable gripping conditions such as form closure and force closure as possible values.
In this application, a combination of a goal state and a known task parameter value is referred to as a goal state/known task parameter value or a goal state parameter, and may be expressed as β _g or β _gi . Here, "i" is a positive integer representing an identification number that identifies each goal state/known task parameter value. When the task of the robot 5 is pick-and-place, the final target position of the end effector of the robot 5 and the position of the operation target object correspond to the goal state parameters. In this case, the size of the operation target object corresponds to the known task parameter.

By treating the differences in the goal states of a task and the differences in known task parameter values as skill parameters, tasks that differ in either the goal states or known task parameter values, or both, can be executed as a single skill.

For example, when the learning device 1 performs processing related to skill learning using a predictor, a target state and known task parameter values can be input to the predictor to obtain an output value corresponding to the target state and known task parameter values. The predictor is configured using a learning model (a model in machine learning) such as a neural network or a Gaussian Process (GP).

Note that, depending on the skill, there may be cases where the known task parameter is not set. In this case, the goal state/known task parameter information may be configured as a set of values that the goal state can take. Also, the goal state/known task parameter value β _g may be a value indicating the goal state.

The unknown task parameter information is information about an unknown task parameter. For example, the unknown task parameter information may indicate a probability distribution of data about the unknown parameter. When one skill has multiple unknown task parameters, the unknown task parameter information may indicate information about each unknown task parameter.
Values corresponding to the target state/known task parameters and unknown task parameters may be fixed values or may be variable.

In this application, an unknown task parameter value may be represented as τ or τ _j , where “j” is a positive integer representing an identification number that identifies the unknown task parameter value.
Although it is difficult to express unknown task parameters numerically because it is difficult to systematically quantify their values, it is possible to determine whether unknown task parameter values are equal. For example, if an unknown task parameter represents the shape of an object, it is possible to determine whether the unknown task parameter values are equal by comparing the shapes of two objects.
The control system 100 treats two tasks as the same task if the unknown task parameter values in the two tasks are the same, and treats the two tasks as separate tasks if the unknown task parameter values are different. A task may be represented by τ or τ _j . The above "j" may be regarded as a positive integer representing an identification number that identifies an individual task.

The execution time information is information about a time limit for executing a skill. For example, the execution time information may indicate the execution time of a skill (the time required to execute the skill), or the allowable condition value of the time from the start to the end of the skill execution, or both.
The general constraint information is information indicating general constraint conditions, such as conditions related to limits on the range of motion of the robot 5, speed limits, and input limits.

The skill database is a database that has a skill tuple that is set for each skill. The skill tuple may include information about a high-level controller for executing the target skill, information about a low-level controller for executing the target skill, and information about a set of states in which the target skill can be executed (e.g., the initial state of the skill) and combinations of goal states/known task parameter values. The set of states in which the target skill can be executed and goal states/known task parameter values is also referred to as an executable state set.

The feasible state set may be defined in an abstract space obtained by abstracting an actual space. The feasible state set may be expressed by, for example, a level set function estimated by Gaussian Process Regression (GPR) or Level Set Estimation (LSE), or an approximation function of the level set function. In other words, whether the feasible state set includes a combination of a certain state and a target state/known task parameter value may be determined by whether a value (e.g., an average value) of the Gaussian process regression for the combination of the certain state and the target state/known task parameter value, or a value of an approximation function for the combination of the certain state and the target state/known task parameter value, satisfies a constraint condition for determining feasibility.
In the following, a case where a level set function is used as a function indicating a feasible state set will be described as an example, but the present invention is not limited to this.

The robot controller 3 is a control device that formulates an operation plan for the robot 5 based on the measurement signals supplied from the measuring device 4, a skill database, etc. The robot controller 3 generates control commands (control inputs) for causing the robot 5 to execute the planned operations, and supplies the control commands to the robot 5.

For example, the robot controller 3 converts the tasks to be executed by the robot 5 into a sequence indicating tasks that the robot 5 can accept for each predetermined time step (time interval). The robot controller 3 then controls the robot 5 based on control commands equivalent to execution commands for the tasks instructed in the generated sequence. The control commands correspond to the control inputs output by the low-level controller.

The measurement device 4 includes one or more sensors and detects the state within the workspace in which the robot 5 executes a task. The sensor is, for example, a camera, a range sensor, a sonar, or a combination of these. The measurement device 4 supplies the generated measurement signal to the robot controller 3. The measurement device 4 may include a self-propelled or flying sensor (including a drone) that moves within the workspace. The measurement device 4 may also include a sensor provided on the robot 5 itself, and a sensor provided on another object in the workspace. The measurement device 4 may also include a sensor that detects sound within the workspace, i.e., a microphone. In this way, the measurement device 4 may include various sensors that detect the state within the workspace and are provided at any location.

The robot 5 executes operations related to the tasks instructed based on the control commands supplied from the robot controller 3. The robot 5 may be, for example, a robot that operates in various factories such as an assembly plant or a food factory, or at a logistics site. The robot 5 may be a vertical articulated robot, a horizontal articulated robot, or a robot having another structure. The robot 5 may supply a status signal indicating the status of the robot 5 itself to the robot controller 3. This status signal may be an output signal of a sensor that detects the status (e.g., position, angle, etc.) of the entire robot 5 or a specific part (e.g., a joint, etc.) or may be a signal indicating the progress of the operation of the robot 5.

1 may be modified in various ways. For example, the robot controller 3 and the robot 5 may be integrated. In another example, at least two of the learning device 1, the storage device 2, and the robot controller 3 may be integrated.
Furthermore, the control target of the control system 100 is not limited to the robot 5. Various control targets that the learning device 1 can learn may be the control targets of the control system 100.

(2) Hardware Configuration Next, an example of the hardware configuration of the learning device 1 according to this embodiment will be described.
4 is a diagram showing an example of the hardware configuration of the learning device 1 according to this embodiment. The learning device 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 10 so as to be able to input and output various types of data.

The processor 11 functions as a controller (computing device) that controls the entire learning device 1 by executing a program stored in the memory 12. In this application, "executing a program" or "executing a program" may refer to executing processing instructed by various commands written in the program. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 is not limited to one processor, and may be configured to include multiple processors. The processor 11 constitutes the computer of the learning device 1.

Memory 12 comprises a storage medium that stores various information referenced by learning device 1 or other hardware. Memory 12 is composed of various volatile and non-volatile memories, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. Memory 12 stores programs that contain commands that instruct the processing to be executed by processor 11. Some of the information stored in memory 12 may be stored in one or more external storage devices (e.g., storage device 2) that can communicate with learning device 1, or may be stored in storage media that are removable from other components of learning device 1.

The interface 13 includes an interface for connecting the learning device 1 to other devices so that various data can be input and output. The interface 13 may be a wireless interface such as a network adapter for wirelessly transmitting and receiving data to and from other devices, or a hardware interface for wired transmission and reception of data to and from other devices. For example, the interface 13 may be connected to an input device that accepts user input (external input) such as a touch panel, button, keyboard, or voice input device, a display device such as a display or projector, or a sound output device such as a speaker.

The hardware configuration of the learning device 1 is not limited to the configuration illustrated in FIG. 4. For example, the learning device 1 may incorporate at least one of a display device, an input device, and a sound output device. The learning device 1 may also be configured to include a storage device 2.

FIG. 5 is a diagram showing an example of the hardware configuration of the robot controller 3 according to this embodiment. The robot controller 3 includes, as hardware, a processor 31, a memory 32, and an interface 33. The processor 31, the memory 32, and the interface 33 are connected via a data bus 30 so that various types of data can be input and output.

The processor 31 functions as a controller (computing device) that performs overall control of the robot controller 3 by executing the programs stored in the memory 32. The processor 31 is, for example, a processor such as a CPU, a GPU, or a TPU. The processor 31 is not limited to a single processor, and may be configured to include multiple processors.

The memory 32 is configured to include one or more storage media. The memory 32 includes various types of volatile and non-volatile memory, such as RAM, ROM, and flash memory. The memory 32 also stores programs executed by the processor 31. Some of the information stored in the memory 32 may be stored in one or more external storage devices (e.g., storage device 2) capable of communicating with the robot controller 3, or may be stored in a storage medium that is detachable from other parts of the robot controller 3.

The interface 33 is an interface for connecting the robot controller 3 to other devices so that various data can be input and output. The interface 33 may include a wireless interface such as a network adapter for wirelessly transmitting and receiving data to and from other devices, or may include a hardware interface for wired transmission and reception of data to and from other devices.

The hardware configuration of the robot controller 3 is not limited to the configuration exemplified in FIG. 5. For example, the robot controller 3 may incorporate at least one of a display device, an input device, and a sound output device. The robot controller 3 may also be configured to include a storage device 2.

(3) Abstract Space Next, the abstract space according to this embodiment will be described. The abstract space is a virtual space separate from the real space, and is used by the robot controller 3 to formulate an operation plan for the robot 5 based on the skill tuples.

FIG. 6 illustrates a robot (manipulator) 5 that grasps an object in real space, and an object 6 to be grasped.
FIG. 7 represents the system state illustrated in FIG. 6 in an abstract space.

In general, to formulate an operation plan for the robot 5 having a pick-and-place task, strict calculation is required that takes into consideration the shape of the end effector of the robot 5, the geometric shape of the object 6 to be grasped, the grasping position and posture of the robot 5, and the object characteristics of the object 6 to be grasped. In contrast, in this embodiment, the robot controller 3 formulates an operation plan in an abstract space in which the states of each object, such as the robot 5 and the object 6 to be grasped, are abstractly (simply) represented. In the example of FIG. 7, in the abstract space, an abstract model 5x corresponding to the end effector of the robot 5, an abstract model 6x corresponding to the object 6 to be grasped, and a region (see dashed frame 60) in which the robot 5 can grasp the object 6 to be grasped are defined. Note that, in the abstract space as well, as described above, the executable state set is represented as a set of combinations of the initial state and the target state/known task parameter value in which the skill can be executed. 7, a set of combinations of initial states and goal states/known task parameter values that allow the grasping skill to be executed is illustrated as a grasp operation executable region in a dashed frame 60. The executable state set or goal states/known task parameter values correspond to operation parameters that define the robot's operation goals and operating environment.
In this way, the state of the end effector and the like are abstractly represented by the state of the robot in the abstract space. Also, the state of each object corresponding to the operation target or the environmental object can be abstractly represented in a coordinate system based on the position of a reference object such as a workbench.

In this embodiment, the robot controller 3 uses skills to formulate an action plan in an abstract space that abstracts the actual system. This makes it possible to effectively reduce the computational costs required for action planning even in multi-stage tasks. In the example of FIG. 7, the robot controller 3 formulates an action plan to execute skills for performing grasping in a graspable area (dashed frame 60) defined in the abstract space, and generates control commands for the robot 5 based on the formulated action plan.

Hereinafter, the state of a system in real space (sometimes referred to as a "real system" in this application) may be denoted as "x" and the state of a system in abstract space (sometimes referred to as an "abstract system" in this application) may be denoted as "x'" to distinguish between them. The state x' is expressed as a vector (sometimes referred to as an "abstract state vector" in this application). For example, for a task such as pick-and-place, the abstract state vector includes a vector representing the state of an object to be manipulated (e.g., position, orientation, velocity, etc.), a vector representing the state of an end effector of a manipulable robot 5, and a vector representing the state of an environmental object. The state x' is defined as a state vector that abstractly represents the state of some elements in the real system.
Similarly, a goal state/known task parameter value in the real space may be expressed as "β _g ", and a goal state/known task parameter value in the abstract space may be expressed as "β _g '" to distinguish between them.

(4) Control System for Skill Execution Next, a configuration example of a control system for skill execution according to this embodiment will be described. Fig. 8 is a diagram showing a configuration example of a control system for skill execution according to this embodiment. The processor 31 of the robot controller 3 functionally includes a motion planning unit 34, a high-level control unit 35, and a low-level control unit 36. Furthermore, the system 50 corresponds to an actual system (an actual system including the robot 5).
In this application, the high level control section 35 is also called a high level controller and may be represented as π _H. The high level control section 35 corresponds to an example of a control means. The low level control section 36 is also called a low level controller and may be represented as π _L.
The robot controller 3 is an example of a control device that controls the robot 5 .

The motion planning unit 34 formulates a motion plan for the robot 5 based on the state x' in the abstract system and the skill database. The motion planning unit 34 expresses the goal state by a logical formula based on temporal logic, for example. The motion planning unit 34 may express the logical formula by using a preset temporal logic such as linear temporal logic, metric temporal logic (MTL), or signal temporal logic (STL).
The action planning unit 34 converts the generated logical formula into a sequence (action sequence) for each time step. This action sequence includes, for example, information about the skills used in each time step.

The high-level control unit 35 recognizes the skill to be executed for each time step from the action sequence generated by the action planning unit 34. Then, the high-level control unit 35 generates a parameter "α" to be input to the low-level control unit 36 based on the high-level controller "π _H " included in the skill tuple corresponding to the skill to be executed in the current time step.

The high-level control unit 35 generates a control parameter α as shown in equation (1) when the combination of the state “x _s ′” in the abstract space and the target state/known task parameter value “β _g ′” at the start of execution of the skill to be executed belongs to the executable state set “χ ₀ ′” of that skill.

The initial state at the start of skill execution is represented, for example, by a system state in an abstract space.
The approximation function g^ of the level set function is preset in the robot controller 3. The robot controller 3 can determine whether the initial state x _s ' belongs to the feasible state set χ ₀ ' by determining whether the approximation function g^ of the level set function satisfies equation (2).

Equation (2) can be regarded as a constraint that determines the feasibility of the skill from a certain state. Alternatively, the approximation function g can be regarded as a model for evaluating whether a goal state can be reached from an initial state _xs ' under known task parameter values.
As will be described later, the approximation function g^ is obtained by performing learning by the learning device 1. In this application, the approximation function g^ itself may be referred to as a level set function.

The robot controller 3 executes a skill using the low-level control unit 36, and can determine whether or not the abstract system state x'(T) at the time T has elapsed since the start of execution belongs to a goal state set _χ'd , as shown in formula (3). The goal state set _χ'd is a set of goal states in an abstract space after the execution of the skill to be determined. T indicates the execution time of the skill to be determined.

The low-level control unit 36 generates an input “u” based on the control parameter α generated by the high-level control unit 35, and the current state x and the target state/known task parameter value _βg in the real system obtained from the system 50. The low-level control unit 36 generates the input u as a control command as shown in equation (4) based on the low-level controller “π _L ” included in the skill tuple.

It should be noted that the low-level controller π _L is not necessarily limited to an input/output relationship that can be expressed by equation (4), and may have a different input/output relationship.

The low-level control unit 36 recognizes the state x of the robot 5 and the environment using a predetermined state recognition technology based on the measurement signals output from the measurement device 4 (which may include signals obtained from the robot 5) and the like.
The model of the system 50 is expressed by a state equation (equation (5)) that expresses a function "f" that is an output and a time change x ^· of the state x based on an input u to the robot 5 and the state x. In equation (5), the operator " ^· " represents a differentiation with respect to time or a difference with respect to time.

(5) Updating the Skill Database Next, updating of the skill database according to this embodiment will be described. Fig. 9 is a diagram showing an example of a functional configuration of the learning device 1 related to updating of the skill database according to this embodiment. The processor 11 of the learning device 1 functionally comprises an abstract system model setting unit 14, a skill learning unit 15, and a skill tuple generating unit 16. Note that Fig. 9 shows an example of the type of data exchanged for each block, but is not limited to this.

The abstract system model setting unit 14 sets an abstract system model based on the detailed system model information. The abstract system model that is set is a model obtained by simplifying the detailed system model identified by the detailed system model information. The detailed system model is a model that represents the system 50 in Figure 8.

The abstract system model is a model that provides an abstract system state indicated by an abstract state vector x' that is constructed based on a detailed system state x in the detailed system model. The motion planner 34 formulates a motion plan for the robot 5 using the abstract system model.
The abstract system model setting unit 14 derives the abstract system model from the detailed system model based on, for example, an algorithm stored in advance in the storage device 2 or the like.

In addition, information regarding the abstract system model may be stored in advance in the storage device 2 or the like. In this case, the abstract system model setting unit 14 may acquire information regarding the abstract system model from the storage device 2 or the like. The abstract system model setting unit 14 supplies information regarding the set abstract system model to the skill learning unit 15 and the skill tuple generation unit 16. In this application, the abstract system model information and the detailed system model information may be collectively referred to as system model information.

The skill learning unit 15 learns control related to skill execution based on the abstract system model set by the abstract system model setting unit 14 and the detailed system model information, low-level controller information, and target parameter information stored in the storage device 2. The skill learning unit 15 learns, for example, a control parameter α output from the high-level controller π _H and provided to the low-level controller π _L. In addition, the skill learning unit 15 learns a level set function and acquires training data for learning the control parameter α, as described later. At this time, the skill learning unit 15 evaluates, for example, the prediction accuracy of the level set function.

The skill tuple generation unit 16 generates a tuple by correlating the following information: information on the feasible state set χ ₀ ' obtained by learning in the skill learning unit 15, information on the high-level controller π _H , information on the abstract system model set by the abstract system model setting unit 14, low-level controller information, and target parameter information. The skill tuple generation unit 16 then registers the generated skill tuple in a skill database. The data in the skill database is used by the robot controller 3 to control the robot 5.

The functions of the abstract system model setting unit 14, the skill learning unit 15, and the skill tuple generating unit 16 can be realized, for example, by the processor 11 executing a predetermined program. Also, the program to be executed may be recorded in advance in any non-volatile storage medium, and the processor 11 may install and execute the program to realize these functions. Note that some or all of these functions are not limited to being realized by executing software, but may be realized by hardware or a combination of hardware and software. Also, some or all of these functions may be realized using integrated circuits that can be programmed by the user, such as a field-programmable gate array (FPGA) or a microcontroller. In that case, these integrated circuits execute programs for realizing the above functions. Also, the hardware used to realize the above functions may be other types of hardware, such as an application specific standard produce (ASSP), an application specific integrated circuit (ASIC), or a quantum computer control chip. This point is the same in other embodiments described later. Furthermore, each of these functions may be realized by multiple computers working together, for example, using cloud computing technology.

(6) Configuration of the Skill Learning Unit Next, an example of the configuration of the skill learning unit 15 according to this embodiment will be described.
10 is a diagram showing an example of the configuration of the skill learning unit 15 according to this embodiment. The skill learning unit 15 functionally comprises a search point set setting unit 210, a data acquisition unit 220, and a learning setting unit 230.

The search point set setting unit 210 includes a search point set initialization unit 211 and a search point selection unit 212 .
The data acquisition unit 220 includes a system model setting unit 221 , a problem setting calculation unit 222 , and a data update unit 223 .
The learning setting unit 230 includes a level set function learning unit 231 , a prediction accuracy evaluation function setting unit 232 , a prediction accuracy evaluation unit 233 , a controller learning evaluation function setting unit 234 , and a high-level controller learning unit 240 .

As described above, the skill learning unit 15 generates training data and uses the generated training data to learn the high-level controller π _H. In addition, the skill learning unit 15 learns a level set function.
The search point set setting unit 210 acquires a plurality of sets of search points each consisting of a combination of an initial state x _s and a target state/known task parameter value β _g as candidates for task setting to be learned by _the high-level controller π H. The search point set setting unit 210 selects a search point from which training data is to be acquired from the acquired candidates. The training data is used for learning the control of the robot 5 by the robot controller 3.
The search point set setting unit 210 corresponds to an example of a search point setting means.

The search point set initialization unit 211 sets a search point set. The search point set is a set including a plurality of search points as elements. Each search point is a candidate for task setting to be the target of learning of the high-level controller π _H and the level set function. More specifically, each search point corresponds to a combination of an initial state x _s and a target state/known task parameter value β _g , that is, an operation parameter that is a candidate for an executable state set. The search point set initialization unit 211 sets a predetermined number N (N is a predetermined integer equal to or greater than 2) of search points so that they are randomly distributed in a value range predetermined for each task, for example.

In this application, the set of search points may be represented as Ξ _check , and each search point may be represented as (x _s , β _g ) or ξ.
By determining the search point (x _s , β _g ), a task setting is specified and the behavior of the robot 5 is determined. That is, the search point (x _s , β _g ) can also be regarded as a parameter indicating the behavior of the robot 5 for each task.

The search point selection unit 212 selects a search point ξ _i (i is an integer between 1 and N) for next acquiring training data from the search point set Ξ _check . The search point selection unit 212 outputs the selected search point ξ _i to the data acquisition unit 220 and the learning setting unit 230. The search point selection unit 212 randomly selects one search point ξ _i from a plurality of search points constituting the search point set Ξ _check , for example. After the level set function g^ is learned at least once in the learning setting unit 230, the search point selection unit 212 may select the search point ξ _i with the maximum evaluation value calculated using the prediction accuracy evaluation function J _g^ (ξ _i , π _h (ξ _i )).

The prediction accuracy evaluation function J _^g ( _ξi , _πh ( _ξi )) is a function that indicates the estimated accuracy of the evaluation value obtained for a search point using the level set function g^. The larger the evaluation value of the prediction accuracy evaluation function, the higher the estimation accuracy. The level set function g^ determines an evaluation value that indicates the possibility of reaching the target state. The smaller the evaluation value of the level set function g^, the higher the possibility of reaching the target state. In this case, the search point that gives the evaluation value of the level set function g^ is determined to be capable of reaching the target state.

The data acquisition unit 220 acquires training data for learning the high-level controller π _H using the search points ξ _i input from the search point set setting unit 210, and also acquires training data for learning the level set function g.
The system model setting unit 221 performs settings related to a system model to solve an optimal control problem based on the search point ξ _i . The system model setting unit 221 sets problem setting information indicating previously set system model information, constraint conditions, a low-level controller, a target time, a solution search function, and the search point ξ _i in the problem setting calculation unit 222. The system model information in this example includes detailed system model information and abstract system model information. The constraint conditions include constraint conditions related to the task and constraint conditions related to the operation of the robot.

The optimal control problem (OCP) is a problem of finding a control parameter α that minimizes the evaluation value of a level set function g(x(T), ξ) that indicates the possibility of reaching a target state _χd within a predetermined target time T as shown in formula (6). In this application, optimization includes the meaning of searching for the most appropriate value possible, and is not limited to determining an absolutely optimal value. In other words, optimization in this application does not exclude cases in which the level set function and other evaluation values temporarily change to more inappropriate values during the processing process, or in which an absolutely optimal value cannot be obtained as a processing result. Minimization also includes the meaning of searching for the smallest possible value, and is not limited to determining an absolute minimum value. A model of the system 50, i.e., a system model, is given by formula (5).

The control input u to the robot 5 is the control output from a low-level controller π L (x(t), α, β _g ) determined by the system state x(t) at time t (t is a real number between ₀ and T), the control parameter α, and the target state/known task parameter value β _g . The control parameter α is output from a high-level controller π _H (ξ) as shown in equation (1). As described above, the set of the initial state x _s and the target state/known task parameter value β _g at time t=0 corresponds to ξ _i .

The problem setting calculation unit 222 sets a solution search problem indicating task execution by the robot 5, based on the problem setting information set by the system model setting unit 221. The solution search problem indicates a problem of finding a solution that satisfies the constraint conditions indicated in the problem setting information.
More specifically, the problem setting calculation unit 222 sets the above optimal control problem under the constraint conditions indicated in the problem setting information.

The problem setting calculation unit 222 can solve the set optimal control problem, set the smallest possible evaluation value of the level set function g as the optimal value g _i ^* , and calculate the control parameter α _i ^* that gives the optimal value g _i ^* . The problem setting calculation unit 222 outputs optimal control solution information indicating the optimal value g _i ^* and the control parameter α _i ^* calculated under the search point ξ _i to the data update unit 223. Note that the problem setting calculation unit 222 may further set a system state x(t) under a system model set based on the control parameter α _i ^* , and include the set system state x(t) in the optimal control solution information.

The data update unit 223 updates the training data D opt of the high-level controller π _H and the level set function so as to include a set of the search point ξ _i , the optimal value g _i ^* , and the control parameter α _i ^* indicated in the optimal control solution information input from the problem setting calculation unit 222. The training data D _opt is a data set in _which a set of the optimal value g _i ^* and the control parameter α _i ^* for each search point ξ _i is accumulated, and is also referred to as an acquired data set.

The learning setting unit 230 learns the level set function g^ using the acquired data set D _opt , and sets a prediction accuracy evaluation function J _g^ and a controller learning evaluation function J _h based on the level set function g^ obtained by learning. The learning setting unit 230 uses the prediction accuracy evaluation function J _g^ to determine whether or not learning of the level set function needs to be continued. As described above, the level set function g^ is a function for calculating an evaluation value related to the possibility of reaching the target state for a combination of the system state and the target state/known task parameter value. The prediction accuracy evaluation function J _g^ is a function for evaluating the prediction accuracy of the level set function g^ and, in turn, the prediction accuracy of the high-level controller π _H. The prediction accuracy is used to determine whether or not learning of the level set function g^ needs to be continued. The controller learning evaluation function J _h is a function used as a loss function in learning the high-level controller π _H.

The level set function learning unit 231 uses the acquired data set D _opt as training data to learn the level set function g^. A model of the level set function g^, i.e., a function form, is set in advance in the level set function learning unit 231. In learning the level set function g^, the level set function learning unit 231 searches for parameters of the level set function g^ that minimize the evaluation function for learning the level set function, for example. The evaluation function for learning the level set function can use, for example, the sum of squares of the absolute values of the difference between the evaluation value g^(ξ _i , α _i ) of the level set function g^ for each search point ξ _i that is an element of the acquired data set D _opt and the target value g _i as shown in Equation (7). In formula (7), E _{(ξi, gi, αi)εDopt} [...] denotes the expected value of the evaluation value... for the acquisition data set _Dopt including a set of search point _ξi , target value g _i and control parameter _αi as elements.

The target value g _i may or may not be a negative value depending on whether or not it is possible to reach the target state of the system state x(t) derived by the search point ξ _i . The level set function learning unit 231 sets the level set function g determined by learning to the prediction accuracy evaluation function setting unit 232 and the controller learning evaluation function setting unit 234.

Therefore, the level set function learning unit 231 may determine whether the system state x(t) can reach the target state for each search point ξ _i , which is an element of the acquisition data set D _opt , and add the evaluation result of whether or not the target state can be reached to the acquisition data set D _opt .
For example, the level set function learning unit 231 may add a smaller target value g _i to a search point ξ _i where the final system state x(T) at time T is closer to the target state. Also, the level set function learning unit 231 may add a smaller target value g _{i to a search point ξ i where} the system state _x (t) can reach the target state earlier.

The prediction accuracy evaluation function setting unit 232 refers to the acquired data set D _opt and sets the prediction accuracy evaluation function J _g^ based on the level set function g^ set by the level set function learning unit 231. The prediction accuracy evaluation function setting unit 232 determines, for example, the variance σ _g^ (ξ _i , α _i ) of the evaluation value of the level set function g^ for each search point ξ _i as the evaluation value of the prediction accuracy evaluation function J _g^ . The variance σ _g^ (ξ _i , α _i ) functions as an index indicating that the prediction accuracy of the reachability of the target state by the level set function g^ is low, since the larger the value of the variance σ g^ (ξ i , α _{i )} is, the more different the determination result of whether the target state can be reached for each search point ξ i is. In addition, since the system state is optimized based on the level set function g^, the variance σ g^ (ξ i , α _i ) can also be used as an index of the prediction accuracy of the system state predicted by the high-level controller π _H , the larger the value of the variance σ _g^ (ξ _i , α i ). The prediction accuracy evaluation function setting unit 232 sets the evaluation value of the set prediction accuracy evaluation function J _g^ in the prediction accuracy evaluation unit 233 .

The prediction accuracy evaluation unit 233 uses the evaluation value of the prediction accuracy evaluation function J _g^ set by the prediction accuracy evaluation function setting unit 232 to determine whether or not the learning of the level set function g^ needs to be continued during learning. For example, the prediction accuracy evaluation unit 233 determines whether or not the learning needs to be continued based on whether or not the evaluation value of the prediction accuracy evaluation function J g _^ is greater than a predetermined judgment threshold value of the prediction accuracy evaluation function. In addition, the prediction accuracy evaluation unit 233 may determine whether or not the learning needs to be continued by further considering the learning conditions of the level set function g^. For example, the prediction accuracy evaluation unit 233 determines whether or not the learning needs to be continued based on whether or not the amount of change in the level set function g^ at the current time from the level set function g^ obtained based on the previous search point ξ _i-1 is equal to or greater than a predetermined reference amount of change.

When it is determined that learning needs to be continued, the prediction accuracy evaluation unit 233 outputs a learning continuation flag indicating the need for learning to the search point set setting unit 210. When the learning continuation flag is input from the prediction accuracy evaluation unit 233, the search point selection unit 212 selects a new search point ξ _i from the acquisition data set D _opt and outputs the selected search point ξ _i to the data acquisition unit 220. Therefore, learning of the level set function g^ is continued based on the new search point ξ _i .

If it is determined that continued learning is not necessary, the prediction accuracy evaluation unit 233 does not output a continuing learning flag to the search point set setting unit 210, but causes the level set function learning unit 231 to terminate learning of the level set function g^, and causes the high-level controller learning unit 240 to terminate learning of the high-level controller π _H.
Thereafter, the level set function learning unit 231 sets the level set function g^ at the time when learning is completed in the motion planning unit 34 of the robot controller 3. In addition, the high-level controller learning unit 240 sets the parameters of the high-level controller π _H at the time when learning is completed in the motion planning unit 34 and the high-level control unit 35.

The controller learning evaluation function setting unit 234 determines a controller learning evaluation function Jh based on the evaluation value of the level set function g^ set by the level set function learning unit 231 and the high-level controller π _H set by the high-level controller learning unit 240. The controller learning evaluation function setting unit 234 sets the determined controller learning _{evaluation function Jh to} the high-level controller learning unit 240. The controller learning evaluation function setting unit 234 determines, for example, as shown in equation (8), the average value μ _g^ (prediction average value) of the level set function g^, the variance μ _g^ (prediction variance) of the level set function g^, and the weighted sum of the squared difference |π _H (ξ _i ) - α _i | ² between the control output from the high-level controller π _H and the control parameter α _i as the controller learning evaluation function _Jh . In equation (8), k and λ respectively indicate weighting parameters of the variance μ _g^ and the sum of squared differences |π _H (ξ _i )-α _i | ^2. The weighting parameters k and λ are each a predetermined positive real value of 0 or greater than 0.

As exemplified in equation (8), by including the average value μ _g^ of the level set function g^ in the components of the controller learning evaluation function J _h , the high-level controller π _H is trained to reach the target state. By including the variance σ _g^ of the level set function g^ in the components of the controller learning evaluation function J _h , the high-level controller π _H is trained to reach the target state stably. By including the sum of squared differences |π _H (ξ _i ) - α _i | ² in the components of the controller learning evaluation function J _h , the high-level controller π _H is trained to make it easier to obtain the control parameter α as a control output.

The high-level controller learning unit 240 refers to the acquisition data set D _opt as training data and performs learning of the high-level controller π _H using the controller learning evaluation function J _h . A model of the high-level controller π _H is set in advance in the high-level controller learning unit 240. In learning the high-level controller π _H , the high-level controller learning unit 240 searches for parameters of the high-level controller π _H that minimize the controller learning evaluation function J _h , for example. The high-level controller learning unit 240 sets the high-level controller π _H obtained by learning in the problem setting calculation unit 222 and the controller learning evaluation function setting unit 234.

Next, an example of data flow in the skill learning unit 15 according to this embodiment will be described below. Fig. 11 is a data flow diagram illustrating data flow in the skill learning unit 15 according to this embodiment.
The search point set initialization unit 211 of the search point set setting unit 210 sets the search point set Ξ _check using the target parameter information stored in the storage device 2. The search point set initialization unit 211, for example, refers to the target parameter information, defines a possible combination of one initial state x _si and one target state/known task parameter value β _g as a search point ξ, and configures a search point set Ξ _check including N different search points ξ.

The search point selection unit 212 randomly selects one unprocessed search point ξ _i from the search point set Ξ _check configured in the search point set setting unit 210. When the prediction accuracy evaluation function J _g^ is set by the prediction accuracy evaluation function setting unit 232, the search point selection unit 212 selects the search point ξ _i having the maximum evaluation value of the prediction accuracy evaluation function J _g^ from among the unprocessed search points ξ _i . The search point selection unit 212 outputs the selected search point ξ _i to the data acquisition unit 220.

The system model setting unit 221 of the data acquisition unit 220 configures problem setting information using the system model information stored in the storage device 2, other setting information, and the search point ξ _i input from the search point set setting unit 210. The problem setting information includes the system model information, constraint conditions, low-level controller, target time, solution search function, and information on the search point ξ _i . The system model setting unit 221 sets the configured problem setting information in the problem setting calculation unit 222.

The problem setting calculation unit 222 uses a solution search function to solve an optimal control problem as a solution search problem for the search point ξ _i under the system model, constraint conditions, and low-level controller indicated in the problem setting information set by the system model setting unit 221. The problem setting calculation unit 222 uses the solution search function to determine the minimum evaluation value g i * as the optimal value g _i ^* , and calculates the control parameter α _i ^* that gives the optimal value g _i ^* . The problem setting calculation unit 222 outputs optimal control solution information including a set of the optimal value g _i ^* and the control parameter α _i ^* for the search point ξ _i to the data update unit 223.

The data update unit 223 updates the acquisition data set D _opt so as to include a set of the search point ξ _i , the optimal value g _i ^* , and the control parameter α _i ^* indicated in the optimal control solution information input from the problem setting calculation unit 222 .

The level set function learning unit 231 of the learning setting unit 230 refers to the acquisition data set D _opt updated by the data updating unit 223 as training data, and learns the level set function g^ indicated in the preset level set function information. The level set function learning unit 231 sets the level set function g^ obtained by learning in the prediction accuracy evaluation function setting unit 232 and the controller learning evaluation function setting unit 234.

The prediction accuracy evaluation function setting unit 232 refers to the acquired data set D _opt and determines the prediction accuracy evaluation function J _g^ based on the level set function g^ set by the level set function learning unit 231. The prediction accuracy evaluation function setting unit 232 sets the evaluation value of the determined prediction accuracy evaluation function J _g^ in the prediction accuracy evaluation unit 233.

The prediction accuracy evaluation unit 233 uses the evaluation value of the prediction accuracy evaluation function Jg^ set by the prediction accuracy evaluation function setting unit 232 to determine whether or not it is necessary to continue learning the level set function g^ during learning. If it determines that learning needs to continue, the prediction accuracy evaluation unit 233 outputs a learning continuation flag indicating that learning needs to continue to the search point set setting unit 210.

The controller learning evaluation function setting unit 234 determines a controller learning evaluation function Jh based on the evaluation value of the level set function g^ set by the level set function learning unit 231 and the high-level controller set by the high-level controller learning unit 240. The controller learning evaluation function setting unit 234 sets the determined controller learning evaluation _{function Jh} in the high-level controller learning unit 240.

The high-level controller learning unit 240 refers to the acquisition data set D _opt as training data and uses the controller learning evaluation function J _h to learn the high-level controller π _H. The high-level controller learning unit 240 sets the high-level controller π _H obtained by learning in the problem setting calculation unit 222 and the controller learning evaluation function setting unit 234.

(7) Processing Flow Next, an example of the learning process by the skill learning unit 15 according to this embodiment will be described. Fig. 12 is a flowchart illustrating the learning process according to this embodiment.
(Step S101) The search point set initialization unit 211 of the search point set setting unit 210 sets the search point set Ξ _check using the target parameter information stored in the storage device 2. Then, the process proceeds to loop L11, where data acquisition/learning processing is started.

(Step S102) The search point selection unit 212 selects a search point ξ _i for acquiring data next from the search point set Ξ _check .
(Step S103) The system model setting unit 221 sets problem setting information including system model information in the problem setting calculation unit 222 based on the selected search point ξ _i .
(Step S104) The problem setting calculation unit 222 solves the optimal control problem based on the problem setting information and the high-level controller, and calculates the optimal value g _i ^* of the evaluation value with the level set function as the evaluation function, and the control parameter α _i ^* that gives the optimal value g _i ^* .
(Step S105) The data update unit 223 updates the acquisition data set D _opt by adding a set of search points ξ _i , optimal values g _i ^* , and control parameters α _i ^* to be processed.

(Step S106) The level set function learning unit 231 learns the level set function g^ by using the acquisition data set D _opt .
(Step S107) The controller learning evaluation function setting unit 234 determines a controller learning evaluation function J _h based on the level set function g^ and the high-level controller π _H , and sets it in the high-level controller learning unit 240.
(Step S108) The high-level controller learning unit 240 refers to the acquisition data set D _opt and learns the high-level controller π _H by using the controller learning evaluation function J _h . The high-level controller learning unit 240 sets the learned high-level controller π _H in the problem setting calculation unit 222 and the controller learning evaluation function setting unit 234.

(Step S109) The prediction accuracy evaluation function setting unit 232 determines a prediction accuracy evaluation function J _g^ for evaluating the prediction accuracy of the level set function g^ and the high-level controller π _H based on the level set function g^. The prediction accuracy evaluation function setting unit 232 sets the evaluation value of the determined prediction accuracy evaluation function J _g^ in the prediction accuracy evaluation unit 233 and the search point set setting unit 210. The evaluation value of the prediction accuracy evaluation function J _g^ can be referred to in the processing of step S102 related to the next search point ξ _i+1 .

(Step S110) The prediction accuracy evaluation unit 233 uses the evaluation value of the prediction accuracy evaluation function J _^g and predetermined learning conditions to determine whether or not it is necessary to continue learning the level set function g^ that is being learned.
(Step S111) When it is determined that learning needs to be continued (step S111: YES), the prediction accuracy evaluation unit 233 outputs a learning continuation flag to the search point set setting unit 210. By proceeding to the process of step S102, the process of loop L11 is repeated.
The prediction accuracy evaluation unit 233 considers the search point ξ _i at that point to be processed, and increments the number of processed search points by 1. If the number of processed search points has not reached N, the process of loop L11 is repeated for the next search point. If the number of processed search points reaches N, the process exits from loop L11 and ends the data acquisition/learning process.
When it is determined that learning should not be continued (step S111 NO), the process immediately leaves the loop L11 and ends the data acquisition/learning process. After that, the process in FIG. 12 ends. The level set function learning unit 231 sets the level set function g^ in the motion planning unit 34 of the robot controller 3. The high-level controller learning unit 240 sets the parameters of the high-level controller π _H in the motion planning unit 34 and the high-level control unit 35.

(8) Motion Plan The motion planner 34 according to this embodiment calculates, as an evaluation value g^*, an evaluation value g^(ξ, π _H (ξ)) of the learned level set function g^ set for the search point ξ and the control parameter π _H (ξ) calculated based on the search point ξ in the motion plan of the robot ^5. The control parameter π _H (ξ) is obtained as an output from the learned high-level controller π _H for the search point ξ. In general, the search point ξ related to the newly set motion plan may differ from the search point ξ _i used in learning the level set function g^, etc., and therefore the possibility of reaching the target state is unknown. The motion planner 34 determines the possibility of reaching the target state based on the calculated evaluation value. As described above, the search point ξ is a parameter set including the initial state x _s of the task to be executed and the target state/known task parameter value β _g as elements. The motion planner 34 can determine whether the target state can be reached based on whether the evaluation value g^ ^* is equal to or less than 0. The motion planning unit 34 may further determine whether or not a control parameter α (=π _H (ξ)) obtained using the high-level controller π _H based on the search point ξ satisfies a predetermined constraint condition, and may determine that the target state can be reached if it is determined that the evaluation value g^ ^* is less than or equal to 0 and satisfies the constraint condition.

When the motion planning unit 34 determines that the target state can be reached, it outputs the control parameter π _H (ξ) obtained based on the search point ξ as the control parameter α to the low-level control unit 36. Thus, the motion of the robot 5 is controlled based on the search point ξ determined to be capable of reaching the target state.

When determining that the target state has not been reached, the motion planning unit 34 defines the sum π _H (ξ)+Δα obtained by adding the adjustment amount Δα to the control parameter π _H (ξ) as the adjusted control parameter α', and calculates the evaluation value g^(ξ,α') of the level set function g^ for the search point ξ and the adjusted control parameter α' as the evaluation value g^ ^* . As exemplified in equation (9), the motion planning unit 34 searches for the control parameter α' for which the evaluation value g^ ^* is 0 or less as the control parameter that enables the target state to be reached.

The action planning unit 34 may further use the control parameter α' to determine whether or not it satisfies a predetermined constraint condition, determine whether or not to adopt the control parameter α' obtained by the search, and if it is determined not to adopt the control parameter α', reject the obtained control parameter α', set the evaluation value g^ ^* to 0 or less, and newly search for a new control parameter α' that satisfies the constraint condition.
The motion planning unit 34 outputs the obtained control parameter α' to the low-level control unit 36. Therefore, even if it is determined that the search point ξ cannot reach the target state, the system state can be made to reach the target state by adjusting the control parameter that is the optimal solution for the search point ξ.

The search point ξ does not give the target state, nor is the control parameter α' an optimal solution for the search point ξ, but a control parameter α' that makes it possible to reach the target state is searched for. For example, assume that in the operating environment of the robot 5, another moving object enters the planned path of travel estimated by the control parameter that is the optimal solution at a certain search point ξ. Under this assumption, the motion planning unit 34 updates the search point ξ to a position sufficiently away from the predicted path of the other moving object, which is an element of the target state.

Then, the motion planning unit 34 executes the above processing for the set search point ξ. That is, the motion planning unit 34 judges whether the target state can be reached for the set search point ξ and the control parameter α that gives the optimal solution. When the motion planning unit 34 judges that the target state can be reached, it derives control parameters that give the optimal solution for the search point ξ. When the motion planning unit 34 judges that the target state cannot be reached, it adjusts the control parameters that give the optimal solution and searches for control parameters that make it possible to reach the target state. Then, after the moving object has passed the planned progression path based on the initial search point ξ, the motion planning unit 34 may reset the search point ξ including the original target state and execute the above processing for the reset search point ξ. Therefore, by changing the search point ξ, contact or collision with the moving object can be avoided, and the possibility of reaching the target state can be improved.

Next, an example of an operation plan according to this embodiment will be described below. Fig. 13 is a flowchart illustrating an operation plan according to this embodiment.
(Step S201) The motion planning unit 34 sets a search point ξ when planning the motion of the robot 5. As described above, the search point ξ indicates the initial state, the target state, and the task parameters of the control system.
(Step S202) The motion planning unit 34 uses a trained high-level controller π _H based on the search point ξ to obtain a control output π _H (ξ) from the high-level controller π _H as a parameter α. The motion planning unit 34 calculates an evaluation value g ^* of a level set function g^(ξ, π(ξ)) for the search point ξ and the control output π _H (ξ).

(Step S203) The motion planning unit 34 judges whether or not the evaluation value g ^* is equal to or less than 0 and the motion plan related to the search point ξ satisfies a predetermined constraint condition. If it is judged that the constraint condition is satisfied (YES in step S203), the process proceeds to step S204. If it is judged that the constraint condition is not satisfied (NO in step S203), the process proceeds to step S204.

(Step S204) The motion planning unit 34 outputs the control parameter π _H (ξ) as the control parameter α to the low-level control unit 36. Therefore, the motion planning unit 34 can control the robot 5 using the control parameter α output from the high-level controller as it is.
(Step S205) The motion planning unit 34 searches for an adjustment value Δα such that the level set function g^(ξ, π _H (ξ)+Δα) is less than or equal to 0 and satisfies the constraint condition even when the adjusted control parameter π _H (ξ)+Δα is used.
(Step S206) The motion planning unit 34 outputs the adjusted control parameter π _H (ξ)+Δα obtained by the search as the control parameter α' to the low-level control unit 36. As a result, the motion planning unit 34 controls the robot 5 using the adjusted control parameter α'. Then, the processing in FIG. 13 ends.

Second Embodiment
Next, a second embodiment of the present invention will be described, focusing mainly on the differences from the first embodiment. Common reference numerals are used for common features with the first embodiment, and the description thereof will be used unless otherwise specified.
FIG. 14 is a data flow diagram illustrating a data flow in the skill learning unit 15 according to this embodiment.
The data acquisition section 220 according to this embodiment includes, instead of the data update section 223, a first data update section 223-1 and a second data update section 223-2.

The problem setting calculation unit 222 solves the optimal control problem using the level set function as the evaluation function as described above, determines the minimum evaluation value as the optimal value g _i ^* , and calculates the control parameter α _i ^{* that gives the optimal value g i *} _. ^However , the problem setting calculation unit 222 according to this embodiment saves a set of one or more evaluation values g _i (sometimes referred to as non-optimal solutions in this application) calculated in the calculation process until the optimal value g _i ^* is obtained by completing the calculation process related to the optimal control problem, and the control parameter α _i that gives each evaluation value g _i . The problem setting calculation unit 222 outputs optimal control solution information including a set of the search point ξ _i , the optimal value g _i ^* , and the control parameter α _i ^* to the first data update unit 223-1 and the second data update unit 223-2. The problem setting calculation unit 222 outputs non-optimum control solution information including the search point ξ _i , the non-optimum value g _i obtained in the calculation process, and a set of the control parameter α _i to the first data update unit 223-1. Therefore, both the optimal control solution information and the non-optimum control solution information are provided to the first data update unit 223-1. In contrast, the second data update unit 223-2 is provided with the optimal control solution information, but not with the non-optimum control solution information.

The first data update unit 223-1 accumulates a set of search points ξ _i , optimal values g _i ^*, and control parameters α _i ^* indicated in the optimal control solution information input from the problem setting calculation unit 222, and a set of search points ξ _i , non-optimal values g _i , and control parameters α _i indicated in the non-optimal control solution information, to form a first data set D _g . The first data set D _g differs from the above-mentioned acquired data set D _opt in that it further includes a set of search points ξ _i , non-optimal values g _i , and control parameters α _i . The first data set D _g is used in place of the acquired data set D _opt for learning the level set function g^ in the level set function learning unit 231 and for setting the prediction accuracy evaluation function J _g^ in the prediction accuracy evaluation function setting unit 232.
Since the level set function g^ can be learned by associating it with a larger number of sets of search points ξ _i and control parameters α _i , it is possible to obtain a level set function g that can more accurately explain the dependency on the control parameter α, which is the explanatory variable.

The second data update unit 223-2 accumulates a set of the search point ξ _i , the optimal value g _i ^* , and the control parameter α _i ^* indicated in the optimal control solution information input from the problem setting calculation unit 222, and forms a second data set D _h . The second data set D _h is similar to the above-mentioned acquired data set D _opt . The second data set D _h is used for learning the high-level controller π _H in place of the acquired data set D _opt .

Next, an example of the learning process by the skill learning unit 15 according to this embodiment will be described. FIG. 15 is a flowchart illustrating the learning process according to this embodiment. The process in FIG. 15 includes steps S101 to S103, S114 to S117, S107, S118, and S109 to S111. The explanation of FIG. 12 is used for the processes of steps S101 to S103, S107, and S109 to S111. In the process in FIG. 15, after the process of step S103, the process proceeds to step S114.

(Step S114) The problem setting calculation unit 222 solves the optimal control problem based on the problem setting information and the high-level controller, and calculates an optimal value g _i ^* of the evaluation value with the level set function as the evaluation function and a control parameter α _i ^* that gives the optimal value g _i ^* . The problem setting calculation unit 222 saves a pair of a non-optimal solution g _i calculated in the calculation process (during calculation) until the optimal value g _i ^* is obtained, and a control parameter α _i that gives the non-optimal solution g _i .

(Step S115) The first data update unit 223-1 updates the first data set D g by adding a set of a search point ξ _i , an optimal value g _i ^* , and a control parameter α _i ^* , and a set of a search point ξ _i , a non-optimal value _{g i} , and a control parameter α _i .
(Step S116) The second data update unit 223-2 updates the second data set D _h by adding a set of the search point ξ _i , the optimal value g _i ^* , and the control parameter α _i ^* .

(Step S117) The level set function learning unit 231 learns the level set function g^ by using the first data set D _g .
After that, the process proceeds to step S107. After the process of step S107, the process proceeds to step S118.
(Step S118) The high-level controller learning unit 240 refers to the second data set _Dh and learns the high-level controller _πH using the controller learning evaluation function _Jh . The high-level controller learning unit 240 sets the learned high-level controller _πH in the problem setting calculation unit 222 and the controller learning evaluation function setting unit 234. Then, the process proceeds to step S109.

In addition, experience replay (ER) may be applied in updating one or both of the first data set D _g to the first data update unit 223-1 and the second data set D _h to the second data update unit 223-2, and in learning the level set function g or the high-level controller π _H. Experience replay is a method in which multiple sets of transition information are stored in a storage area called a replay buffer (RB), one set is randomly selected from the stored transition information, and the selected sets of transition information are sequentially used for learning. One set of transition information includes information such as the state before and after a transition in one state transition, the condition of the transition, and the amount of change in the evaluation value of the evaluation function due to the transition. In generating the first data set D _g , a data augmentation technique such as post-event experience replay (Hindsight Experience Replay) may be applied using a non-optimal solution. Post-event experience replay is described in, for example, the following document.
M. Andrychowicz, et al.: “Hindsight Experience Replay”, Proc of NIPS (2017)

In the above example, the second data set _Dh does not include a set of the search point ξ _i , the non-optimal value g _{i ,} and the control parameter α _i , but this is not limiting. The second data set _Dh may include a set of the search point ξ _i , the non-optimal value g _i , and the control parameter α _i . However, the set of the search point ξ _i , the non-optimal value g _i , and the control parameter α _i included in the second data set _Dh is considered to be a part of the set of the search point ξ _i , the non-optimal value g _{i ,} and the control parameter α _i included in the first data set _Dg .

Third Embodiment
Next, the third embodiment of the present invention will be described, focusing mainly on the differences from the first embodiment. The same reference numerals are used for the common points with the first and second embodiments, and the description thereof is incorporated herein unless otherwise specified.
16 is a data flow diagram illustrating a data flow in the skill learning unit 15 according to this embodiment. The skill learning unit 15 according to this embodiment includes a system model learning unit 250.

When controlling the operation of the robot 5 in the target system using a high-level controller that has been learned or is currently being learned, the system model learning unit 250 acquires data indicating the control result (referred to as "control result data" in this application). In the control, the high-level controller uses a search point that includes, as an element, target parameter information related to the skill to be controlled. The control result data includes at least the search point at each time, the control parameters output from the high-level controller, and the system state that forms the control result, and these are associated with each other. The control result data is associated with the target parameter information related to the skill to be controlled. In learning the system model, the system model learning unit 250 can determine the parameters of the system model of the target system that outputs the system state at each time and inputs the control parameters and the target parameters indicated by the search point based on the control result data. The system model learning unit 250 sets the system model information indicating the parameters of the learned system model in the system model setting unit 221. The system model learning unit 250 outputs the acquired control result data to the data update unit 223.

The system model learning unit 250 may acquire an evaluation value of the level set function at each time from the system 50 as a part of the control result, and may include the acquired evaluation value in the control result data in association with the control parameters, search points, and system state at the time. The evaluation value of the level set function is calculated in the process of solving an optimal control problem in the motion control of the robot 5. The data updating unit 223 may include control solution information including the search points and control parameters included in the control result data as input and the evaluation value as output in the acquisition data set D _opt . In this case, the control solution information derived from the control result data is used for learning the level set function in the level set function learning unit 231 and the high-level controller in the high-level controller learning unit 240.

Next, a description will be given of an example of the functional configuration of the system model learning unit 250 according to this embodiment. Fig. 17 is a schematic block diagram showing an example of the functional configuration of the system model learning unit 250 according to this embodiment.
The system model learning unit 250 includes a high-level controller evaluation unit 250a, a low-level controller for skill learning 250b, a data processing unit 250c, a data collection task management unit 250e, a transition data storage unit 250f, and a system model learning processing unit 250g.

The high-level controller parameters learned by the high-level controller learning unit 240 are set in the high-level controller evaluation unit 250a. Using the learned high-level controller, the high-level controller evaluation unit 250a calculates control parameters based on search points derived from target parameter information set by the data collection task management unit 250e. The high-level controller evaluation unit 250a outputs the calculated control parameters to the skill learning low-level controller 250b.

The low-level controller 250b for skill learning calculates a control output based on the system state collected from the system 50, the control parameters input from the high-level controller evaluation unit 250a, and the target state/known task parameter values indicated in the target parameter information, and outputs the calculated control output to the system 50 to be controlled.
The data processing unit 250c configures control result data by associating the control parameters used in controlling the system 50, the control points acquired from the data collection task management unit 250e, the evaluation value of the level set function, and the system state acquired from the system 50. The data processing unit 250c stores the configured control result data in a control result storage unit 250d and a transition data storage unit 250f.

The control result storage unit 250d aggregates the control result data for each task. The control result data aggregated and stored in the control result storage unit 250d is read by the data update unit 223. The control result storage unit 250d may output the acquired control result data to the data update unit 223 every time new control result data is acquired.
The data collection task management unit 250e accumulates control result data for each time, and the accumulated control result data is accumulated to form transition data for each skill.

The system model learning processing unit 250g uses the transition data stored in the transition data storage unit 250f to learn the system model indicated by the preset system model information. The system model learning processing unit 250g receives the control parameters indicated in the transition data and the target parameter information indicated at the search points as input, and determines the system model parameters for each skill to estimate the system state at each time of the target system as output. The system model learning processing unit 250g sets the system model information indicating the learned system model in the system model setting unit 221.

The system model learning processing unit 250g can use a system model exemplified by equation (5). By setting a very small unit time in this system model, the time derivative of the system state exemplified on the left side of equation (5) can be approximated to the amount of change from the system state at the current time to the system state at the time unit time after the current time (i.e., the next time). Therefore, the system model learning processing unit 250g can use, as the system model, a function that outputs the system state at the next time and inputs the system state at the current time, the control parameters, and the target parameter information.

Furthermore, the learning of the system model by the system model learning unit 250 may be performed separately from the learning of the level set function and the high-level controller, or may be performed for each search point related to the learning. As described above, the search points include target parameter information as elements. In the learning of the system model, search points that provide target parameter information used in the operation of the system 50 may be used instead of search points that belong to a preset search point set Ξ _check .

Next, an example of the learning process by the learning device 1 according to this embodiment will be described. Fig. 18 is a flowchart illustrating the learning process according to this embodiment. The processes of steps S301 and S302 are executed independently of the learning of the level set function and the high-level controller. The processes of steps S304 and S305 are executed in association with the learning of the level set function and the high-level controller according to step S303 for each search point ξ _i .

(Step S301) The system model learning unit 250 randomly acquires control result data from the system 50 operating in a real environment. Randomly acquiring includes acquiring in a randomly determined period, randomly determining whether acquisition is necessary for each operation, and acquiring when it is determined that acquisition is necessary, etc. Either or both of the skill and the target parameter may be different for each operation.
(Step S302) The system model learning unit 250 learns the system model for each skill based on the acquired control result data. The system model learning unit 250 sets system model information indicating the parameters of the learned system model to the system model setting unit 221. The system model learning unit 250 outputs the control result data acquired in the learning process of the system model to the data update unit 223. Thereafter, the process proceeds to loop L31, and data acquisition/learning processing is started. However, although the number of search points is N in the example of FIG. 18, the processing in loop L31 may be executed for each operation of the system 50. At least one item of the initial state, task parameter, and target parameter information that constitute the search point may be different for each operation.

(Step S303) The skill learning unit 15 learns the level set function and the high-level controller. The processing of this step may be the same as the processing of steps S102 to S111 of the loop L11 illustrated in FIG. 12. However, when it is determined in step S111 that learning should not be continued (step S111 NO), the prediction accuracy evaluation unit 233 proceeds to the processing of step S303 without changing the number of search points that have been processed.

(Step S304) The system model learning unit 250 acquires control result data from the system 50 that controls the movement of the robot 5 in the real environment using the learned high-level controller.
(Step S305) The system model learning unit 250 learns the system model based on the acquired control result data. The system model learning unit 250 sets system model information indicating parameters of the learned system model in the system model setting unit 221. The system model learning unit 250 outputs the control result data acquired in the process of learning the system model to the data updating unit 223.
The prediction accuracy evaluation unit 233 considers the search point ξ _i at that point to be processed, and increments the number of processed search points by 1. When the number of processed search points has not reached N, the process of loop L31 is repeated for the next search point. When the number of processed search points reaches N, the process exits from loop L31 and ends the data acquisition/learning process. Then, the process of FIG. 18 ends.

The above description has focused on the differences from the first embodiment, but instead of the data update unit 223 as in the skill learning unit 15 according to the second embodiment, the skill learning unit 15 according to this embodiment may include a first data update unit 223-1 and a second data update unit 223-2. In this case, the first data set D _g generated by the first data update unit 223-1 is used for learning the level set function g^ in the level set function learning unit 231 and for setting the prediction accuracy evaluation function J _g^ in the prediction accuracy evaluation function setting unit 232. The second data set D _h generated by the second data update unit 223-2 is used for learning the high-level controller π _H. The process of step S303 illustrated in FIG. 18 is similar to the processes of steps S102, S103, S114 to S117, and S107 to S111 of the loop L21 illustrated in FIG. 15.

The control result data generated by the system model learning unit 250 can be used for learning the level set function g^ and for setting the prediction accuracy evaluation function J _g^ . In this case, the control result data generated by the system model learning unit 250 does not need to be used for learning the high-level controller π _H.

<Minimum configuration example>
Next, an example of a minimum configuration of the embodiment of the present application will be described.
19 is a schematic block diagram showing an example of a minimum configuration of the learning device 1 according to the embodiment of the present application. The learning device 1 includes a level set function learning unit 231 that learns a level set function that receives motion parameters that define a motion target and motion environment of the robot and control parameters of the robot as inputs and outputs an evaluation value related to the attainment of the motion target based on the motion parameters and the control parameters, and a high-level controller learning unit 240 that learns a high-level controller that determines control parameters for realizing a target motion based on the motion parameters of the robot based on the level set function and prediction accuracy of the control parameters.

With this configuration, the high-level controller learning unit 240 learns a high-level controller for determining control parameters for realizing a target motion based on the motion parameters used to control the robot, and the level set function learning unit 231 learns a level set function for determining an evaluation value relating to the achievability of a motion goal based on the motion parameters and the control parameters. Therefore, the achievability of a motion goal based on the motion parameters can be determined using an evaluation value obtained from the motion parameters related to the control of the robot using the learned level set function. This makes it possible to efficiently plan the motion of the robot using the motion parameters.

FIG. 20 is a schematic block diagram showing an example of the minimum configuration of a control device 3' according to an embodiment of the present application. The control device 3' includes a high-level control unit 35 that determines control parameters for realizing a target motion based on motion parameters that define the motion target and motion environment of the robot, and a motion planning unit 34 that calculates an evaluation value indicating the feasibility of the target motion based on the motion parameters and control parameters using a level set function, and uses the control parameters to control the motion of the robot when it is determined that the target motion is realizable based on the evaluation value, and searches for control parameters that make the target motion realizable based on the evaluation value when it is determined that the target motion is not realizable based on the evaluation value.

With this configuration, the high-level control unit 35 obtains control parameters based on the motion parameters, and the motion planning unit 34 evaluates the feasibility of the target motion using the motion parameters based on the motion parameters and the evaluation value obtained from the control parameters. Furthermore, when it is determined that the target motion can be realized, the control parameters obtained by the high-level control unit 35 are used for motion control, and when it is determined that the target motion cannot be realized, a search is made for control parameters that will make the target motion feasible based on the evaluation value. This makes it possible to promote motion planning so that the target motion using the motion parameters can be realized as much as possible.

As described above, an embodiment of the present application may be realized as a control system 100 including a learning device 1, a robot 5 and a high-level controller.
The embodiment of the present application may be realized as a program for causing a computer to function as the learning device 1, or as a non-transitory storage medium that stores the program and is readable by the computer. The computer may include a processor or other integrated circuit and the non-transitory storage medium, and may be capable of executing the processes instructed by the instructions constituting the program, thereby realizing the functions of the learning device 1.

The learning device 1 also includes a search point selection unit 212 that selects one search point from a search point set including a predetermined number of search points based on a prediction accuracy evaluation function that indicates the prediction accuracy of the level set function, and the search point is a pair of an operating parameter and a control parameter.
With this configuration, the prediction accuracy of the level set function is evaluated for a known set of motion parameters and control parameters, and motion parameters and control parameters used in training the level set function are selected based on the evaluated prediction accuracy. By preferentially selecting motion parameters and control parameters that have low prediction accuracy and therefore have a high convergence speed in the training process, the training of the level set function can be made more efficient.

The learning device 1 may also include a prediction accuracy evaluation unit 233 that uses the prediction accuracy evaluation function to determine the need for continued learning of the level set function and the high-level controller.
With this configuration, the need for continuing learning is quantitatively determined based on the prediction accuracy evaluated during the learning process.

Furthermore, the level set function learning unit 231 may learn the level set function using the first learning data, and the high-level controller learning unit 240 may learn the high-level controller learning unit 240 using the second learning data. The first learning data includes an optimal solution set that has as input operation parameters and control parameters that provide an optimal solution of an evaluation value and includes an optimal solution as an output, and a non-optimal solution set that has as input operation parameters and control parameters that provide a non-optimal solution different from the optimal solution and includes a non-optimal solution as an output, and the second learning data includes the optimal solution set.
According to this configuration, the level set function is further trained using first training data including a data set having a non-optimal solution as an output and an operation parameter and a control parameter that give the non-optimal solution as an input. A wider range of control parameters are referenced in training the level set function that regresses the solution of the optimal control problem. Thus, the high-level learner is controlled so that stable control parameters are output even when the system state deviates from the optimal state.

The learning device 1 may include a system model learning unit 250 that uses a system state obtained by controlling the operation of the robot using control parameters based on the operation parameters and control result data including a set of the control parameters to learn a system model that uses operation parameters and control parameters as inputs and outputs a system state.
According to this configuration, the system model is trained using the control result data so that the system state actually obtained for the operation parameters and the control parameters is estimated. By updating the control problem based on the trained system model, the high-level controller can be trained to adapt to the real system environment.

The high-level controller learning unit may learn the high-level controller using the operation parameter related to the control result data as an input and the control parameter as an output.
According to this configuration, a high-level controller is trained using the operational parameters actually used for control as input so that the control parameters actually obtained by the control can be obtained. Therefore, the high-level controller is trained to adapt to the actual system environment.

Note that the programs for executing all or part of the processing performed by the learning device 1, the robot controller 3, and the robot 5 may be recorded on a computer-readable recording medium, and the programs temporarily or non-temporarily recorded on this recording medium may be read into a computer system and executed to perform the processing of each part. Note that the term "computer system" here includes hardware such as the OS and peripheral devices.
Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc Read Only Memory), as well as storage devices such as hard disks built into computer systems. The above-mentioned program may be for realizing part of the above-mentioned functions, or may be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

The above describes the embodiment of the present application in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

Embodiments of the present application can be realized as a learning device, a control device, a control system, a learning method, and a storage medium.

1...Learning device, 2...Memory device, 3...Robot controller, 3'...Control device, 4...Measuring device, 5...Robot, 11...Processor, 12...Memory, 13...Interface, 14...Abstract system model setting unit, 15...Skill learning unit, 16...Skill tuple generation unit, 31...Processor, 32...Memory, 33...Interface, 50...System, 100...Control system, 210...Search point set setting unit, 211...Search point set initialization unit, 212...Search point selection unit, 220...Data acquisition unit, 221...System model setting unit, 222...Problem setting calculation unit, 223 (223-1, 223-2)...Data update unit, 230...Learning setting unit, 231...Level set function learning unit, 232...Prediction accuracy evaluation function setting unit, 233...Prediction accuracy evaluation unit, 234...Evaluation function setting unit for controller learning, 240...High-level controller learning unit, 250...System model learning unit

Claims (10)

a level set function learning unit that learns a level set function that receives as input motion parameters that define a motion goal and a motion environment of a robot and control parameters of the robot, and outputs an evaluation value regarding the attainment of the motion goal based on the motion parameters and the control parameters;
a high-level controller learning unit that learns a high-level controller that determines control parameters for realizing a target motion based on the motion parameters of the robot, based on the level set function and prediction accuracy of the control parameters. From a search point set including a plurality of predetermined search points,
a search point selection unit that selects one search point based on a prediction accuracy evaluation function that indicates the prediction accuracy of the level set function;
The learning device according to claim 1 , wherein the search point is a pair of the operation parameter and the control parameter. The learning device according to claim 2 , further comprising a prediction accuracy evaluation unit that uses the prediction accuracy evaluation function to determine the necessity for continuing learning of the level set function and the high-level controller. The level set function learning unit learns the level set function using first learning data;
The high-level controller learning unit learns the high-level controller learning unit using second learning data,
the first learning data includes an optimal solution set having the operation parameters and the control parameters which provide an optimal solution for the evaluation value as input and including the optimal solution as output, and a non-optimal solution set having the operation parameters and the control parameters which provide a non-optimal solution different from the optimal solution as input and including the non-optimal solution as output,
The learning device according to claim 2 , wherein the second learning data includes the optimal solution set. 2. The learning device according to claim 1, further comprising a system model learning unit configured to learn a system model having the operation parameters and the control parameters as inputs and the system state as an output, by using control result data including a set of a system state obtained by controlling the operation of the robot using the control parameters based on the operation parameters and the control parameters. The learning device according to claim 5, wherein the high-level controller learning unit learns the high-level controller using the operation parameters related to the control result data as input and the control parameters as output. The robot;
the high level controller;
A control system comprising: the learning device according to claim 1 . On the computer,
A computer-readable storage medium storing a program for causing the learning device according to claim 1 to function. A learning method for a learning device, comprising:
The learning device,
a level set function learning step of learning a level set function in which operation parameters defining an operation goal and an operation environment of a robot and control parameters of the robot are input, and an evaluation value regarding the possibility of reaching the operation goal based on the operation parameters and the control parameters is output;
a high-level controller learning step of learning a high-level controller that determines control parameters for realizing a target motion based on the motion parameters of the robot, based on the level set function and prediction accuracy of the control parameters. a high-level control unit that determines control parameters for realizing a target motion based on motion parameters that define a motion target and a motion environment of the robot;
Calculating an evaluation value indicating the feasibility of a target operation based on the operation parameters and the control parameters using a level set function;
When it is determined that the target motion is realizable based on the evaluation value, the control parameter is used for motion control of the robot;
a motion planning unit that searches for control parameters that enable the realization of the target motion based on the evaluation value when it is determined that the target motion is not realizable based on the evaluation value.

Images