WO2020235693A1 - Learning method, learning device, and learning program for ai agent that behaves like human - Google Patents

Abstract

Provided is a learning method for an agent that combines an optimum action exceeding human capability and a human-like behavior which are acquired by reinforcement learning. Play data by a human expert and state data SR and action data AR in the recording of play of the agent of an imitation learning or a reinforcement learning model are inputted. The state data SR is inputted to the learning executor of a blended model of the reinforcement and imitation learning models and the action data AF of the blended model is outputted. A first loss error between the action data AR of the human expert or the agent of the imitation learning model and the action data AF is calculated. A second loss error between the action data ARL outputted on the basis of the state data SR by the executor of the reinforcement learning model or an optimum action algorithm and the action data AF is calculated. A blending error is calculated on the basis of a weight ratio of the first and second loss errors, and the learning executor parameter of the blended model is updated on the basis of the blending error.

Description

AI agent learning methods, learning devices and learning programs that behave like humans

The present invention is a method, device, and program that can skillfully take into account the error with the reinforcement learning model and the error with the behavior of a human expert or the imitation learning model when training the agent AI (artificial intelligence). It is about.

In recent years, reinforcement learning (hereinafter sometimes referred to as "RL"), especially deep reinforcement learning (DRL), is AI (artificial intelligence) used in electronic games, automatic driving control of vehicles such as automobiles, It is applied to algorithms such as autonomous control of robots. Reinforcement learning is a method of repeating learning in units of episodes and adapting to the target environment using the rewards of trial and error in the environment, and learning is performed by optimizing the policy. Reinforcement learning uses the information processing power of a convolutional neural network (CNN) to perform reinforcement learning based on high-dimensional input such as image data.

In reinforcement learning and deep reinforcement learning, a problem is set in which an agent in the environment observes the current state and decides an action to be taken, and the agent receives a reward from the environment by selecting an action. While repeating the interaction between the environment and the agent, learning is performed using the three variables of reward, behavior, and state.
In reinforcement learning and deep reinforcement learning, agents act in an environment with a state space and an action space. At each time step, the policy converts the state into a feature vector with the feature extraction learner, and outputs the action probability for the feature vector with the value function calculation learner. After the agent takes action, the environment prints a scalar reward and the next state. The episode ends when you repeat the action until the final state.

Agent AI constructed by reinforcement learning and deep reinforcement learning can solve various problems by learning through trial and error while interacting with the environment, far exceeding human ability, so game AI, robots It is applied to algorithms such as autonomous control and automatic operation of the above (see, for example, Non-Patent Documents 1 to 5). However, since humans cannot predict the behavior, there is a big problem in terms of usability for the practical application of, for example, a robot that works in collaboration with humans and an automatic driving AI on which a human rides. In addition, agent AI constructed by reinforcement learning and deep reinforcement learning shows high performance because it is trained to maximize profits, but when it is put into practical use, consideration other than such performance indicators is also taken into consideration. There is a need. For example, in a video game, if an NPC (Non Player Character), which is a character that the player of the game cannot operate, is used as an agent AI for reinforcement learning, the agent AI may be too strong for the player to enjoy the game very much. There is. In addition, when applied to autonomous driving, the reinforcement learning agent AI trained for high performance may accelerate or decelerate violently or turn suddenly, causing anxiety to adjacent cars and pedestrians. There is. Therefore, it is necessary to design a human-like agent AI.

On the other hand, imitation learning (hereinafter sometimes referred to as “IL”) is adopted to build an agent AI that imitates human behavior, mainly for the purpose of reproducing the behavior of human experts. .. In imitation learning, it is assumed that the policy followed by the expert is the optimal policy, and learning is performed so that the policy of the agent AI approaches the behavior of the expert. In imitation learning, the optimal policy for a human expert is provided in the form of a series of state-behavior pairs, and then the agent AI policy is inferred from the observed state to the behavior that the expert is likely to take. Being trained, you can expect human-like behavior by imitating human experts. However, since the learning policy is limited to the provided data, there is a problem that the performance of the imitation learning agent AI hardly exceeds the performance of a human expert (for example, Non-Patent Documents 6 to 6). See 8.).

V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp.529, 2015. D. Silver et al., “Mastering the game of go without human knowledge,” Nature, vol. 550, no. 7676, p. 354, 2017. B. Vikas, “Deep reinforcement learning approach to automatic navigation,” 2017. T. P. Lillicrap et al., “Continuous control with deep reinforcement learning,” CoRR, vol. Abs / 1509.02971, 2015. D. Silver et al., “Deterministic policy gradient algorithms,” in Proceedings of the 31st International Conference on International Conference on Machine Learning, Volume 32, ser. ICML'14. JMLR.org, 2014, pp. -395. S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the fourteenth international 627 and stat-conference on artificial intelligence .. J. Ortega et al., “Imitating human playing styles in super mario bros,” Entertainment Computing, vol. 4, no. 2, pp. 93-104, 2013. J. Ho and S. Ermon, “Generative adversarial imitation learning,” CoRR, vol. Abs / 1606.03476, 2016.

In view of this situation, the present invention has achieved both highly efficient optimal behavior that exceeds the human ability acquired by reinforcement learning and human-like behavior acquired by imitating the behavior of a human expert. It is an object of the present invention to provide learning methods, devices and programs for agents.

In order to solve the above problems, the learning method of the agent of the present invention fuses a learning device that realizes a behavior in which an agent judges and takes an optimum action under a predetermined environment and a learning device that realizes a human-like behavior. It is a learning method that optimizes the behavior policy of an agent so as to take the optimum behavior like a human being, and includes the following steps a) to f).
a) Human play data expert or a predetermined input step of inputting status data S _R and behavioral data A _R in at least one of recording of the play data of agents created for the purpose.
b) for the learning execution unit of the fusion model of the imitation learning model involved in the reinforcement learning model and people seems to behavior related to the behavior to take the optimal action, enter the state data S _R output the action data A _F of the fusion model Learning steps to let.
c) The first loss error calculation step for calculating the first loss error between the behavior data _AR and the behavior data _AF .
and behavior data A _RL which is output based on the state data S _R by the execution unit or optimal behavior algorithm of d) reinforcement learning model, the second loss error calculating step of calculating a second loss error between behavior data A _F ..
e) Fusion error calculation step of calculating the fusion error based on the weight ratio of the first and second loss errors.
f) An update step that updates the parameters of the learning executor of the fusion model based on the fusion error.

According to the agent learning method of the present invention, the error between the reinforcement learning model and the behavior of a human expert or the imitation learning model is fed back to construct an agent AI that behaves like a human and takes optimal behavior. can do.
Here, the executor of the reinforcement learning model is set with a policy of observing the current state of the agent in the environment and deciding the action to be taken, as in the case of general reinforcement learning, and the agent has the value of the value function. It is an executor obtained by repeatedly selecting an action from a policy and obtaining a reward so as to increase the size, and learning while feeding back using three variables of reward, action, and state.

The environment is, for example, a set of image data of an electronic game screen or a time series of image data of a moving image taken by a camera mounted on a robot or a vehicle. The state data is data such as individual game images obtained by observing the environment, which is a set of image data, surrounding environment images of the own vehicle, and lane positions. Further, the action data is, for example, a game input device operation and a controller input (movement direction, jump, etc.) in the case of an electronic game, and a steering wheel, accelerator, brake operation, etc. in the case of automatic driving of an automobile.
In this way, reinforcement learning functions by a mechanism of learning while feeding back using three variables of reward, behavior, and state, whereas imitation learning functions by a mechanism using a teacher agent and a student agent.

Moreover, the behavior data _AF of the output of the fusion model is expressed by the probability distribution of the behavior. For example, the probability distribution of each input button operation of the controller, the probability distribution of the steering wheel, the accelerator, and the brake operation. Here, it is assumed that the probability distribution is included as a δ distribution even in the case of only one optimum operation output. More specifically, in the case of discrete behavior, it is expressed by a probability distribution, and in the case of continuous behavior, only one optimum one is output.
Further, the loss error means an error between the current output calculated by using a loss function and the expected output, for example.

In input step of learning how the agent of the present invention, the state data S _R and behavioral data A _R, the following 1) There are cases 1-5), as the case thereof, first and second loss error The processing of the calculation step is different.
1) When recording play data by a human expert In the first loss error calculation step, the error between the behavior data _AR and the behavior data _AF in the play data by a human expert is calculated using a loss function. To do.
Second loss error calculating step includes a behavior data software targeted behavioral data A _RL which are outputted by inputting the state data S _R in reinforcement learning model by knowledge distillation, the error between the behavior data A _F, loss Calculate using a function.

2) When the play data of the agent of the imitation learning model is recorded The first loss error calculation step is the error between the behavior data _AR and the behavior data _AF in the play data of the agent of the imitation learning model, or the imitation learning. The error between the behavior data A _IL soft-targeted by knowledge distillation and the behavior data A _F , which is output by inputting the state data S _R into the model, is calculated using a loss function.
Second loss error calculating step includes a behavior data software targeted behavioral data A _RL which are outputted by inputting the state data S _R in reinforcement learning model by knowledge distillation, the error between the behavior data A _F, loss Calculate using a function.

3) The first loss error calculating step when a record of the play data of the agent of reinforcement learning models, software targeted by Knowledge distillation was enter to output the state data S _R to the imitation learning model behavior data A _IL The error between the behavior data and the behavior data _AF is calculated using the loss function.
Second loss error calculating step includes a behavior data software targeted behavioral data A _RL which are outputted by inputting the state data S _R in reinforcement learning model by knowledge distillation, the error between the behavior data A _F, loss Calculate using a function.

4) When the state data S _HE and the behavior data A _HE in the recording of the play data by the human expert and the state data S _RL and the behavior data A _RL in the recording of the play data of the agent of the reinforcement learning model, the learning step is The state data _SHE is input to output the action data A _F1 , and the state data _SRL is input to output the action data A _F2 .
In the first loss error calculation step, the error between the behavior data A _HE and the behavior data A _F1 in the play data by a human expert is calculated using the loss function.
In the second loss error calculation step, the error between the behavior data A _RL soft-targeted by knowledge distillation and the behavior data A _F2 , which is output by inputting the state data S _RL into the reinforcement learning model, is lost. Calculate using a function.

5) imitation learning model agents play data state data S _IL and behavioral data A _IL in the _recording, and, when the learning step is a state data S _RL and behavioral data A _RL in the recording play data Agent Reinforcement Learning Inputs the state data _SIL to output the action data A _F1 , and inputs the state data _SRL to output the action data A _F2 .
In the first loss error calculation step, the error between the behavior data _AIL soft-targeted by knowledge distillation in the play data of the agent of the imitation learning model and the behavior data A _F1 is calculated using the loss function. ..
In the second loss error calculation step, the error between the behavior data A _RL soft-targeted by knowledge distillation and the behavior data A _F2 , which is output by inputting the state data S _RL into the reinforcement learning model, is lost. Calculate using a function.

In the soft targeting by knowledge distillation in the agent learning method of the present invention, the output of the agent's action policy has a parameter of heat degree T, and corresponds to hard targeting and soft targeting. When the heat degree T is "0", the output of the action policy becomes a hard target in which only one action has a probability greater than 0, and the action is uniquely determined. On the other hand, when the heat degree T is not "0", the plurality of actions have a probability greater than 0, and the actions are stochastically determined.

Next, the learning device of the agent of the present invention will be described.
The agent learning device of the present invention fuses a learning device that realizes a behavior in which an agent judges and takes an optimum action under a predetermined environment and a learning device that realizes a human-like behavior, and makes the agent's behavior policy human-like. It is a learning device that optimizes to behave and is equipped with the following A) to F).
An input unit for inputting status data S _R and behavioral data A _R in at least one of recording of the play data of agents created A) human experts play data, or a predetermined purpose.
Against fusion model and imitation learning model related to reinforcement learning model and human seems behaviors related to behavior taking B) optimal action, learning execution outputting a behavior data A _F of the fused model to input state data S _R ..
C) A first loss error calculator for calculating a first loss error between the behavior data _AR and the behavior data _AF .
And behavior data A _RL which is output based on the state data S _R by the execution unit or optimal behavior algorithm D) reinforcement learning model, the second loss error calculator for calculating a second loss error between behavior data A _F ..
E) A fusion error calculator that calculates the fusion error based on the weight ratio of the first and second loss errors.
F) An update unit that updates the parameters of the learning executor of the fusion model based on the fusion error.

At the input of the agent of the learning device of the present invention, similarly to the input step of learning how the agent of the present invention described above, according to the state data S _R and behavioral data A _R in the case of the above 1) to 5) , The calculation process of the first and second loss error calculators is different.
Further, in the soft targeting by knowledge distillation in the learning device of the agent of the present invention, the output of the action policy of the agent has a parameter of heat degree T and is hard-targeted as in the above-described learning method of the agent of the present invention. When the degree of heat T is "0", the output of the action policy becomes a hard target with a probability that only one action has a probability greater than 0, and the action is uniquely determined, while the action is uniquely determined. If the heat degree T is not "0", the plurality of actions have a probability greater than 0, and the actions are determined probabilistically.

The agent learning program of the present invention is a program for causing a computer to execute all the steps in the above-mentioned agent learning method of the present invention.
Further, the learning program of the agent of the present invention includes an input unit, a learning executor, a first loss error calculator, a second loss error calculator, a fusion error calculator and an update unit in the above-mentioned agent learning device of the present invention. It is a program for operating a computer.

According to the present invention, an agent AI that balances highly efficient optimal behavior that exceeds human ability acquired by reinforcement learning and human-like behavior acquired by imitating the behavior of a human expert is constructed. can do.

Functional block diagram of the learning device of the agent of the present invention Functional block diagram of the learning device of the agent of the first embodiment Schematic flow chart of the learning method of the agent of Example 1 Execution processing flow diagram of the fusion model of the first embodiment Calculation processing flow chart of loss error function _LHE with human behavior of Example 1 Calculation processing flow chart of the loss error function _LRL with the optimum behavior of the first embodiment Calculation processing flow chart of fusion loss error function L _Mix of Example 1 Learning processing flow diagram of the fusion model of Example 1 Functional block diagram of the learning device of the agent of the second embodiment Calculating process flow diagram of a loss error function L _IL with human likely behavioral Example 2 Functional block diagram of the learning device of the agent of the third embodiment Functional block diagram of the learning device of the agent of the fourth embodiment Functional block diagram of the learning device of the agent of the fifth embodiment Schematic flow chart of the learning method of the agent of Example 5 Execution processing flow of the fusion model of Example 5 FIG. Execution processing flow of the fusion model of Example 5 FIG. Calculation processing flow chart of loss error function _LHE with human behavior of Example 5 Calculation processing flow chart of loss error function _LRL with optimum behavior of Example 5 Functional block diagram of the learning device of the agent of the sixth embodiment Calculation processing flow chart of loss error function _LIL of Example 3 Explanatory drawing of soft targeting processor

Hereinafter, an example of the embodiment of the present invention will be described in detail with reference to the drawings. The scope of the present invention is not limited to the following examples and illustrated examples, and many modifications and modifications can be made.

FIG. 1 shows a functional block diagram of the learning device of the agent of the present invention. The learning device 1 is composed of a fusion model learning executor 2, a loss function calculator 3 with human-like behavior, a loss function calculator 4 with optimal behavior, a fusion model loss function calculator 5, and a database (DB) 6. Will be done. The fusion model learning executor 2 acquires state data from the database 6, performs execution processing of the fusion model, and outputs the obtained behavior data to the loss function calculators (3, 4). The loss function calculators (3, 4) input state data or behavior data from the database 6, and take in the input data and the behavior data input from the fusion model learning executor 2 into the loss error function calculator, respectively, and the loss error. The function L _HL and the loss error function L _RL are calculated respectively. The fusion model loss function calculator 5 inputs the loss error L _HL and the loss error L _RL calculated by the loss function calculator (3, 4), and the fusion loss error of the loss error with the behavior more efficient and human-like than human beings. Calculate the function L _Mix . The fusion model learning executor 2 acquires the fusion loss error function L _Mix from the fusion model loss function calculator 5 and performs learning processing of the fusion model.
In the following examples, the functional block diagram of the learning device of the agent of the present invention will be described by dividing into a combination pattern of a fusion model type, a play data type used for learning, and a learning control type. The combination patterns are summarized in Table 1.

FIG. 2 shows a functional block diagram of the learning device of the agent of the first embodiment. As shown in FIG. 2, the learning device 11 includes a fusion model learning executor 2, a loss function calculator 3 with human-like behavior, a loss function calculator 4 with optimal behavior, a fusion model loss function calculator 5, and a database 61. Consists of. Play data by a human expert is stored in the database 61. The loss function calculator 3 is provided with a first loss error calculator 30, and the loss function calculator 4 includes a reinforcement learning model executor 8, a soft targeting processor 9, and a second loss error calculator 40. Is provided.

An agent learning method using the learning device 11 will be described with reference to FIGS. 3 to 8. FIG. 3 shows a schematic flow chart of the learning method of the agent of the first embodiment. As shown in FIG. 3, first, the fusion model learning executor 2 performs the execution processing of the fusion model (step S11). Next, the loss function calculator 3 performs a calculation process of the loss error function _LHE with human behavior (step S12). In addition to this, the calculation process of the loss error function _LRL with the optimum behavior is performed (step S13). Based on the loss error function L _HE with human-like behavior and the loss error function L _RL with optimal behavior, the fusion model loss function calculator 5 is used to calculate the fusion loss error function L _Mix (step S14). .. In the fusion model learning executor 2, the calculated fusion loss error function L _Mix is acquired, and the fusion model learning process is performed (step S15). When the fusion loss error function L _Mix is equal to or greater than a predetermined value (step S16), the fusion model learning executor 2 again executes the fusion model execution process (step S11).

Next, each process shown in FIG. 3 will be described. FIG. 4 shows an execution processing flow diagram of the fusion model of the first embodiment. As shown in FIG. 4, a fusion model learning execution unit 2 acquires the status data _{S R} from the database 61 (step S111). To fusion model π type the state data _{S R} to run (step S112). The behavior data _AF of the fusion model is stored (step S113).

FIG. 5 shows a calculation processing flow diagram of the loss error function _LHE with the human-like behavior of Example 1. As shown in FIG. 5, the first loss error calculator 30 in the loss function calculator 3 and likely human behavior acquires behavior data A _R of the human from the database 61 (step S121). Further, the loss function calculator 3 acquires the behavior data _AF of the fusion model (step S122). The first loss error calculator 30 calculates the loss error function _LHE from the acquired _AR and _AF (step S123).

FIG. 6 shows a calculation processing flow diagram of the loss error function _LRL with the optimum behavior of the first embodiment. As shown in FIG. 6, the loss function calculator 4 with optimal behavior, reinforcement learning model execution unit 8 acquires the status data S _R from the database 61 (step S131). Reinforcement enter the learning model [pi _RL to state data _{S R,} Run, and outputs the behavior data _{A R} (step S132). The action data _AR is input to and executed from the soft targeting processor 9, and the action data A _ST2 is output to the second loss error calculator 40 (step S133). Separately from this, the second loss error calculator 40 acquires the behavior data _AF of the fusion model (step S134). The second loss error calculator 40 calculates the loss error function _LRL from the acquired behavior data A _ST2 and behavior data _AF (step S135).

Here, the soft targeting processor 9 will be described with reference to FIG. In the soft targeting by knowledge distillation in the soft targeting processor 9, as described above, the output of the agent's action policy has a parameter of heat degree T, and corresponds to the hard targeting and the soft targeting. When the heat degree T = 0, the output of the action policy becomes a hard target in which only one action has a probability greater than 0, and the action is uniquely determined. On the other hand, when the heat degree T> 0 (when T is not 0), the plurality of actions have a probability greater than 0, and the actions are stochastically determined.
The case where the soft targeting processor is used as a loss function calculator with the optimum behavior will be described as an example. As shown in FIG. 21 (1), the reinforcement learning model executor inputs the state data, outputs the output data G of the executor to the soft targeting processor, and the soft targeting processor further outputs the output data L. _Is output to the loss error _LRL calculator. The loss error _LRL calculator calculates the loss error function by inputting the behavior data of the output of the fusion model and the output data of the soft targeting processor.

In this case, as shown in FIG. 21 (3), there are roughly three patterns of the soft targeting processor. First, in the processing in the soft targeting processor, the heat degree T is processed, and the loss error function is calculated for each of the data m and the data L separately for the cases of T = 0 and T> 0 (this is referred to as "pattern A"). Then, in the processing in the soft targeting processor, the heat degree T is processed and the loss error function is calculated for the data L where T> 0 (this is referred to as “pattern B”). Then, in the processing in the soft targeting processor, the heat degree T is processed, and the loss error function is calculated for the data G in which T = 1 (this is referred to as “pattern C1”), and T = 0. There is a pattern for calculating the loss error function for the data G (this is referred to as "pattern C2"). Specifically, in the soft targeting process by the heat degree T, the heat degree T is used as a parameter of the exponential function by using the processing function as shown in the graph shown in FIG. 21 (2). Table 2 summarizes the above patterns, data names, and data formats.

FIG. 7 shows a calculation processing flow diagram of the fusion loss error function L _Mix of the first embodiment. As shown in FIG. 7, the fusion model loss function calculator 5 acquires the loss error functions L _HE and L _RL (step S141). The fusion model loss function calculator 5 calculates the fusion loss error function L _Mix from the trade-off coefficients α, L _HE, and L _RL (step S142).

FIG. 8 shows a learning processing flow diagram of the fusion model of the first embodiment. As shown in FIG. 8, the fusion model learning executor 2 acquires the fusion loss error function L _Mix (step S151). The fusion model learning executor 2 changes the parameters of the fusion model so that the fusion loss error function L _Mix becomes small (step S152).

FIG. 9 shows a functional block diagram of the learning device of the agent of the second embodiment. As shown in FIG. 9, the learning device 12 includes a fusion model learning executor 2, a loss function calculator 3 with human-like behavior, a loss function calculator 4 with optimal behavior, a fusion model loss function calculator 5, and a database 62. Consists of. The database 62 stores play data by the imitation learning model π _IL provided in the imitation learning model executor 7. The loss function calculator 3 is provided with a first loss error calculator 30, and the loss function calculator 4 includes a reinforcement learning model executor 8, a soft targeting processor 9, and a second loss error calculator 40. Is provided.
The agent learning device 12 of the second embodiment uses the database 61 in which the play data by a human expert is stored in that the database used is the database 62 in which the play data by the imitation learning model is stored. It is different from the learning device 11 of 1. Therefore, in the learning method for the agent of Example 2, the process of calculating the loss error function L _IL with likely human behavior is different from the first embodiment.

Figure 10 shows a calculation process flow diagram of a loss error function L _IL with human likely behavioral Example 2. As shown in FIG. 10, the first loss error calculator 30 in the loss function calculator 3 and likely human behavior acquires behavior data A _IL imitation learning model from the database 62 (step S211). Further, the loss function calculator 3 acquires the behavior data _AF of the fusion model (step S212). The first loss error calculator 30 calculates the loss error function L _IL from the acquired A _IL and _AF (step S213).

FIG. 11 shows a functional block diagram of the learning device of the agent of the third embodiment. As shown in FIG. 11, the learning device 13 includes a fusion model learning executor 2, a loss function calculator 3 with human-like behavior, a loss function calculator 4 with optimal behavior, a fusion model loss function calculator 5, and a database 62. Consists of. The database 62 stores play data by the imitation learning model π _IL provided in the imitation learning model executor 7. The loss function calculator 4 is provided with a reinforcement learning model executor 8, a soft targeting processor 9, and a second loss error calculator 40.
In the agent learning device 13 of the third embodiment, unlike the second embodiment, the loss function calculator 3 includes not only the first loss error calculator 30, but also the imitation learning model executor 7 and the soft targeting processor. 9 is provided.

Figure 20 shows a calculation process flow diagram of a loss error function L _IL with people likely behavior of Example 3. As shown in FIG. 20, first, the loss function calculator 3 obtains the status data _{S R} from the database 62 (step S411). Imitation learning model mimics learning model π input state data _{S R} to _IL execution unit 7 executes, and outputs the behavior data _{A IL} (step S412). The action data _AIL is input to and executed in the soft targeting processor 9, and the action data A _ST1 is output (step S413). Separately from this, the first loss error calculator 30 acquires the behavior data _AF of the fusion model (step S414). First loss error calculator 30 calculates a than the loss error function _{L IL} and action data _{A ST1} acquired behavioral data _{A F} (step S415).

FIG. 12 shows a functional block diagram of the learning device of the agent of the fourth embodiment. As shown in FIG. 12, the learning device 14 includes a fusion model learning executor 2, a loss function calculator 3 with human-like behavior, a loss function calculator 4 with optimal behavior, a fusion model loss function calculator 5, and a database 63. Consists of. The database 62 stores play data by the reinforcement learning model π _RL provided in the reinforcement learning model executor 8. The loss function calculator 3 is provided with the imitation learning model executor 7, the soft targeting processor 9, and the first loss error calculator 30, as in the third embodiment. Further, the loss function calculator 4 is provided with a reinforcement learning model executor 8, a soft targeting processor 9, and a second loss error calculator 40.
In the agent learning device 14 of the fourth embodiment, the database 62 in which the play data by the imitation learning model is stored is used in that the database used is the database 63 in which the play data by the reinforcement learning model is stored. It is different from the learning device 13 of No. 3, but is the same in other respects.

FIG. 13 shows a functional block diagram of the learning device of the agent of the fifth embodiment. As shown in FIG. 13, the learning device 15 includes a fusion model learning executor 2, a loss function calculator 3 with human-like behavior, a loss function calculator 4 with optimal behavior, a fusion model loss function calculator 5, and a database ( It consists of 61, 63). The loss function calculator 3 is provided with a first loss error calculator 30, and the loss function calculator 4 is provided with a second loss error calculator 40.
The learning device 15 of the fifth embodiment has two types, a database 61 in which play data by a human expert is stored and a database 63 in which play data by the reinforcement learning model π _RL provided in the reinforcement learning model executor 8 is stored. It is different from Examples 1 to 4 in that it uses the database of.

Therefore, the learning method of the agent using the learning device 15 will be described with reference to FIGS. 14 to 18. FIG. 14 shows a schematic flow chart of the learning method of the agent of Example 5. As shown in FIG. 14, first, the fusion model learning executor 2 performs the execution process 1 of the fusion model (step S31). Separately from this, the fusion model learning executor 2 performs the execution process 2 of the fusion model (step S32). After the execution process 1 of the fusion model in step S31, the loss function calculator 3 performs a calculation process of the loss error function _LHE with human behavior (step S33). Further, after the execution process 2 of the fusion model in step S32, the calculation process of the loss error function _LRL with the optimum action is performed (step S34). The step S33 may be performed after the step S31, and the step S34 may be performed after the step S32. For example, the step S32 may be performed before the step S31.
Based on the loss error function L _HE with human-like behavior and the loss error function L _RL with optimal behavior, the fusion model loss function calculator 5 is used to calculate the fusion loss error function L _Mix (step S35). .. In the fusion model learning executor 2, the calculated fusion loss error function L _Mix is acquired, and the fusion model learning process is performed (step S36). When the fusion loss error function L _Mix is equal to or greater than a predetermined value (step S37), the fusion model learning executor 2 again performs the fusion model execution processes 1 and 2 (steps S31 and S32).

Next, each process shown in FIG. 14 will be described. FIG. 15 shows an execution processing flow FIG. 1 of the fusion model of the fifth embodiment. As shown in FIG. 15, the fusion model learning executor 2 acquires the state data S _R1 from the database 61 (DB1) (step S311). The state data S _R1 is input to the fusion model π and executed (step S312). The behavior data A _F1 of the fusion model is stored (step S313).

FIG. 16 shows an execution processing flow FIG. 2 of the fusion model of the fifth embodiment. As shown in FIG. 16, the fusion model learning executor 2 acquires the state data S _R2 from the database 63 (DB2) (step S321). The state data S _R2 is input to the fusion model π and executed (step S322). The behavior data A _F2 of the fusion model is stored (step S323).

FIG. 17 shows a calculation processing flow diagram of the loss error function _LHE with the human-like behavior of Example 5. As shown in FIG. 17, the first loss error calculator 30 in the loss function calculator 3 and likely human behavior acquires behavior data A _R of the human from the database 61 (step S331). Further, the loss function calculator 3 acquires the behavior data A _F1 of the fusion model (step S332). First loss error calculator 30, from the obtained _{A R} and _{A F1,} calculates the loss error function _{L HE} (step S333).

FIG. 18 shows a calculation processing flow diagram of the loss error function _LRL with the optimum behavior of the fifth embodiment. As shown in FIG. 18, a loss function calculator 4 with optimal action acquires behavior data _{A RL} reinforcement learning model from the database 63 (DB2) (step S341). Separately from this, the second loss error calculator 40 acquires the behavior data A _F2 of the fusion model (step S342). The second loss error calculator 40 calculates the loss error function L _RL from the acquired behavior data A _RL and the behavior data A _F2 (step S343).

The calculation process of the fusion loss error function L _{Mix in} step S35 shown in FIG. 14 is the same as the calculation process flow diagram of the fusion loss error function L _Mix of Example 1 shown in FIG. 7. Further, the learning process of the fusion model in step S36 shown in FIG. 14 is performed in the same manner as the learning process flow diagram of the fusion model of Example 1 shown in FIG.

FIG. 19 shows a functional block diagram of the learning device of the agent of the sixth embodiment. As shown in FIG. 19, the learning device 16 includes a fusion model learning executor 2, a loss function calculator 3 with human-like behavior, a loss function calculator 4 with optimal behavior, a fusion model loss function calculator 5, and a database ( It consists of 62,63). The loss function calculator 3 is provided with an imitation learning model executor 7, a soft targeting processor 9, and a first loss error calculator 30, and the loss function calculator 4 is provided with a reinforcement learning model executor 8, software. A targeting processor 9 and a second loss error calculator 40 are provided.
The learning device 16 of the sixth embodiment has a database 62 (DB1) in which play data by the imitation learning model π _IL provided in the imitation learning model executor 7 is stored, and a reinforcement learning provided in the reinforcement learning model executor 8. It differs from Example 5 in that the database (61, 63) is used in that it uses two types of databases, the database 63 (DB2) in which the play data by the model π _RL is stored.

(About performance evaluation of fusion model)
As explained in the above embodiment, the learning method of learning a human-like agent AI while maintaining the high performance of the reinforcement learning model is a process of learning an agent AI that takes an efficient optimum action that surpasses the human performance. And, it is composed of two processes of learning the agent AI that selects an action like a human being.
Each process is tackled as a task of reinforcement learning and imitation learning. Therefore, in the present invention, the fusion model of reinforcement learning and imitation learning is a method based on the distillation of measures in the case of discrete action space and based on hostile imitation learning in the case of continuous action space.
If π ^* is the optimal policy based on the reinforcement learning model, π _HE is the human (expert) policy, and the parameter that determines the ratio of these two policies is α ∈ (0,1), the objective function is as follows. It becomes like an expression.

In the case of discrete behavioral space, the objective function of imitation learning is defined as the following cross entropy loss according to the previous research of imitation learning.

Human (expert) policy π _HE is difficult to define as a mathematical model, so we will train on experimentally sampled data. Despite the fact that soft targets cannot be obtained from π _HE , better performance can be obtained for policy distillation by calculating the weighted average of hard and soft targets. Therefore, the data provided to the person (expert) is set as the hard target, and the output of the policy π ^(T) _RL of the trained model adjusted by the heat degree T is set as the soft target. Finally, the loss function looks like this:

In the case of a continuous action space, a known GAIL (Generative Adversarial Imitation Learning) method is used as an imitation learning method (for the GAIL method, see Non-Patent Document 8). The GAIL method requires trajectories τ-π sampled from the teacher model π. The objective function maximized by the classifier D _w in the GAIL method and minimized by the student model are as follows.

Here, τ is the orbits τ to π sampled from the student model. In order to make the teacher model a human (expert) and a reinforcement learning model, the trajectories τ _HE to π _HE and τ _RL to π _RL are sampled from each expert. In addition, the fusion loss function can be replaced as follows: Intuitively, the discriminator D _w is trained to recognize the fusion policy between the person (expert) and the policy of the reinforcement learning model, and the student model π trained to deceive this classifier becomes the fusion policy. Expected to approach and mimic the strengths of both experts.

The experiments carried out will be described.
(Experiment 1) Atari 2600 game (Gopher)
The learning device of Example 1 was first applied to a game of an Atari 2600 system named Gopher in a discrete action space. In this game, the user of the game acts as a farmer (behavior) and moves left and right or fills a hole so that the mouse (Gopher) coming out from the basement to the ground cannot take the carrot. A person (expert) and a trained learning model each provided 55,000 frames, and the training set was 50,000 and the test set was 5,000. In particular, an Adam optimizer with a learning rate of ^10-4 and a Dropout rate of 0.5 were used to train the student model. In addition, the fusion model was trained with the heat of knowledge distillation set to T = 0.1 and the trade-off coefficient set to α = 0.93.

(Experiment 2) Torcs
Torcs (see Wymann, B., “The open racing car simulator”, (2015)) is one of the most commonly used simulators in autonomous driving research. This experiment using Torcs was based on the GymTorcs environment. The observation space of the agent AI consists of 65 continuous values in total, such as the distance from the car to the edge, the distance to the enemy car, the current speed and acceleration. The action space consists of two elements, "left and right" and "acceleration / deceleration", and the possible values are limited to the range [-1.0, 1.0].
The reward function was the distance traveled, and the reinforcement learning model was trained based on OpenAI Baselines. Furthermore, in order to show the situation where humanity can be discerned, a stop bot was made to the Torcs simulator, and a person (expert) was made to play 220 episodes of 60 seconds, and the data was collected. A mere imitation learning agent was trained on human (expert) data using the GAIL method of OpenAI in the same way as the training of the reinforcement learning model. Finally, the learner was updated in the GAIL method, and the fusion model was trained with a trade-off coefficient α = 0.5 in order to equalize the influences of both experts.
(Experiment 3) Apple game In the Apple game, the player collects apples that randomly appear on the screen and gives a score based on the number of collected apples.

(3) Kansei test of humanity Except for the evaluation of each model, a double-blind sensitivity test was conducted to evaluate the humanity of the model. The test targeted a total of 26 judges, 23 males and 3 females. The age range is 27 to 59 years and the average age is 44 years. All the judges of the Kansei test had no contact with the materials related to the content of the present invention before this investigation. First, we explained the rules of each game to the judges and held a hands-on session for each game so that they could understand the human behavior (behavior). As for the content of the survey, each judge provided two videos (15 seconds for Gopher and 30 seconds for Torcs) for each game, and asked them to judge whether they were humans or AI and why.

The experimental results will be described.
(1) Atari 2600 game (Gopher)
Regarding the performance, the reinforcement learning model (Comparative Example 3) was followed by the fusion model of the example, and finally the person (Comparative Example 1) and the imitation learning model (Comparative Example 2) in descending order of the score. In the fusion model of the example, although the target provided in the reinforcement learning model (Comparative Example 3) was prioritized at α = 0.8, the score was improved by only 3 points, and the score of the single reinforcement learning model was obtained. There is a big difference with. Regarding the Kansei test, it was judged that the reinforcement learning model (Comparative Example 3) was not very human-like, but the fusion model of the Example was not only in terms of score but also in humanity as well as human (Comparative Example 1) and its imitation (Comparative Example). 2) Showed a higher score.
It can be seen that the fusion model was able to learn the learning tendency and the behavior of the person (expert) toward the goal of reinforcement learning. Surprisingly, the fusion model was judged to be more human than human (expert). In order to clarify the reason, the impressions often expressed by the judge's comment analysis are "less useless movement", "fine movement, program feeling in movement", "trying to fill the hole in order". "Met. Therefore, it is considered that people (experts) were not expected to have high performance, especially for judges who do not play many games. The results of the scores are summarized in Table 3.

(2) Torcs
Regarding the evaluation of performance, first, the scores of the DDPG and the fusion model of each person (expert), imitation learning of the person by GAIL, and reinforcement learning were compared. The results of the scores are summarized in Table 4. From experiments, it was observed that GAIL excels in imitating trained reinforcement learning models and deterministic bots, but the efficiency of imitation of humans (experts) is surprisingly low. It is presumed that the human (expert) policy is complicated and difficult to handle with a basic neural network. On the other hand, the fusion model of the example succeeded in imitating the characteristics of, for example, the high speed of the reinforcement learning model (Comparative Example 3) and the bending method of a person (Comparative Example 1). Furthermore, it became possible to run the entire track like a person (Comparative Example 1) and a reinforcement learning model (Comparative Example 3).
In ascending order of the percentage of people judged to be human, the reinforcement learning model (Comparative Example 3) was judged to be not very human because it "runs too fast" or "turns a corner at high speed". Surprisingly, it was judged that the person (Comparative Example 1) was not human. For the same video, the judges' comments are diverse, but the imitation agent AI of the person who showed lower performance was judged to be more human than the person (expert), so like Gopher, the person (Comparative Example 1) It can be said that the performance shown in) is high. Finally, the fusion model was judged to be the most human, and showed performance close to that of the reinforcement learning model (Comparative Example 3).

(Experiment 3) Apple game An Apple game is a game that moves a player's avatar to the position of an apple that appears. Reinforcement learning achieved the highest score, followed by the learning method of the examples, and finally the person (Comparative Example 1) and its imitation learning.
As far as humanity is concerned, the human agent (Comparative Example 1) was the best. The learning method of the example showed more human-like behavior than the agent of the reinforcement learning model (Comparative Example 3) while surpassing the agent of the human (Comparative Example 1) and its imitation learning model (Comparative Example 2) in the score, and then. Followed by a person (Comparative Example 1). From this result, it can be seen that the learning method of the embodiment balances human behavior and high performance in this game.

The present invention is useful for learning agents in a wide range of fields such as automatic driving of automobiles and automatic control of industrial robot arms.

1,11-16 Learning device 2 Fusion model learning executor 3,4 Loss function calculator 5 Fusion model loss function calculator 6,61-63 Database 7 Imitation learning model executor 8 Reinforcement learning model executor 9 Soft targeting process Instrument 30 First loss error calculator 40 Second loss error calculator

Claims (12)

The learning device that realizes the behavior that the agent judges and takes the optimum action under the predetermined environment and the learning device that realizes the human-like behavior are fused, and the action policy of the agent is optimized to take the optimum action like a person. It ’s a learning method
An input step for inputting status data S _R and behavioral data A _R in the play data, or at least one of recording of the play data of the agent created with given purpose by a human expert,
Against learning executor fusion model and imitation learning model related to the human seems behavior as reinforcement learning model related to behavior taking the optimal action, the action data A _F of the fused model to input the state data S _R Learning steps to output and
A first loss error calculating step of calculating a first loss error between the behavior data A _F and the action data A _R,
And behavior data A _RL which is output based on the state data S _R by the execution unit or optimal behavior algorithm of the reinforcement learning model, the second loss error calculation for calculating a second loss error between the behavior data A _F Steps and
A fusion error calculation step that calculates the fusion error based on the weight ratio of the first and second loss errors, and
An agent learning method comprising: an update step of updating a parameter of a learning executor of the fusion model based on the fusion error. In the input step, the status data S _R and behavioral data A _R is, in the case of recording the play data by human experts,
The first loss error calculation step described above is
And said action data A _R in the play data by human experts, the error between the behavior data A _F, is calculated by using the loss function,
The second loss error calculation step described above is
Soft targeting the behavioral data with the status data S _R has been inputted to output the behavior data A _RL in the RL model by Knowledge distillation, the error between the behavior data A _F, is calculated by using the loss function The agent learning method according to claim 1, wherein the agent learning method. In the input step, when the state data S _R and behavioral data A _R is a record of the play data of the agent of imitation learning model,
The first loss error calculation step described above is
The imitation learning model Agent error between the behavior data A _R in the play data and the action data A _F, or the imitation learning model in the state data S _R knowledge behavior data A _IL that enter to output The error between the behavior data softly targeted by distillation and the behavior data _AF was calculated using a loss function.
The second loss error calculation step described above is
The error between the behavior data A _RL soft-targeted by knowledge distillation and the behavior data A _F , which is output by inputting the state data S _R into the reinforcement learning model, is calculated using a loss function. The agent learning method according to claim 1, wherein the agent learning method. In the input step, the status data S _R and behavioral data A _R is, in the case of recording the play data of the agent of reinforcement learning model,
The first loss error calculation step described above is
The soft targeting the behavioral data imitation learning model in the state data S _R behavioral data A _IL that enter to output by knowledge distillation, the error between the behavior data A _F, is calculated by using the loss function ,
The second loss error calculation step described above is
The error between the behavior data A _RL soft-targeted by knowledge distillation and the behavior data A _F , which is output by inputting the state data S _R into the reinforcement learning model, is calculated using a loss function. The agent learning method according to claim 1, wherein the agent learning method. In the input step, the status data S _R and behavioral data A _R is, status data S _HE and behavioral data A _HE at recording play data by human _experts, and, in the recording of the play data of the agent of the reinforcement learning model In the case of state data S _RL and behavior data A _RL ,
In the learning step, the state data _SHE is input to output the action data A _F1 , and the state data _SRL is input to output the action data A _F2 .
The first loss error calculation step described above is
The error between the behavior data A _HE and the behavior data A _F1 in the play data by a human expert is calculated by using the loss function.
The second loss error calculation step described above is
The error between the behavior data A _RL soft-targeted by knowledge distillation and the behavior data A _F2 output by inputting the state data S _RL into the reinforcement learning model is calculated by using a loss function. The agent learning method according to claim 1, wherein the agent learning method. In the input step, the state data S _R and the behavior data _AR are the state data _SIL and the behavior data A _IL in the recording of the play data of the agent of the imitation learning model, and the play data of the agent of the reinforcement learning model. In the case of state data S _RL and behavior data A _RL in the recording of
In the learning step, the state data _SIL is input to output the action data A _F1 , and the state data _SRL is input to output the action data A _F2 .
The first loss error calculation step described above is
The error between the behavior data A _IL soft-targeted by knowledge distillation in the play data of the agent of the imitation learning model and the behavior data A _F1 is calculated by using a loss function.
The second loss error calculation step described above is
The error between the behavior data A _RL soft-targeted by knowledge distillation and the behavior data A _F2 output by inputting the state data S _RL into the reinforcement learning model is calculated by using a loss function. The agent learning method according to claim 1, wherein the agent learning method. The soft targeting by knowledge distillation according to any one of claims 2 to 6, wherein the output of the agent's action policy has a parameter of heat degree T and can correspond to hard targeting and soft targeting. The described agent learning method. A learning device that optimizes the behavior policy of the agent so that it behaves in a human-like manner by fusing a learning device that realizes the behavior that the agent judges and takes the optimum behavior under a predetermined environment and a learning device that realizes the human-like behavior. And
An input unit for inputting the state data S _R and behavioral data A _R in the play data, or at least one of recording of the play data of the agent created with given purpose by a human expert,
Against fusion model and imitation learning model related to the human seems behavior as reinforcement learning model related to behavior taking the optimal action, learning execution for outputting behavior data A _F of the fused model to input the state data S _R With a vessel
A first loss error calculator for calculating a first loss error between the behavior data A _F and the action data A _R,
And behavior data A _RL which is output based on the state data S _R by the execution unit or optimal behavior algorithm of the reinforcement learning model, the second loss error calculation for calculating a second loss error between the behavior data A _F With a vessel
A fusion error calculator that calculates the fusion error based on the weight ratio of the first and second loss errors, and
An agent learning device including an update unit that updates parameters of the learning executor of the fusion model based on the fusion error. Depending on the state data S _R and behavioral data A _R in the input section, the following 1) to 5) the agent of the learning apparatus according to claim 8, characterized in that it comprises a configuration in which any one of the calculation process of :
1) the state data S _R and behavioral data A _R is, in the case of recording the play data by human experts,
The first loss error calculator described above is
And said action data A _R in the play data by human experts, the error between the behavior data A _F, is calculated by using the loss function,
The second loss error calculator described above is
Soft targeting the behavioral data with the status data S _R has been inputted to output the behavior data A _RL in the RL model by Knowledge distillation, the error between the behavior data A _F, is calculated by using the loss function ,
2) If the state data S _R and behavioral data A _R is a record of the play data of the agent of imitation learning model,
The first loss error calculator described above is
The imitation learning model Agent error between the behavior data A _R in the play data and the action data A _F, or the imitation learning model in the state data S _R knowledge behavior data A _IL that enter to output The error between the behavior data softly targeted by distillation and the behavior data _AF was calculated using a loss function.
The second loss error calculator described above is
Soft targeting the behavioral data with the status data S _R has been inputted to output the behavior data A _RL in the RL model by Knowledge distillation, the error between the behavior data A _F, is calculated by using the loss function ,
3) the state data S _R and behavioral data A _R is, in the case of recording the play data of the agent of reinforcement learning model,
The first loss error calculator described above is
The soft targeting the behavioral data imitation learning model in the state data S _R behavioral data A _IL that enter to output by knowledge distillation, the error between the behavior data A _F, is calculated by using the loss function ,
The second loss error calculator described above is
Soft targeting the behavioral data with the status data S _R has been inputted to output the behavior data A _RL in the RL model by Knowledge distillation, the error between the behavior data A _F, is calculated by using the loss function ,
4) the state data S _R and behavioral data A _R is, status data S _HE and behavioral data A _HE at recording play data by human _experts, and state data S in the recording of the play data of the agent of the reinforcement learning model _RL and behavior data A If _RL ,
The learning executor inputs the state data _SHE to output the action data A _F1 , and inputs the state data _SRL to output the action data A _F2 .
The first loss error calculator described above is
The error between the behavior data A _HE and the behavior data A _F1 in the play data by a human expert is calculated by using the loss function.
The second loss error calculator described above is
The error between the behavior data A _RL soft-targeted by knowledge distillation and the behavior data A _F2 , which was output by inputting the state data S _RL into the reinforcement learning model, was calculated using a loss function. ,
5) The state data S _R and the behavior data _AR are used in the recording of the state data _SIL and the behavior data A _IL in the recording of the play data of the agent of the imitation learning model, and the play data of the agent of the reinforcement learning model. In the case of state data S _RL and behavior data A _RL ,
The learning executor inputs the state data _SIL to output the action data A _F1 , and inputs the state data _SRL to output the action data A _F2 .
The first loss error calculator described above is
The error between the behavior data A _IL soft-targeted by knowledge distillation in the play data of the agent of the imitation learning model and the behavior data A _F1 is calculated by using a loss function.
The second loss error calculator described above is
The error between the behavior data A _RL soft-targeted by knowledge distillation and the behavior data A _F2 output by inputting the state data S _RL into the reinforcement learning model is calculated using a loss function. .. The learning of the agent according to claim 9, wherein the soft targeting by the knowledge distillation is characterized in that the output of the action policy of the agent has a parameter of heat degree T and can correspond to the hard targeting and the soft targeting. apparatus. An agent learning program for causing a computer to execute all steps in the agent learning method according to any one of claims 1 to 7. A computer is used as an input unit, a learning executor, a first loss error calculator, a second loss error calculator, a fusion error calculator, and an update unit in the learning device of any of the agents of claims 8 to 10. An agent learning program to make it work.

Images