KR102709703B1 - Apparatus and method for designing reinforcement learning based on game behavior replication - Google Patents

Abstract

The present invention relates to a behavior replication technology, and more particularly, to a design device for improving learning performance through game behavior replication-based reinforcement learning. According to an embodiment of the present invention, when policy-based reinforcement learning is applied using a preprocessing technology and a behavior replication algorithm, the learning time and state space problems occurring during learning are optimized, so that reinforcement learning can be applied in real time in a complex game environment such as a ball game with attack-defense transitions. In addition, when the episode and reward function are designed according to the preprocessing technology of the present invention and other complex game environments, it can be applied to various games.

Description

{APPARATUS AND METHOD FOR DESIGNING REINFORCEMENT LEARNING BASED ON GAME BEHAVIOR REPLICATION}

The present invention relates to a behavior replication technology, and more specifically, to a design device for improving learning performance through game behavior replication-based reinforcement learning.

Recently, research on game artificial intelligence (AI) has been actively conducted. Finite State Machine (FSM)-based AI is being applied to various commercial games. However, FSM-based AI only performs uniform actions in the same situation, so user satisfaction is low. Therefore, various studies are being conducted on applying reinforcement learning to AI that replaces non-player characters (NPCs) or players.

Reinforcement learning algorithms can generally be classified into value function-based reinforcement learning such as Q-Learning and DQN (Deep Q-Network) and policy-based reinforcement learning such as A2C and A3C. Since Q-Learning evaluates the value function through the Q-table for the state provided by the environment and the action selected by the agent through it, it requires a lot of memory when the state and action space of the agent increase, and there is a problem that learning does not work well when the previous observation data affects the next observation data. DQN is a method to predict the value function by combining a deep neural network with Q-Learning, and solves the data correlation problem of Q-Learning by using a replay buff and a target network, but requires a large amount of replay buff and a lot of learning time. A2C and A3C are policy-based reinforcement learning algorithms that allow agents to learn while interacting with the environment in real time, enabling faster learning speeds and convergence. They also enable smooth learning in a continuous action space, enabling more accurate learning than value function-based reinforcement learning in environments with various actions.

The behavioral replication algorithm is a representative imitation learning algorithm. The behavioral replication algorithm extracts states and actions from the collected expert game data and uses them as learning data. Based on this, AI learning that behaves similarly to the expert's policy is possible.

Reinforcement learning is an artificial intelligence algorithm that solves problems through rewards obtained by agents through interaction with the surrounding environment. Since it learns without defining an environment model, it is suitable for application to complex game environments, but it requires a lot of time and cost for learning. Therefore, a design method for reinforcement learning that can be applied in complex game environments in real time by reducing the time and cost required for reinforcement learning is needed.

The background technology of the present invention is published in Republic of Korea Patent Publication No. 2020-0108718.

The purpose of the present invention is to provide a design method and device for policy-based reinforcement learning using a preprocessing technique for reducing the complexity of learning and a behavior replication algorithm for minimizing trial and error in the early stage of learning in order to apply reinforcement learning in real time in a complex game environment with offense and defense transitions, such as a ball sports game.

In order to achieve the above purpose, the present invention seeks to solve learning time and state space problems that occur when applying reinforcement learning to a ball sports game with complex offense-defense transitions by using three preprocessing techniques and a behavior replication algorithm.

First, in order to reduce the amount of learning increased by all raw data generated in a ball sports game environment with air-to-ground transitions, the raw data is expressed as normalized relative information, thereby reducing the state space of reinforcement learning and enabling stable learning by expressing the raw data in an unstable game environment as a state suitable for learning.

Second, because there are various difficulties in defining reward functions to be applied to reinforcement learning in ball sports games with offense and defense transitions, reward design is performed by considering the main actions related to offense and defense and the frequency of the main actions in a complex environment.

Third, because learning multiple episodes at once increases the learning complexity, we can reduce the learning complexity by dividing the episodes by situation, such as an offensive situation where the own team has the ball, a defensive situation where the enemy team has the ball, and other situations where no one has the ball, which are representative situations that occur in ball sports games with offense-defense transitions.

In addition, reinforcement learning often causes trial and error in the early stage of learning when learning is performed with an arbitrary policy. Since trial and error lowers learning performance, in order to improve this part, a behavior replication algorithm that extracts states and actions from expert game data and performs AI learning is applied to the reinforcement learning algorithm to increase learning performance, and a game behavior replication-based reinforcement learning design device and method that can be applied in real time in complex game environments with offense and defense transitions such as ball sports games are provided.

The technical problems to be achieved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

According to one aspect of the present invention, a device for designing reinforcement learning based on game action replication is provided.

A game action replication-based reinforcement learning design device according to one embodiment of the present invention may include a preprocessing unit that performs preprocessing for learning by normalizing state representation in a game environment, evaluating actions, and classifying episodes, a reinforcement learning unit that performs reinforcement learning based on the preprocessing, and an action determination unit that receives a state, determines an action, and applies it to a game.

According to another aspect of the present invention, a game action replication-based reinforcement learning design method and a computer program for executing the same are provided. A game action replication-based reinforcement learning design method according to an embodiment of the present invention may include a step of performing preprocessing for normalizing a state representation in a game environment, evaluating an action, and classifying an episode to learn, a step of performing reinforcement learning based on the preprocessing, and a step of receiving a state, determining an action, and applying the same to a game.

According to an embodiment of the present invention, when policy-based reinforcement learning is applied using a preprocessing technique and a behavior replication algorithm, the learning time and state space problems that occur during learning are optimized, so that reinforcement learning can be applied in real time in a complex game environment such as a ball game with attack-defense transitions. In addition, when the episode and reward function are designed according to the preprocessing technique of the present invention and other complex game environments, it can be applied to various games.

In addition, the effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the composition of the invention described in the description or claims of the present invention.

FIG. 1 is a drawing for explaining a game action replication-based reinforcement learning design device according to one embodiment of the present invention.
FIG. 2 is a diagram for explaining a preprocessing unit of a game action replication-based reinforcement learning design device according to one embodiment of the present invention.
FIG. 3 is a diagram explaining the structure of a game action replication-based reinforcement learning design device according to one embodiment of the present invention.
FIG. 4 is a diagram for explaining a normalization process for agent position state values of a device for designing reinforcement learning based on game action replication according to a basketball game, which is an embodiment of the present invention.
FIGS. 5 and 6 are examples for comparing a game action replication-based reinforcement learning design device according to one embodiment of the present invention with a conventional reinforcement learning device.
Figure 7 is an example to explain the experimental results of a traditional FSM-based AI and a game action replication-based reinforcement learning design device.
Figure 8 is a diagram illustrating the average successful actions per game of a traditional FSM-based AI and a game action replication-based reinforcement learning design device.
FIG. 9 is a diagram for explaining a game action replication-based reinforcement learning design method according to one embodiment of the present invention.

The present invention can have various modifications and various embodiments, and specific embodiments of basketball games are illustrated in the drawings and described in detail through detailed descriptions. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention. In describing the present invention, if it is determined that a specific description of a related known technology may unnecessarily obscure the gist of the present invention, a detailed description thereof will be omitted. In addition, the expressions "a," "an," and "the" used in this specification and claims should generally be interpreted to mean "one or more" unless otherwise stated.

Throughout the specification, when a part is said to be "connected (connected, contacted, joined)" to another part, this includes not only cases where it is "directly connected" but also cases where it is "indirectly connected" with another member in between. Also, when a part is said to "include" a certain component, this does not mean that other components are excluded, unless otherwise specifically stated, but that other components can be included.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. As used herein, the terms "comprises" or "has" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood to not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

Hereinafter, the present invention will be described with reference to the attached drawings. However, the present invention can be implemented in various different forms, and therefore is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are assigned similar drawing reference numerals throughout the specification.

FIG. 1 is a drawing for explaining a game action replication-based reinforcement learning design device according to one embodiment of the present invention.

Referring to Fig. 1, a game action replication-based reinforcement learning design device includes a preprocessing unit (110), a reinforcement learning unit (120), and an action decision unit (130).

The preprocessing unit (110) performs preprocessing for normalizing state representation in the game environment, evaluating actions, and classifying episodes for learning. The preprocessing unit (110) may include a normalization module for state representation, a reward module for calculating a reward function, and an episode classification module for classifying episodes. The preprocessing unit (110) will be described in more detail in FIG. 2 below.

그리고 업데이트 버퍼

를 초기화를 수행한 후, 학습은 리턴값이 수렴할 때까지 다음과 같이 반복 수행한다. 상태 를 이용해 에서 행동 을 구하고, 상태 와 행동 에 따라 보상 이 결정된다. 전문가 정책 에서 전문가 행동 을 구하고 업데이트 버퍼

에 상태 , 행동 , 전문가 행동 , 보상 을 기록한다. 만약, 에피소드가 끝나면 학습 파라미터 는 정책 손실 함수를 통해 업데이트하며, 파라미터 는 가치 손실 함수로 업데이트한다. 한 에피소드에 대한 업데이트가 끝나면 업데이트 버퍼를 초기화하고, 리턴 값이 수렴할 때까지 반복 수행할 수 있다.The reinforcement learning unit (120) can perform reinforcement learning based on the preprocessing performed in the preprocessing unit (110). The reinforcement learning unit (120) classifies the state, action, and reward of the entire episode into attack, loose ball, and defense situations in the episode classification module and transmits them to the A2C (Advantage Actor-Critic) module corresponding to each, and performs policy-based reinforcement learning for each classified episode by imitating the preprocessing and expert actions. The policy that can imitate expert actions can also be applied to attack, defense, and other situations. That is, the reinforcement learning unit (120) can solve the learning time and state space problems that occur when applying reinforcement learning in a complex game environment with offense and defense transitions, such as a ball game, by using a normalization module, a reward module, an episode classification model, and a behavior replication algorithm that can imitate expert actions. At this time, the pseudo-code of the behavior replication-based reinforcement learning is learning parameter and parameter

And update buffer

After initializing, learning is repeated as follows until the return value converges. State By using Action in Find and state Wow action Compensation according to This is decided by expert policy Professional behavior in Find and update buffer

In the state , action , professional behavior , compensation . If the episode ends, the learning parameters is updated through the policy loss function, and the parameters is updated with a loss function. After the update for one episode is finished, the update buffer can be initialized and repeated until the return value converges.

The action decision unit (130) can receive a state, determine an action, and apply it to the game. That is, the action that performed reinforcement learning can be reflected by applying the action to a basketball game, such as defense, offense, and loose ball.

FIG. 2 is a diagram for explaining a preprocessing unit of a game action replication-based reinforcement learning design device according to one embodiment of the present invention.

Referring to FIG. 2, the preprocessing unit (110) may include a state expression unit (111), a reward unit (113), and an episode classification unit (115).

The state expression unit (111) provides environmental information to the agent at a specific time. In addition, the state expression unit (111) can use a normalization technique based on relative state values to stabilize the state. The state expression unit (111) can use a normalization module to express the state. In this case, the normalization module receives only the state from the game and performs normalization. The normalization module performs normalization by inputting the state value expressed as an absolute value. Normalization is performed so that the maximum value becomes 1 while having a relative value based on the state value. In a ball sports game, the agent location, the location of the own team and the enemy team, the location of the ball, the distance to the ball, the ball clear condition, the remaining time of the game, the team score, etc. can be used, and at this time, the location can be expressed in two dimensions or three dimensions. This can be expressed, for example, as in [Table 1].

situation formula AI location Team Relative Position Enemy relative position Ball relative position Ball height (absolute value) Rim relative position Enemy relative angle Team Relative Angle Mark relative angle Ball distance Rim distance Whether the ball is clear or not Whether to pivot Remaining time in the game Foul remaining time Team Score

At this time, team(AI, n) means the nth team member of the AI team, enemy(AI, m) means the mth team member of the AI opponent team, and mark(AI) means the mark status. In addition, the location indicates the location of the character, basketball, and rim. Each character can express his or her own location as the origin, and other characters, basketballs, and rims can be expressed as relative locations. The angle is not a status provided in the environment, but can be calculated and used as a status value. It is used as a reference value to determine whether the character is open, and the distance can be used as a status, for example, the distance between the reference object and the opponent object in meters divided by the maximum allowable distance of 10m. At this time, the condition can be whether the AI team clears the ball or not ball_clear(AI), and whether the AI pivots or not pivot(AI). In addition, the condition can have a value of 0 or 1. The time is the time remaining until the end of the game in a basketball game in seconds and the violation time. For example, the time is divided into the maximum time of 240 seconds and 20 seconds, respectively, and normalized to a value between 0 and 1. The score represents the current score of the AI team, and the value divided by the maximum allowable score of 30 can be used as the status.

The reward unit (113) reflects the value of the agent's actions at a specific point in time. The reward unit (113) can apply reward values to situations such as shooting, passing, breakthrough, marking, evading, ball clearing, ball picking up, and goal bottom situations, and can apply weights based on the frequency of occurrence in situations such as breakthrough, marking, evading, and ball picking up. The reward unit (113) can use a reward module that designs a detailed reward-based reward function predefined for action evaluation. The reward module considers the main actions required for offense and defense and the frequency of occurrence of the main actions in a complex game environment with offense and defense transitions, such as a ball game, in order to apply a reinforcement learning algorithm. can perform the design of compensation can be defined as a shot, pass, breakthrough, mark, dodge, etc., and can be calculated by adding up all the rewards that occurred. For example, a shot is open (0.0/1.0). Whether the shot is within range (0.0/1.0) Whether it's a shot (0.0/1.0), whether it's a pass or a pivot (0.0/1.0) Whether to pass (0.0/1.0), whether breakthrough is within the breakthrough effective range (0.0/1.0) Breakthrough (0.0/1.0) Weight (0.5), Mark is the mark angle 30 degrees or not (0.0/1.0) Mark Distance 2m or not (0.0/1.0) Weight (0.002), evasion is open or not (0.0/1.0) Whether the shot is within range (0.0/1.0) Weight (0.01), Ball Clear is whether the ball is successfully cleared (0.0/1.0), Ball Pickup is whether the ball is successfully picked up (0.0/1.0) - whether the ball moves in a different direction (0.0/1.0) Weight (0.002) and the bottom of the goal is the rim distance 2m or not (0.0/1.0) It can be calculated with a weight (0.001).

The episode classification unit (115) can learn by classifying at least one episode. In addition, the episode classification unit (115) can learn the state of the entire episode to reduce learning complexity. , action , compensation Preprocessing is performed so that reinforcement learning can be performed by classifying the situations into attack, defense, and others. The criteria for episode classification are as follows: if the team owns the ball, it is an attack; if the enemy team owns the ball, it is a defense; and if no one has the ball, it is others. Depending on the classified episode, the states, rewards, and actions used in learning can be applied differently to perform reinforcement learning. Depending on the results of the episode classification, the states, rewards, and actions used in learning are different for each attack, loose ball, and defense situation. For example, in the case of an attack episode, the actions of the attack episode can be expressed with 11 actions consisting of 8-way movement, shooting, breakthrough, and passing. This has the advantage of reducing the number of actions, which results in a smaller learning space, which can improve learning performance with the same episode.

FIG. 3 is a diagram explaining the structure of a game action replication-based reinforcement learning design device according to one embodiment of the present invention.

Referring to Fig. 3, the preprocessing process of the game behavior replication-based reinforcement learning design device includes a normalization module for expressing a state, a reward module for calculating a reward function, and an episode classification module for classifying episodes. After that, the game behavior replication-based reinforcement learning design device performs reinforcement learning based on a policy for each classified episode by imitating the preprocessing and expert behavior. In Fig. 4, a policy that can imitate expert behavior is applied to defense, but it can also be applied to offense and other situations. The normalization module is a state value expressed as an absolute value. Normalization is performed so that the maximum value becomes 1 while having a relative value based on the state value. In a ball sports game, the agent location, the location of the own team and the enemy team, the location of the ball, the distance to the ball, the ball clear condition, the remaining time of the game, the team score, etc. are used, and the location can be expressed in two or three dimensions. At this time, the condition has a value of 0 or 1 for whether the own team clears the ball, and the time can be normalized based on the maximum time, and the score can be normalized based on the maximum allowed score. The reward module in the preprocessing step is used to apply the reinforcement learning algorithm to a complex game environment with offense and defense transitions such as a ball sports game, and in the case of offense and defense, the reward is considered by considering the main actions required and the frequency of occurrence of the main actions. The design of the episode classification module can be performed in the preprocessing stage to reduce the learning complexity by classifying the state of the entire episode. , action , compensation It is possible to classify the situations into attack, defense, and other situations and perform reinforcement learning. The criteria for classifying the episodes are based on whether the team owns the ball or not: if the team owns the ball, it is attack; if the enemy team owns the ball, it is defense; and if no one has the ball, it is other. Depending on the classified episode, the states, rewards, and actions used in learning can be applied differently to perform reinforcement learning. The preprocessing-based reinforcement learning process can solve the learning time and state space problems that occur when applying reinforcement learning in a complex game environment with offense and defense transitions, such as a ball game, by using a normalization module, a reward module, an episode classification model, and a behavior replication algorithm that can imitate expert actions. After performing preprocessing-based reinforcement learning, it can be applied to the game.

FIG. 4 is a diagram for explaining a normalization process for agent position state values of a device for designing reinforcement learning based on game action replication according to a basketball game, which is an embodiment of the present invention.

Referring to Figure 4, the agent's location status value You can see that the normalization process for . As shown in Fig. 4, the blue circle means the reference agent, and the red circle indicates the location of the opposing team. After expressing the relative coordinates based on the blue circle, normalization can be performed based on the size of the stadium. The AI location uses a relative value based on the center of the stadium, and the location is expressed in two dimensions, but only the height of the ball can be considered. For example, the maximum allowable height of the ball can be 5m based on 0.

FIGS. 5 and 6 are examples for comparing a game action replication-based reinforcement learning design device according to one embodiment of the present invention with an existing reinforcement learning device.

Fig. 5 (a) is the experimental result of a traditional A2C without applying behavior replication. Fig. 5 (a) is a graph showing the number of occurrences of evasion rewards and the number of occurrences of mark rewards. Fig. 5 (b) is the experimental result of a learning of an A2C with applying behavior replication. Fig. 5 (b) is a graph showing the number of occurrences of evasion rewards and the number of occurrences of mark rewards. The evasion rewards of an attack episode without an expert policy can show similar values in the experimental results of a traditional A2C and the reinforcement learning design device based on game behavior replication according to an embodiment of the present invention. On the other hand, in a defense episode where an expert policy exists and behavior replication is applied, for example, the average of the last 100 data at the end of learning is 224 times for the number of occurrences of rewards of a traditional A2C and 428 times for the number of occurrences of rewards of a game behavior replication based reinforcement learning design device based on game behavior replication according to an embodiment of the present invention, and it can be confirmed that the mark success of a game behavior replication based reinforcement learning design device based on game behavior replication according to an embodiment of the present invention is about 1.9 times higher than that of a traditional A2C. Since the occurrence numbers of evasion and mark are unstable, the experimental results of the game behavior replication-based reinforcement learning design device according to an embodiment of the present invention and the traditional A2C can be compared with a linear trend line based on 0. The red line in Fig. 5 (b) is a trend line of the number of mark reward occurrences of the game behavior replication-based reinforcement learning design device according to an embodiment of the present invention, and the red line in Fig. 5 (a) is a trend line of the number of mark reward occurrences of the traditional A2C. The game behavior replication-based reinforcement learning design device has a slope of about 1.47, and the traditional A2C has a slope of about 0.68. The game behavior replication-based reinforcement learning design device has a slope of about 2.1 times higher. It can be seen that the game behavior replication-based reinforcement learning design device according to an embodiment of the present invention selects and performs an action that generates a relatively large reward compared to the traditional A2C.

Fig. 6 is an example of the results of a traditional FSM-based AI opponent experiment. Fig. 6 (a) is a score graph of a game behavior replication-based reinforcement learning design device according to an embodiment of the present invention, and Fig. 6 (b) is a score graph of a traditional FSM-based AI. Fig. 6 (c) is an accumulation of wins/draws/losses. It means the accumulated results of wins (+1), draws (+0), and losses (-1) based on the behavior replication-based A2C AI according to the game results. Fig. 6 (d) is a sum of rewards for each game obtained while the AI of the game behavior replication-based reinforcement learning design device according to an embodiment of the present invention performs learning. For example, the score difference is about 0.43 points on average, and the score of the game behavior replication-based reinforcement learning design device according to an embodiment of the present invention is relatively high. The average score difference of the games from the first to the 1,000th, where the FSM-based AI has a higher win rate, is about 0.4 points, which is higher for the FSM-based AI. According to an embodiment of the present invention, the average score difference of the last 1,000 games in which the AI winning rate of the game behavior replication-based reinforcement learning design device has a relatively high score shows that the proposed method recorded a score that is about 0.83 points higher. The accumulation of wins/draws/losses shows that the game behavior replication-based reinforcement learning design device according to an embodiment of the present invention has relatively many losses in the early stage of learning. However, as learning progresses, the losses gradually decrease and the winning rate increases. The graph also shows that the total reward records a low reward in the early stage, but gradually increases and converges to a constant value. In addition, for example, the results of the last 1,000 games show that the game behavior replication-based reinforcement learning design device according to an embodiment of the present invention recorded 512 wins, 146 draws, and 342 losses, for a winning rate of 60%.

Figure 7 is an example to explain the experimental results of a traditional FSM-based AI and a game action replication-based reinforcement learning design device.

Figure 7 is an example result of an experiment in which the AI of the game action replication-based reinforcement learning design device competes against the FSM-based AI designed by a basketball game expert. For example, when a total of 103 games were played, it can be confirmed that the performance was 68 wins and 35 losses.

Figure 8 is a diagram illustrating the average successful actions per game of a traditional FSM-based AI and a game action replication-based reinforcement learning design device.

Figure 8 shows the average successful actions per game of the AI of the game action replication-based reinforcement learning design device compared to the FSM-based AI designed by basketball game experts. Referring to Figure 8, the average number of successful actions per game of each AI can be confirmed. It can be confirmed that the AI of the game action replication-based reinforcement learning design device has relatively high indicators in the attack (2-point shot) and defense (block, steal) episodes compared to the FSM-based AI designed by experts.

FIG. 9 is a diagram for explaining a game action replication-based reinforcement learning design method according to one embodiment of the present invention.

Referring to FIG. 9, at step S910, a game action replication-based reinforcement learning design device inputs to express a state.

In step S930, the game action replication-based reinforcement learning design device performs preprocessing for learning by normalizing the state representation in the game environment, evaluating the action, and classifying the episode. It may include a normalization module for representing the state, a reward module for calculating a reward function, and an episode classification module for classifying the episode. The state representation may provide environment information to the agent at a specific time, and may use a normalization technique based on relative state values for stabilization. A normalization module may be used to represent the state, and at this time, the normalization module receives only the state from the game in step S910 and performs normalization. The normalization module may be configured to normalize the state value expressed as an absolute value. Normalization is performed so that the maximum value becomes 1 while having a relative value based on the state value. In a ball sports game, the agent position, the position of the own team and the enemy team, the position of the ball, the distance to the ball, the ball clear condition, the remaining time of the game, the team score, etc. can be used, and at this time, the position can be expressed in two dimensions or three dimensions. The reward reflects the value of the agent's action at a specific point in time. The reward can apply the reward value to the situations of shooting, passing, breaking through, marking, avoiding, ball clearing, ball picking up, and goal, and can apply a weight based on the frequency of occurrence in the situations of breaking through, marking, avoiding, and ball picking up. The game action replication-based reinforcement learning design device can use a reward module that designs a detailed reward-based reward function that is predefined for action evaluation. The reward module considers the main actions required for offense and defense and the frequency of occurrence of the main actions in a complex game environment with offense and defense transitions such as a ball sports game in order to apply a reinforcement learning algorithm. can perform the design of compensation can be defined as a shot, pass, breakthrough, mark, dodge, etc., and can be calculated by adding up all the rewards that occurred. For example, a shot is open (0.0/1.0). Whether the shot is within range (0.0/1.0) Whether it's a shot (0.0/1.0), whether it's a pass or a pivot (0.0/1.0) Whether to pass (0.0/1.0), whether breakthrough is within the breakthrough effective range (0.0/1.0) Breakthrough (0.0/1.0) Weight (0.5), Mark is the mark angle 30 degrees or not (0.0/1.0) Mark Distance 2m or not (0.0/1.0) Weight (0.002), evasion is open or not (0.0/1.0) Whether the shot is within range (0.0/1.0) Weight (0.01), Ball Clear is whether the ball is successfully cleared (0.0/1.0), Ball Pickup is whether the ball is successfully picked up (0.0/1.0) - whether the ball moves in a different direction (0.0/1.0) Weight (0.002) and the bottom of the goal is the rim distance 2m or not (0.0/1.0) can be calculated as a weight (0.001). The game action replication-based reinforcement learning design device can learn by classifying at least one episode. In addition, the episode classification step reduces the learning complexity by classifying the state of the entire episode. , action , compensation Preprocessing is performed so that reinforcement learning can be performed by classifying the situations into attack, defense, and others. The criteria for episode classification are as follows: if the team owns the ball, it is an attack; if the enemy team owns the ball, it is a defense; and if no one has the ball, it is others. Depending on the classified episode, the states, rewards, and actions used in learning can be applied differently to perform reinforcement learning. Depending on the results of the episode classification, the states, rewards, and actions used in learning are different for each attack, loose ball, and defense situation. For example, in the case of an attack episode, the actions of the attack episode can be expressed with 11 actions consisting of 8-way movement, shooting, breakthrough, and passing. This has the advantage of reducing the number of actions, which results in a smaller learning space, which can improve learning performance with the same episode.

를 초기화를 수행한 후, 학습은 리턴값이 수렴할 때까지 다음과 같이 반복 수행한다. 상태 를 이용해 에서 행동 을 구하고, 상태 와 행동 에 따라 보상 이 결정된다. 전문가 정책 에서 전문가 행동 을 구하고 업데이트 버퍼

에 상태 , 행동 , 전문가 행동 , 보상 을 기록한다. 만약, 에피소드가 끝나면 학습 파라미터 는 정책 손실 함수를 통해 업데이트하며, 파라미터 는 가치 손실 함수로 업데이트한다. 한 에피소드에 대한 업데이트가 끝나면 업데이트 버퍼를 초기화하고, 리턴 값이 수렴할 때까지 반복 수행할 수 있다.In step S950, the game behavior replication-based reinforcement learning design device can perform reinforcement learning based on the preprocessing performed in step S930. In the episode classification module, the state, action, and reward of the entire episode are classified into attack, loose ball, and defense situations, and are transmitted to the A2C (Advantage Actor-Critic) module corresponding to each, so that policy-based reinforcement learning can be performed for each classified episode by imitating the preprocessing and expert behavior. The policy that can imitate expert behavior can be applied to attack, defense, and other situations. In other words, when applying reinforcement learning in a complex game environment with offense and defense transitions, such as a ball game, the problem of learning time and state space that occurs can be solved by using a normalization module, a reward module, an episode classification model, and a behavior replication algorithm that can imitate expert behavior. At this time, the pseudo-code of the behavior replication-based reinforcement learning is learning parameter and parameter And update buffer

After initializing, learning is repeated as follows until the return value converges. State By using Action in Find and state Wow action Compensation according to This is decided by expert policy Professional behavior in Find and update buffer

In the state , action , professional behavior , compensation . If the episode ends, the learning parameters is updated through the policy loss function, and the parameters is updated with a loss function. After the update for one episode is finished, the update buffer can be initialized and repeated until the return value converges.

In step S970, the game action replication-based reinforcement learning design device can determine an action and apply it to the game. That is, the action that performed reinforcement learning can be reflected by applying the action to, for example, a basketball game such as defense, offense, and loose ball.

In the above, although all the components constituting the embodiments of the present invention have been described as being combined as one or operating in combination, the present invention is not necessarily limited to such embodiments. That is, within the scope of the purpose of the present invention, all of the components may be selectively combined as one or more and operate.

Although the operations are depicted in the drawings in a particular order, it should not be understood that the operations must be performed in the particular order depicted or in any sequential order, or that all depicted operations must be performed to achieve a desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various components in the embodiments described above should not be understood to imply that such separation is necessary, and that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products.

The present invention has been described with reference to embodiments thereof. Those skilled in the art will understand that the present invention can be implemented in modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a restrictive perspective. The scope of the present invention is indicated by the claims, not the foregoing description, and all differences within the scope equivalent thereto should be interpreted as being included in the present invention.

100: Game Behavior Replication-Based Reinforcement Learning Design Device
110: Preprocessing unit
120: Reinforcement Learning Department
130: Action Decision Section

Claims (11)

In a device for designing reinforcement learning based on game action replication,
A preprocessing unit that performs preprocessing to normalize state representation in the game environment, evaluate actions, and classify episodes for learning;
A reinforcement learning unit that performs reinforcement learning based on the above preprocessing; and
Action decision unit that receives status, decides action, and applies it to the game;
Including, but not limited to,
The above preprocessing unit
Provides environmental information to an agent at a specific time, and uses a normalization module to express a state, wherein the normalization module performs normalization by receiving only a state from a game and, when performing normalization, performing normalization so that the state value expressed as an absolute value has a relative value and the maximum value becomes 1, and the state value uses the agent position, the position of the own team and the enemy team, the position of the ball, the distance to the ball, the ball clear condition, the remaining time of the game, and the team score in a ball sports game, and a state expression unit that expresses the position in two or three dimensions;
A reward unit that reflects the value of an agent's actions at a certain point in time, applying reward values to situations such as shooting, passing, breaking through, marking, avoiding, ball clearing, ball picking up, and goal-side situations, and applying weights based on the frequency of occurrence in the situations of breaking through, marking, avoiding, and ball picking up; and
Includes an episode classification unit that classifies the state, actions, and rewards of the entire episode into attack, defense, and loose ball situations and passes them to the corresponding modules.
A device for designing reinforcement learning based on game action replication.
delete delete delete delete In a method for designing a reinforcement learning design device based on game action replication for reinforcement learning,
A step of performing preprocessing to normalize state representation in the game environment, evaluate actions, and classify episodes for learning;
A step of performing reinforcement learning based on the above preprocessing; and
The step of receiving the status, deciding on an action, and applying it to the game.
Including,
The steps for performing preprocessing to learn by normalizing state representation, evaluating actions, and classifying episodes in the above game environment are as follows.
A method of providing environmental information to an agent at a specific time and using a normalization module to express a state, wherein the normalization module performs normalization by receiving only a state from a game and, when performing normalization, performing normalization so that the state value expressed as an absolute value has a relative value and the maximum value becomes 1, and the state value uses the agent position, the position of the own team and the enemy team, the position of the ball, the distance to the ball, the ball clear condition, the remaining time of the game, and the team score in a ball sports game, and expressing the position in two or three dimensions;
A step of reflecting value to the agent's actions at a certain point in time, applying reward values to situations such as shooting, passing, breaking through, marking, avoiding, ball clearing, ball picking up and goal bottom situations, and applying weights based on the frequency of occurrence in the situations of breaking through, marking, avoiding and ball picking up; and
Including a step of learning by classifying the episode into attack, defense, and loose ball situations and passing the state, action, and reward of the entire episode to the corresponding module.
A method for designing reinforcement learning based on game action replication.
delete delete delete delete In Article 6,
A computer program that executes a game action replication-based reinforcement learning design method and is recorded on a computer-readable recording medium.