WO2022083029A1 - Decision-making method based on deep reinforcement learning - Google Patents

Abstract

A decision-making method based on deep reinforcement learning, the method comprising: an intelligent agent makes a decision according to environment information and selects an action after the decision; the intelligent agent compares said action with a knowledge base, and decides, on the basis of a set ruleset in the knowledge base, to execute said action or to replace said action; the intelligent agent executes said action or the replaced action in the environment, obtains a reward and new environment information from the environment, combines the old environment information, the action, the reward and the new environment information into experience information, and stores the experience information into an experience playback pool; and a set amount of experience information is randomly selected from the experience playback pool, so as to update a deep reinforcement learning model to thereby guide the next iteration. By utilizing the present application, training time may be shortened, catastrophic decision-making may be avoided, and the method may be widely applied to the field of dynamic decision-making.

Description

A decision-making method based on deep reinforcement learning technical field

The present invention relates to the field of artificial intelligence, and more particularly, to a decision-making method based on deep reinforcement learning.

Background technique

Reinforcement learning is a field in machine learning used to describe and solve problems in which agents learn strategies to maximize rewards or achieve specific goals in the process of interacting with the environment.

Currently, deep reinforcement learning has been successfully applied to a variety of dynamic decision-making domains, especially those with large state spaces. However, deep reinforcement learning also faces some problems. First, its training process can be very slow and resource-intensive. The final system is often fragile, the results are difficult to interpret, and it performs poorly for long periods of time at the beginning of training. Furthermore, for applications in robotics and critical decision support systems, the use of deep reinforcement learning can even make catastrophic decisions with costly consequences.

Therefore, improvements to existing technologies are required to obtain more efficient and safer decision-making methods.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the above-mentioned defects of the prior art, and to provide a decision-making method based on deep reinforcement learning, which is a new technical solution for dynamic decision-making by combining high abstraction level rules and deep reinforcement learning.

The present invention provides a decision-making method based on deep reinforcement learning. The method includes the following steps:

The agent makes decisions according to the environmental information and selects the actions after the decision;

The agent compares the action after the decision with the knowledge base, and decides whether to replace the action after the decision with the random action in the rule set based on the set rule set in the knowledge base;

In the case that it is judged to replace the action after the decision, execute the replaced action in the environment, obtain the reward and new environment information from the environment, and combine the old environment information, action, reward and new environment information into experience information, Stored in the experience playback pool;

A set amount of experience information is randomly selected from the experience replay pool to update the deep reinforcement learning model to guide the next iteration.

In one embodiment, determining whether to replace the post-decision action with a random action in the rule set according to the set rule set in the knowledge base includes:

Determine whether the rule set in the knowledge base satisfies the predetermined condition;

If the set conditions are met, replace the decided action with a random action in the rule set with a set probability.

In one embodiment, the post-decision action is replaced with a random action from the set of compliant actions α(R,t) with probability P _t = p ₀ ·γ ^t , where p ₀ is the initial rule intervention probability, t is the running time, γ is the decay rate, R represents the rule set, and α represents all actions that conform to the rule set R and at time t.

In one embodiment, the rule set is set according to a decision application scenario to avoid catastrophic decision-making or to improve learning efficiency, and is used to guide the actions of the agent in the application scenario.

In one embodiment, combining old environment information, actions, rewards and new environment information into one experience information, and storing it into the experience playback pool includes:

After obtaining the new environment information, store one unit of experience information (φ(s _t ), at , r _t , φ(s _{t +1} )) into the experience playback pool D;

If the capacity of the experience pool exceeds the set threshold N when new experience information is stored, the earlier experience information will be deleted with reference to the storage time.

In one embodiment, randomly selecting a set amount of experience information from the experience replay pool to update the deep reinforcement learning model includes:

In each step t of each round of interaction between the agent and the environment, D randomly selects a certain amount of experience information (φ(s _j ), a _j , r _j , φ(s _j+1 )) in the experience playback pool, And calculate the value of the current moment j of each experience information:

Take (y _j -Q(φ(s _j ), a _j ; θ)) ² as the objective function to do gradient descent to optimize the neural network parameter θ;

Finally, every fixed number of steps C, the target action-value function Q* is synchronized to the action-value function Q;

Among them, a' represents the optional action at time j+1, a _j represents the action at time j, s _j and s _j+1 represent the environmental information at time j and time j+1 respectively, and φ represents the preprocessing process.

Compared with the prior art, the present invention has the advantage that, in deep reinforcement learning, in addition to considering all possible Q-values for actions, applicable rules are also considered, by combining high abstraction level rules with deep reinforcement learning , which improves training performance and avoids catastrophic decisions.

Other features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention.

1 is a flowchart of a decision-making method based on deep reinforcement learning according to an embodiment of the present invention;

2 is a schematic diagram of a rule intervention learning framework according to an embodiment of the present invention;

FIG. 3 is a screen diagram in the Flappybird game according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of the effective range of rules in the Flappybird game according to an embodiment of the present invention;

5 is a graph of experimental results of average reward and average Q value in the Flappybird game according to an embodiment of the present invention;

FIG. 6 is a screen diagram in the Spacewar game according to an embodiment of the present invention;

7 is a graph of experimental results of average reward and average Q value in the Spacewar game according to an embodiment of the present invention;

8 is a screen diagram in a Breakout game according to an embodiment of the present invention;

9 is a graph of experimental results of average reward and average Q value in the Breakout game according to an embodiment of the present invention;

FIG. 10 is a screen diagram in the GirdWorld game according to an embodiment of the present invention;

Fig. 11 is the average reward experiment result graph in the GirdWorld game of one embodiment of the present invention;

In the attached figure, Average Reward - average reward; Average Q value - average Q value; Training Epochs - training period.

Detailed ways

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the invention unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

In all examples shown and discussed herein, any specific values should be construed as illustrative only and not limiting. Accordingly, other instances of the exemplary embodiment may have different values.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

Referring to FIG. 1 , the decision-making method based on deep reinforcement learning provided by this embodiment includes the following steps:

Step S110, preset the rule set and related hyperparameters in the knowledge base, and initialize the deep reinforcement learning model.

In order to facilitate the understanding of the present invention, referring to Fig. 2 first, the proposed decision-making method combining high-level abstraction rules and deep reinforcement learning includes a rule intervention learning (RIL) framework, specifically including a deep Q network (ie DQN, a A deep reinforcement learning algorithm that combines Q-learning and deep learning), knowledge base and environment, in which DQN and knowledge base have interactions for combining the rules in the knowledge base with deep Q-learning. In deep Q-learning, in addition to the In addition to considering all possible Q-values for action, consider whether any rules apply.

For example, a deep reinforcement learning model utilizes a neural network consisting of 3 convolutional layers, 1 hidden layer, and 1 output layer. For example, the first convolutional layer is 32 8*8*4 convolution kernels, the stride is 4, and the output result passes through the 2*2 max pooling layer; the second convolutional layer is 64 4*4 *32 convolution kernel with stride 2; the third convolutional layer is 64 3*3*64 convolution kernels with stride 1; the hidden layer consists of 256 fully connected layers to form ReLU function nodes; Since the number of actions may be different for different embodiments, the output layer may be different.

Among them, the ReLU function is:

It should be noted that the specific structure, number of layers, convolution kernel size, activation function, etc. of the neural network can be set according to the requirements for training accuracy and training time, which are not limited in the present invention.

The initialization process of step S110 includes setting the rule set in the knowledge base, the rule intervention probability, the preprocessing process, the loss function, the experience playback pool, the number of training rounds, and the like.

Set the rule set R in the knowledge base, the rule set can contain a variety of different types of rules, and can be set according to different application scenarios. It should be noted that each type of rule can work alone or in combination with other types of rules. Each rule in the rule set is used to limit the effective scope of the rule and the actions corresponding to the scope.

Set rule intervention conditions or intervention probabilities. For example, the rule intervention probability at the current moment is set as P _t =p ₀ ·γ ^t , where p ₀ is the initial rule intervention probability, γ is the decay rate, and t is the running time. It should be understood that the intervention probability is only exemplary, and other forms of regular intervention probability, such as P _t =γ ^t and the like, may also be used.

Preprocessing sets φ, which is used to process the initial information of the input. For example, a colored game screen becomes a gray screen after preprocessing settings. Specifically, the agent interacts with the environment in M rounds, but before each step of decision-making, the environmental information must be preprocessed. If the initial information of the environment in each round is s ₁ , the preprocessed initial information is φ(s ₁ ).

Exploration probability ε, in addition to selecting the action with the largest Q value, the agent randomly selects an action with probability ε. Specifically, in each step t of each round of interaction with the environment, the agent randomly selects an action to perform in the environment with a probability of size ε.

Set the experience playback pool D and its maximum capacity is N, which is used to store experience information for deep Q-learning training.

Initialize the action-value function Q and its parameters are randomly set to θ, and initialize the target action-value function Q ^* and its parameter θ ^* value size is set to θ, the target action-value function Q* is used to select the one with the largest Q* value action, expressed as:

a _t = argmax _a Q ^* (φ(s _t ), a; θ) (2)

Then, the loss function is calculated based on the action-value function Q and the value of the current moment j to update the network parameter θ, for example, the loss function is expressed as:

(y _j -Q(φ(s _j ),a _j ,θ)) ² (3)

Among them, the value of the current moment j is expressed as:

Among them, a' represents the optional action at time j+1.

In addition, the upper limit value M of the number of training rounds needs to be set, and when the number of training rounds reaches the upper limit value M, the training ends.

In step S120, the agent observes the environment and obtains information obtained from the environment.

In each step t of each round of interaction between the agent and the environment, the agent observes the environment and obtains environmental information s _t , and the agent preprocesses the environment _st into φ(s _t ), for example, the color game screen s _t is preprocessed The gray screen φ(s _t ) is displayed after processing the setting. It should be noted that each time the agent receives environmental information from the environment, it must go through this preprocessing operation.

Step S130, the agent makes a decision according to the environmental information, and selects an action after the decision.

In each step t of each round of interaction between the agent and the environment, after observing the preprocessed environmental information φ(s _t ), the agent randomly selects an action with the probability of ε, otherwise, according to the formula a _t = argmax _a Q ^* (φ(s _t ), a; θ) selects the post-decision action.

In step S140, the agent compares the action after the decision with the knowledge base, and judges whether there is a more suitable action according to the rules in the knowledge base. If there is a more suitable action, the action after the decision is replaced according to certain conditions. Substitution executes the post-decision action.

如果合规动作集α(R,t)连续非空且

则以P _t＝p ₀·γ ^t的概率用规则 集中的一个随机动作替换决策后的动作a _t，其中R表示规则集，t表示运行时间，α表示符合规则集R和在时间t下的所有动作。 Specifically, in each step t of each round of interaction between the agent and the environment, the agent compares the action after the decision with the knowledge base, and judges whether there is a more appropriate action according to the set of rules set in the knowledge base. If there is a more suitable action, the action after the decision is replaced according to certain conditions, and if there is no replacement, the action after the decision is executed. For example, set the ruleset

If the compliance action set α(R,t) is consecutively non-empty and

Then, with the probability of P _t = p ₀ ·γ ^t , the decision-making action at is replaced by a random action in the rule set, where R represents the rule set, _t represents the running time, and α represents the compliance with the rule set R and at time t. All actions.

In step S150, the agent executes the action after the decision or the action after the replacement in the environment, obtains rewards and other new information from the environment, combines the old environment information, action, reward and new environment information into one experience information, and saves it. into the experience playback pool.

In each step t of each round of interaction between the agent and the environment, execute the final selected action a _t to the environment and obtain the reward value r _t and the new environment observation value s _t+1 , and then preprocess the environment observation value to get φ(s _t+1 ), and combine φ(s _t ), at , r _t and φ(s _{t +1} ) into a unit of empirical information, denoted as (φ(s _t ), at , _{r t} , φ(s _t+1 )), stored in the experience playback pool D.

It should be noted that the maximum capacity of the experience playback pool D is N. If the capacity of the experience pool has reached the maximum value N when new experience information is stored, the earlier experience information needs to be deleted to make room.

In step S160, a certain amount of experience information is randomly selected from the experience playback pool to update the model to guide the next iteration.

In each step t of each round of interaction between the agent and the environment, D randomly selects a certain amount of experience information (φ(s _j ), a _j , r _j , φ(s _j+1 )) in the experience playback pool, Calculate the value of the current time j of each experience information

Then, take (y _j -Q(φ(s _j ), a _j ; θ)) ² as the objective function to do gradient descent to optimize the neural network parameter θ, and finally, every fixed number of steps C, the target action-value The function Q* is synchronized to the action-value function Q, and the updated model will guide the next iteration.

At the end of each round of interaction between the agent and the environment, the agent automatically enters the next round of interaction, and steps S120 to S160 are repeated until the preset upper limit number of training rounds M, the preset number of iterations or the loss function reaches the preset convergence condition.

In order to further illustrate the effect of the present invention, a specific application scenario will be taken as an example for description below.

Referring to Figure 3, in the Flappybird game, a bird manipulated by the agent tries to fly through pairs of pipes while avoiding hitting any pipes. In this scenario, the agent has two actions available, that is, control the bird to flap its wings or do nothing. By flapping its wings, the bird gets a temporary upward acceleration, so the bird can ascend a certain distance. If nothing is done, the bird will descend a certain distance due to gravity. Birds flying through pairs of pipes will be rewarded, but if the bird hits the pipes or falls to the ground, the round will end and a certain amount of the reward will be lost. For this game, the present invention uses a set of rules to tell the bird not to fly too high or too low when flying over a pair of pipes, the rules are only when flying within the box of Figure 4 (ie the area between opposing upper and lower pipes) Affects bird training.

In this embodiment, the acceleration rule is used, and the effective probability is set as:

P _t =p ₀ ·γ ^t

Where 0<γ<1, p ₀ is a constant, for example, set p ₀ =1, γ=0.8.

Formally, Flappybird's knowledge base is R _fb = {r ₁ , r ₂ }, η(r ₁ ) (where r ₁ represents rule ₁ , r ₂ represents rule 2, and η(r ₁ ) represents the first-order logic propositions) are:

crossing(p _u ,p _l )∧less(distance(bird,p _u ),size(bird)),

And δ(r ₁ ) (representing the recommended action of rule r ₁ ) is:

{flap},

Among them, δ(r ₁ ) represents the recommended action of rule r ₁ . The above rule means that when the bird is between the upper and lower pipes, and the distance between the bird and the upper pipe is greater than the vertical height of a bird, it will fly up.

η(r ₂₎ is:

crossing(p _u ,p _l )∧less(distance(bird,pl ₎ ,size(bird)),

And δ(r ₁₎ is

{null}.

The above rule means that when the bird is between the upper and lower pipes, and the distance between the bird and the lower pipe is greater than the vertical height of a bird, no action will be taken.

where (p _u ,p _l ) represents a pair of pipes the bird is flying over, null means do nothing, and flap means flapping its wings.

The average reward and average Q value of FLappybird using the acceleration rule are shown in Figure 5, where RIL corresponds to the present invention (marked by curve S11), and DQN corresponds to traditional reinforcement learning (marked by curve S12). In the experiment, a time limit for the training phase was set, and the reward for each game increased as the time continued to increase. It can be seen from the average reward shown in Fig. 5 that, in the same training time, compared with the traditional DQN, the RIL of the present invention can obtain better performance with fewer training sets. Moreover, the average Q value also shows that the rules introduced by the present invention can speed up the learning progress.

Referring to Figure 6, in the Spacewar game, enemy aircraft randomly appear from the top of the screen and fly vertically to the bottom of the screen. The agent means that our plane continuously shoots the enemy plane at a certain frequency, and each time it hits the enemy plane, it will get a certain reward. If our plane collides with the enemy plane, the game ends and the reward is lost. In this scene, our plane can only move horizontally, and the available actions are left and right.

For example, the rule set used in this game is a greedy strategy that always moves to the closest bandit in the level.

In this embodiment, the acceleration rule is used, and the effective probability is set as:

P _t =p ₀ ·γ ^t

Where 0<γ<1, set p ₀ =1 and γ=0.8.

Formally, Spacewar's knowledge base is _Raw = {r ₃ , r ₄ }, and η(r ₃ ) is

on_left(nearest_jet,agent),

And δ(r3) is:

{move_left},

The above rule means that when the closest bandit is on the left, move to the left.

η(r4) is:

on_right(nearest_jet,agent),

And δ(r4) is

{move_right}.

The above rule means that when the closest bandit is on the right, move to the right.

Among them, move_left means move left, move_right means move right.

The effect of Spacewar using accelerated rules is shown in Figure 7. It can be seen that the learning speed of the DQN driven by the rule intervention of the present invention is much faster than that of the traditional DQN.

Referring to Figure 8, for the Breakout game, the following rules are used: if the ball is on the left side of the racket, the racket moves to the left, and if the ball is on the right side of the racket, the racket moves to the right.

The acceleration rule is used in this embodiment, and the effective probability is

P _t =p ₀ ·γ ^t ,

Where 0<γ<1, set p ₀ =1 and γ=0.8.

Formally, Breakout's knowledge base is R _bo = {r ₅ , r ₆ }, and η(r ₅ ) is

on_left(ball, paddle),

and δ(r ₅ ) is

{move_left},

The above rule means that when the ball is on the left side of the racket, move left.

η(r ₆ ) is

on_right(ball, paddle),

and δ(r ₆ ) is

{move_right}.

The effect of Breakout using accelerated rules is shown in Figure 9, and the learning effect of the rule-based intervention of the present invention is significantly improved compared to the traditional DQN.

The experimental results of the above three embodiments are shown in Table 1, which shows that the use of accelerated rule sets can effectively improve the learning efficiency.

Table 1: Comparison of experimental results

Referring to Figure 10, for the GridWorld game, which contains destination areas, unreachable walls (such as black grids), and traps, etc., once a trap is caught, the game will end and the agent will be heavily penalized. The ultimate goal of the game is to find the shortest path to your destination without falling into a trap. For example, the agent gets a negative reward of -1 every time it moves. If it falls into a trap, the negative reward is -600, and the reward for reaching the destination is 100.

In this embodiment, safety rules are used to ensure the safety of the agent during training. Safety rules are used to prevent the agent from making catastrophic decisions with irreversible consequences. Unlike the above acceleration rules, the set safety rules are always in effect during the training process.

In this embodiment, a single security rule is adopted in the knowledge base, which is expressed as R _gw ={r ₇ }, and η(r ₇ ) is,

near_trap∧trap_in(directions),

η(r ₇ ) is:

Α-{move(dir):dir∈directions}

The above rule says that when the agent is next to a trap, it chooses an action that removes the direction of the trap.

For example: the optional actions are "up", "down", "left" and "right", and the trap is on the "right" of the agent, then choose 1 of the other 3 actions.

where A is the set of all actions, this rule just prevents the agent from entering the trap, this rule is mandatory.

The GridWorld experimental results, shown in Fig. 11, show that even in the initial stage of training, the performance of the rule-based intervention based learning (RIL) of the present invention is much better than that of the traditional DQN. Due to the safety rules, the agent will never make catastrophic actions, thus guaranteeing the safety of the agent. Using safety rules to avoid catastrophic decisions is especially effective when it comes to cold-driven problems.

To sum up, the present invention can guide the agent to move forward in a more "correct" direction in a specific scenario by combining high-level abstraction rules with deep reinforcement learning. In addition, the set rule set can be individually designed according to specific application scenarios, so that under the premise of improving the generality of the model, it can better meet the individual needs in terms of security, learning time, and learning efficiency. Therefore, by using the present invention, the training time can be shortened and catastrophic decision-making can be avoided, and it can be widely used in the field of dynamic decision-making.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

The computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code, written in any combination, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are all equivalent.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims (8)

A decision-making method based on deep reinforcement learning, including the following steps: The agent makes decisions according to the environmental information and selects the actions after the decision; The agent compares the action after the decision with the knowledge base, and decides whether to replace the action after the decision with the random action in the rule set based on the set rule set in the knowledge base; In the case that it is judged to replace the action after the decision, perform the replaced action in the environment, obtain the reward and new environment information from the environment, and combine the old environment information, action, reward and new environment information into experience information, Stored in the experience playback pool; A set amount of experience information is randomly selected from the experience replay pool to update the deep reinforcement learning model to guide the next iteration. The method according to claim 1, wherein determining whether to replace the decided action with a random action in the rule set according to the set rule set in the knowledge base comprises: Determine whether the rule set in the knowledge base satisfies the predetermined condition; If the set conditions are met, replace the decided action with a random action in the rule set with a set probability. The method according to claim 2, wherein, under the condition that the set condition is satisfied, with the probability of P _t = p ₀ ·γ ^t , a random action in the set of compliant actions α(R, t) is used to replace the post-decision decision where _p0 is the initial rule intervention probability, t is the running time, γ is the decay rate, R is the rule set, and α is all actions that conform to the rule set R and at time t. The method according to claim 1, wherein the rule set is set according to a decision application scenario to avoid catastrophic decision-making or to improve learning efficiency, and is used to guide the action of the agent in the application scenario. The method according to claim 1, wherein combining the old environment information, the action, the reward and the new environment information into one experience information, and storing it into the experience playback pool comprises: After obtaining the new environment information, store one unit of experience information (φ(s _t ), at , r _t , φ(s _{t +1} )) into the experience playback pool D; If the capacity of the experience pool exceeds the set threshold N when new experience information is stored, the earlier experience information will be deleted with reference to the storage time. The method of claim 5, wherein randomly selecting a set number of experience information in the experience replay pool to update the deep reinforcement learning model comprises: In each step t of each round of interaction between the agent and the environment, D randomly selects a certain amount of experience information (φ(s _j ), a _j , r _j , φ(s _j+1 )) in the experience playback pool, And calculate the value of the current moment j of each experience information:

Take (y _j -Q(φ(s _j ), a _j ; θ)) ² as the objective function to do gradient descent to optimize the neural network parameter θ; Finally, every fixed number of steps C, the target action-value function Q* is synchronized to the action-value function Q; Among them, a' represents the optional action at time j+1, a _j represents the action at time j, s _j and s _j+1 represent the environmental information at time j and time j+1 respectively, and φ represents the preprocessing process. A computer-readable storage medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6. A computer device, comprising a memory and a processor, a computer program that can be run on the processor is stored in the memory, and characterized in that, when the processor executes the program, any one of claims 1 to 6 is implemented The steps of the method described in item.

Images