CN113095498A - Divergence-based multi-agent cooperative learning method, divergence-based multi-agent cooperative learning device, divergence-based multi-agent cooperative learning equipment and divergence-based multi-agent cooperative learning medium - Google Patents
Divergence-based multi-agent cooperative learning method, divergence-based multi-agent cooperative learning device, divergence-based multi-agent cooperative learning equipment and divergence-based multi-agent cooperative learning medium Download PDFInfo
- Publication number
- CN113095498A CN113095498A CN202110315995.0A CN202110315995A CN113095498A CN 113095498 A CN113095498 A CN 113095498A CN 202110315995 A CN202110315995 A CN 202110315995A CN 113095498 A CN113095498 A CN 113095498A
- Authority
- CN
- China
- Prior art keywords
- network
- divergence
- strategy
- target
- representing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Feedback Control In General (AREA)
Abstract
The invention discloses a divergence-based multi-agent cooperative learning method, a divergence-based multi-agent cooperative learning device, divergence-based multi-agent cooperative learning equipment and a divergence-based storage medium, wherein the divergence-based multi-agent cooperative learning method comprises the following steps of: initializing a value network, a policy network and a target policy network; changing the updating modes of the value network, the strategy network and the target strategy network according to a preset regular term based on divergence to obtain a latest updating mode; training the plurality of agents according to the value network, the strategy network and the target strategy network to obtain experience data, and updating the value network, the strategy network and the target strategy network according to the experience data and the latest updating mode; and a plurality of agents acquire observation data from the environment, and make a decision by combining the experience data and the updated strategy network to obtain action data. The multi-agent cooperative learning method provided by the embodiment of the disclosure utilizes the regular term based on divergence to enhance the exploration capability of the agents and solve the cooperative problem of the multi-agents.
Description
Technical Field
The invention relates to the technical field of machine learning, in particular to a divergence-based multi-agent cooperative learning method, a divergence-based multi-agent cooperative learning device, divergence-based multi-agent cooperative learning equipment and divergence-based multi-agent cooperative learning media.
Background
Reinforcement learning agents may accomplish autonomous learning of behavioral strategies by interacting with the environment, and thus have been successfully applied to tasks in the single agent domain, such as robotic arm control, board games, and games. However, many tasks in real life often require multiple agents to complete through cooperation, such as logistics robots, unmanned driving, large-scale instant strategy games, and other tasks. Therefore, multi-agent cooperative learning has become more of a concern in recent years.
In a cooperative multi-agent task, each agent typically perceives only local information within its visual range due to communication limitations. It is difficult to form an effective collaboration between agents if each agent learns from their respective local information. In the prior art, the exploration capacity of an agent is improved by adding an entropy regular term, but the original Markov decision process is modified by adding the entropy regular term, so that a convergence strategy obtained by reinforcement learning based on entropy regular is not the optimal strategy of the original problem. Bias may be introduced to the convergence strategy.
Disclosure of Invention
The disclosed embodiments provide a divergence-based multi-agent cooperative learning method, apparatus, device and medium. The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview and is intended to neither identify key/critical elements nor delineate the scope of such embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
In a first aspect, the disclosed embodiments provide a divergence-based multi-agent cooperative learning method, including:
initializing a value network, a policy network and a target policy network;
changing the updating modes of the value network, the strategy network and the target strategy network according to a preset regular term based on divergence to obtain a latest updating mode;
training a plurality of agents according to the value network, the strategy network and the target strategy network to obtain experience data, and updating the value network, the strategy network and the target strategy network according to the experience data and the latest updating mode;
and a plurality of agents acquire observation data from the environment, and make a decision by combining the experience data and the updated strategy network to obtain action data.
In an optional embodiment, before changing the updating manner of the value network, the policy network, and the target policy network according to a preset regular term based on divergence, the method further includes:
and constructing a maximized objective function of the multi-agent according to the divergence-based regularization item.
wherein, pi represents a policy network, atRepresents the motion, stRepresenting the state and p representing the target policy network.
In an optional embodiment, changing the update mode of the value network according to a preset regular term based on divergence includes:
the value network is updated according to the maximized objective function:
where λ is the regular term coefficient, π represents the policy network, ρ represents the target policy network, φ is a parameter of the value network,is a parameter of the target value network, s represents all information in the environment, a represents actions, y represents target values to be fitted, r represents global rewards, E represents mathematical expectations, γ represents discountsA deduction factor, s 'represents the new state to which the environment is transferred after the agent makes a decision, a' represents the action made by the agent in the new state,a loss function representing a network of values,representing a network of target values, QφAnd is used for representing a value network, and tau is used for representing a parameter for performing moving average updating of the target value network.
In an optional embodiment, changing the update mode of the policy network according to a preset regular term based on divergence includes:
the policy network updates according to the minimization objective function:
wherein, thetaiRepresenting a policy network piiIs determined by the parameters of (a) and (b),representing the loss function of the policy network, E representing the mathematical expectation, aiRepresenting an action, s representing a state,representing a function of values given a strategy pi and a target strategy p, lambda representing the regular term coefficient, piRepresenting a target policy network, DKLIndicating KL divergence.
In an optional embodiment, changing the update mode of the target policy network according to a preset regular term based on divergence includes:
the target policy network updates according to the running average of the policy network:
wherein the content of the first and second substances,a parameter representing a target policy network of agent i, τ a parameter of the target policy network for moving average updating, θiRepresenting parameters of the policy network.
In an optional embodiment, the training of the plurality of agents according to the value network, the policy network, and the target policy network to obtain the experience data, and the updating of the value network, the policy network, and the target policy network according to the experience data and the latest updating manner include:
the intelligent agent obtains observation data from the environment, and makes a decision according to a policy network by using experience data to obtain action data;
the environment gives a reward according to the current state and the joint action, and moves to the next state;
storing a multi-element group as experience data into a cache database, wherein the multi-element group comprises a current environment state, a current action, a next environment state, observation data and global rewards;
and after a period of experience is finished, acquiring a plurality of pieces of experience data from the cache database for training, and updating the value network, the strategy network and the target strategy network.
In a second aspect, embodiments of the present disclosure provide a divergence-based multi-agent cooperative learning apparatus, comprising:
the initialization module is used for initializing a value network, a strategy network and a target strategy network;
the change module is used for changing the update modes of the value network, the strategy network and the target strategy network according to a preset regular term based on divergence to obtain a latest update mode;
the training module is used for training the plurality of agents according to the value network, the strategy network and the target strategy network to obtain experience data, and updating the value network, the strategy network and the target strategy network according to the experience data and the latest updating mode;
and the execution module is used for acquiring observation data from the environment by a plurality of agents, and making a decision by combining the experience data and the updated strategy network to obtain action data.
In a third aspect, the disclosed embodiments provide a divergence-based multi-agent cooperative learning device, comprising a processor and a memory storing program instructions, the processor being configured to execute the divergence-based multi-agent cooperative learning method provided by the above embodiments when executing the program instructions.
In a fourth aspect, embodiments of the present disclosure provide a computer-readable medium having computer-readable instructions stored thereon, the computer-readable instructions being executable by a processor to implement a divergence-based multi-agent cooperative learning method provided by the above-mentioned embodiments.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:
the disclosed embodiment provides a cooperative learning method of multiple agents, which is used for solving the cooperative problem of the multiple agents. The method adds a target strategy to the intelligent agent, adds a regular term based on divergence in a reward function of a general Markov decision process by using the target strategy, and provides a new strategy updating mode aiming at the Markov decision process based on divergence. And after the regular item is added, off-strategy training can be realized, and the problem of sampling efficiency in a multi-agent environment is solved. Meanwhile, the regular term also controls the step length of strategy updating, and the stability of strategy improvement is enhanced. The method provided by the embodiment of the disclosure has certain flexibility and has important application value in the fields of intelligent transportation, robot control and the like.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a schematic flow diagram illustrating a divergence-based multi-agent cooperative learning method in accordance with an exemplary embodiment;
FIG. 2 is a schematic flow diagram illustrating a divergence-based multi-agent training method in accordance with an exemplary embodiment;
FIG. 3 is a schematic diagram illustrating the architecture of a divergence-based multi-agent cooperative learning apparatus in accordance with an exemplary embodiment;
FIG. 4 is a schematic diagram illustrating a configuration of a divergence-based multi-agent cooperative learning device, according to an exemplary embodiment;
FIG. 5 is a schematic diagram illustrating a computer storage medium in accordance with an exemplary embodiment.
Detailed Description
The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.
It should be understood that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of systems and methods consistent with certain aspects of the invention, as detailed in the appended claims.
In the description of the present invention, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
In a multi-agent collaboration problem, the state of the environment, i.e. all information in the environment, is s. Each agent i can obtain partial information o in the environmentiBased on the history information τi={(oi,ai) And its own policy piiGet and execute own action aiAccording to the current state s and the joint action a of all agents (a ═ a)1,a2,…,an) And gets the same global reward r (s, a) while the context transitions to the next state s' P (· | s, a). Remember the union policy of all agents as pi ═ pi1×π2×…×πnThen each agent maximizes the common objective function as:
wherein E represents mathematical expectation, π represents strategy, atRepresents the motion, stIndicating the status and gamma a discount factor.
The reinforced learning based on entropy regularization is mainly to subtract log pi (a) from the reward functiont|st) I.e. modifying the optimization objective of the agent to:
wherein E represents mathematical expectation, π represents strategy, atRepresents the motion, stRepresenting the state, gamma representing the discount factor, and lambda representing the coefficient of the regularization term. The addition of the entropy regulation term can improve the exploration of the intelligent agentThe method has the advantages that the conventional Markov decision process is modified by the entropy regular term, so that the convergence strategy obtained by reinforcement learning based on entropy regular is not the optimal strategy of the original problem, and deviation is brought to the convergence strategy.
The embodiment of the disclosure provides a new regular term based on divergence, and provides a new strategy updating mode aiming at the Markov decision process based on divergence, the exploration capacity of the intelligent agent is enhanced by using the regular term based on divergence, and meanwhile, when the strategy of the intelligent agent is converged, the regular term disappears, and the deviation brought to the convergence strategy by adding the regular term is avoided. And after the regular item is added, off-strategy training can be realized, and the problem of sampling efficiency in a multi-agent environment is solved. Meanwhile, the regular term also controls the step length of strategy updating, and the stability of strategy improvement is enhanced.
The divergence-based multi-agent cooperative learning method provided by the embodiment of the present application will be described in detail below with reference to fig. 1 to 2.
The method is suitable for the situation that a plurality of cooperative intelligent agents exist, each intelligent agent interacts with the environment in the training process, in real life, a plurality of tasks are often completed by the cooperation of the plurality of intelligent agents, such as logistics robots, unmanned driving, large-scale instant strategy games and the like, and the system which can complete the tasks by the cooperation of the plurality of intelligent agents is called a multi-intelligent-agent system, namely a multi-intelligent-agent system. For example, a warehouse logistics system is a multi-agent, wherein each logistics robot is an agent.
Referring to fig. 1, the method specifically includes the following steps.
S101 initialization value network Q and strategy network piiAnd target policy network ρi。
S102, changing the updating modes of the value network, the strategy network and the target strategy network according to a preset regular term based on divergence to obtain a latest updating mode.
In a possible implementation, before executing step S102, constructing a maximization objective function of the multi-agent according to the divergence-based regularization term is further included.
Specifically, a target strategy ρ is added to each agentiSubtracting one from the reward function using the newly added objective strategyThe term is called a divergence-based regularization term. Wherein, pi represents a policy network, atRepresents the motion, stRepresenting the state and p representing the target policy network.
Further, the maximum objective function of the multi-agent is obtained by using the divergence-based regularization term as follows:
wherein E represents mathematical expectation, π represents strategy, atRepresents the motion, stRepresenting the state, gamma representing the discount factor, lambda representing the coefficient of the regularization term, and p representing the target policy network.
The embodiment of the disclosure changes the entropy regular term into the regular term based on the divergence, and obtains a new optimization objective function by using the regular term based on the divergence. When the target strategy is taken as the past strategy, the regular item can give a reward to reduce when the probability of a certain action is increased, otherwise the reward can be improved, namely the regular item encourages the intelligent agent to take an action with the decreased probability and discourages the intelligent agent from taking an action with the increased probability, so that the intelligent agent can be more fully explored in the training process. Meanwhile, the addition of the regular term also controls the step length of strategy updating actually, namely, the new strategy cannot deviate from the past strategy too much, and the stability of strategy improvement is facilitated. And after the regular term based on divergence is added, the latest updating mode is obtained according to derivation, and off-policy (off-policy) training can be realized, so that the problem of sampling efficiency faced by the reinforced learning method based on the policy under the condition of multiple intelligent agents is solved.
Specifically, changing the update mode of the value network according to a preset regular term based on divergence includes:
in general reinforcement learning, a value function is generally defined as:
wherein Q represents a value function, π represents a policy, atRepresents the motion, stRepresenting the status, gamma representing the discount factor, E representing the mathematical expectation, and r representing the global reward.
During the training process, the updating is usually performed according to the bellman optimal equation, that is:
wherein Q is*Representing an updated value function, pi representing a policy, a representing an action, s representing a state, gamma representing a discount factor, E representing a mathematical expectation, r representing a global reward, s 'representing a state at a next time, a' representing an action at a next time.
After adding the regularization term based on divergence, a new iterative formula for the optimal value function is obtained, namely:
wherein the content of the first and second substances,representing an optimal value function, a representing an action, s representing a state, γ representing a discount factor, E representing a mathematical expectation, r representing a global reward, s 'representing a state at a next time, a' representing an action at a next time, λ representing a coefficient of a regularization term, ρ representing a target policy network,representing the optimal strategy. This formula, in addition to adding a regularization term based on divergence, is not comparable to the Bellman optimal equationThe limit a' is the optimal action which can be selected arbitrarily, so that the flexibility in training is increased, and meanwhile, the objective function design of TD (lambda) can be adopted in off-strategy training.
According to the above iterative formula, the designed value network is updated according to the maximized objective function:
where λ is the regular term coefficient, π represents the policy network, ρ represents the target policy network, φ is a parameter of the value network,is a parameter of the network of target values, s represents all information in the environment, a represents actions, y represents target values to be fitted, r represents global rewards, E represents mathematical expectations, γ represents a discount factor, s 'represents a new state to which the environment is transferred after the agent makes a decision, a' represents actions made by the agent in the new state,a loss function representing a network of values,targeting of a network of target values, QφAnd is used for representing a value network, and tau is used for representing a parameter for performing moving average updating of the target value network.
Further, changing the updating mode of the policy network according to a preset regular term based on divergence, comprising:
policy-based reinforcement learning methods are usually rootThe policy network update mode is obtained according to the policy gradient theorem, but the policy gradient theorem limits that the update must be in policy (on-policy). After adding the regular term based on divergence, a new strategy lifting theorem is obtained, namely an old strategy pi is givenoldOnly new strategy is needednewSatisfy the requirement of
WhereinDKLRepresents the KL divergence, then pinewIs just to pioldMore preferably. According to the above theorem, the gradient of the strategy can be calculated by deducing according to the following formula:
wherein, thetaiRepresenting a policy network piiIs determined by the parameters of (a) and (b),representing a cache database for storing training data,representing the loss function of the policy network, E representing the mathematical expectation, aiRepresenting an action, s representing a state,representing a value function given a policy pi and a target policy p, where lambda represents the regularization term coefficient and p represents the target policy network.
The above gradient formula, except for adding the regular term, is the biggest difference from the policy gradient theorem in that it no longer limits the distribution of the state s, so that the off-policy update can be performed by using the above update mode. In a multi-agent complexThe counter-fact baseline function can be set to reduce the computation in addition to the credit allocation problem by subtracting a counter-fact baseline function from the gradient and reducing the variance of the gradient update. Taking the baseline function asThe gradient formula can be further rewritten as follows, and the update method of the policy network according to the minimization objective function is also shown as the following formula:
wherein, thetaiRepresenting a policy network piiIs determined by the parameters of (a) and (b),representing the loss function of the policy network, E representing the mathematical expectation, aiRepresenting an action, s representing a state,representing a function of values given a strategy pi and a target strategy p, lambda representing the regular term coefficient, piRepresenting a target policy network, DKLIndicating KL divergence.
Further, changing the updating mode of the target policy network according to a preset regular term based on divergence, comprising:
for the objective strategy, in the most ideal case, the following iterative approach should be used:
i.e. taking the constant rho ═ pitUnder the condition of obtaining the optimal strategy pit+1When such iterative process converges, pit+1=πtAnd then haveTherefore, the convergence strategy is the optimal strategy of the original Markov decision process, and the deviation brought to the convergence strategy by adding the regular term is avoided. However, such an ideal case requires training through reinforcement learning until convergence every iteration, and is too costly, so the following approximation method can be adopted:
wherein the content of the first and second substances,a parameter representing a target policy network of agent i, τ a parameter of the target policy network for moving average updating, θiRepresenting parameters of the policy network. I.e., one-step gradient update instead of maximization, while letting the target strategy take the moving average of the strategy.
The target policy network updates according to the running average of the policy network:
wherein the content of the first and second substances,a parameter representing a target policy network of agent i, τ a parameter of the target policy network for moving average updating, θiRepresenting parameters of the policy network.
According to the step, the latest updating mode of the value network, the strategy network and the target strategy network can be obtained according to the regular item based on the divergence, and through the latest updating mode, when the strategy of the intelligent agent is converged, the regular item disappears, so that the deviation of the convergence strategy caused by adding the regular item is avoided, the off-strategy training can be realized, the problem of sampling efficiency in the multi-intelligent-agent environment is solved, the strategy updating step length is controlled, and the strategy improving stability is enhanced.
S103, training the plurality of agents according to the value network, the strategy network and the target strategy network to obtain experience data, and updating the value network, the strategy network and the target strategy network according to the experience data and the latest updating mode.
Fig. 2 is a flow diagram illustrating a divergence-based multi-agent training method according to an exemplary embodiment, as shown in fig. 2, the multi-agent training process includes: s201, an intelligent agent obtains observation data from the environment, and makes a decision according to a policy network by using experience data to obtain action data; s202, the environment gives rewards according to the current state and the joint action, and moves to the next state; s203, storing a multi-element group as experience data into a cache database, wherein the multi-element group comprises a current environment state, a current action, a next environment state, observation data and global rewards; s204, after a period of experience is finished, a plurality of pieces of experience data are obtained from the buffer database for training, and the value network, the strategy network and the target strategy network are updated.
In particular, the training data is obtained by the agent interacting with the environment, i.e. agent i obtains observation data o from the environmentiAnd using empirical data τiAccording to a strategy ofiTo make a decision to obtain an action aiThe context then gives the reward r according to the current state s and the associated action a and moves to the next state s ', storing the tuple (s, a, s', o, r) as a piece of experience data in the cache database D. After a period of experience is over, a plurality of experiences are taken from D for training, and the value network Q and the strategy network pi are updated according to the latest updating modeiAnd a target policy network ρi。
In one exemplary scenario, at game star dispute 2, a player will maneuver several units to combat an enemy, with the goal of eliminating all units of the enemy. In the training, each unit is regarded as an agent, each agent has a visual field range, the obtained observation data is the relevant indexes (such as life value, shield value, unit type and the like) of the unit in the visual field range, the actions which can be taken by the agent comprise movement, attack, release skill and the like, the strategy of the agent is generated by a strategy network and is a probability distribution of the actions which can be taken by the agent under the current state, the agent can obtain rewards when hurting or destroying the enemy unit, the state of each step (data corresponding to all information of the current game and only available during the training), the observation of all agents, the actions of all agents, the rewards obtained after the actions are taken and new states transferred to are stored as experience, after a plurality of steps are executed or a game is finished, the experience data obtained before is taken out for training, the strategy of the intelligent agent is updated, new experience is obtained by using the new strategy, and the process is repeated until the intelligent agent obtains better performance.
S104, a plurality of agents acquire observation data from the environment, and make a decision by combining the experience data and the updated strategy network to obtain action data.
Further, in execution, for agent i: each time observation data o is acquired from the environmentiIn combination with empirical data τiAccording to the updated strategy network piiTo make a decision to obtain an action ai。
According to the cooperative learning method of the multi-agent, provided by the embodiment of the disclosure, the cooperative problem of the multi-agent is solved, the exploration capacity of the agent is enhanced by utilizing the regular item based on divergence, and meanwhile, when the strategy of the agent is converged, the regular item disappears, so that the deviation of the convergence strategy caused by adding the regular item is avoided. And after the regular item is added, off-strategy training can be realized, and the problem of sampling efficiency in a multi-agent environment is solved. Meanwhile, the regular term also controls the step length of strategy updating, and the stability of strategy improvement is enhanced. The method provided by the embodiment of the disclosure has certain flexibility, for example, in the field of robot control, the problem of cooperative control of multiple robots can be solved, and in the field of games, the method can solve the cooperative control of multiple game roles and serve as an artificial intelligence system of multiple intelligent agents in games.
The disclosed embodiment also provides a divergence-based multi-agent cooperative learning apparatus, configured to perform the divergence-based multi-agent cooperative learning method of the foregoing embodiment, as shown in fig. 3, the apparatus includes:
an initialization module 301, configured to initialize a value network, a policy network, and a target policy network;
a changing module 302, configured to change the update modes of the value network, the policy network, and the target policy network according to a preset regular term based on divergence, to obtain a latest update mode;
the training module 303 is configured to train the multiple agents according to the value network, the policy network, and the target policy network to obtain experience data, and update the value network, the policy network, and the target policy network according to the experience data and the latest update mode;
and the execution module 304 is configured to obtain observation data from the environment by the multiple agents, and make a decision by combining the experience data and the updated policy network to obtain action data.
It should be noted that, when the divergence-based multi-agent cooperative learning apparatus provided in the foregoing embodiment executes the divergence-based multi-agent cooperative learning method, only the division of the function modules is illustrated, and in practical applications, the function distribution may be completed by different function modules according to needs, that is, the internal structure of the device may be divided into different function modules, so as to complete all or part of the functions described above. In addition, the divergence-based multi-agent cooperative learning device provided by the above embodiment and the divergence-based multi-agent cooperative learning method embodiment belong to the same concept, and the detailed implementation process thereof is referred to as the method embodiment, which is not described herein again.
The embodiment of the present disclosure further provides an electronic device corresponding to the divergence-based multi-agent cooperative learning method provided in the foregoing embodiment, so as to execute the divergence-based multi-agent cooperative learning method.
Referring to fig. 4, a schematic diagram of an electronic device provided in some embodiments of the present application is shown. As shown in fig. 4, the electronic apparatus includes: a processor 400, a memory 401, a bus 402 and a communication interface 403, wherein the processor 400, the communication interface 403 and the memory 401 are connected through the bus 402; the memory 401 has stored therein a computer program executable on the processor 400, the processor 400 executing the divergence-based multi-agent cooperative learning method provided by any of the foregoing embodiments of the present application when executing the computer program.
The Memory 401 may include a high-speed Random Access Memory (RAM) and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 403 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.
The electronic device provided by the embodiment of the application and the divergence-based multi-agent cooperative learning method provided by the embodiment of the application have the same beneficial effects as the method adopted, operated or realized by the electronic device.
Referring to fig. 5, the computer readable storage medium is shown as an optical disc 500, on which a computer program (i.e., a program product) is stored, and when the computer program is executed by a processor, the computer program performs the divergence-based multi-agent cooperative learning method provided by any of the foregoing embodiments.
It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, or other optical and magnetic storage media, which are not described in detail herein.
The computer-readable storage medium provided by the above-mentioned embodiment of the present application and the divergence-based multi-agent cooperative learning method provided by the embodiment of the present application have the same beneficial effects as the method adopted, operated or implemented by the application program stored in the computer-readable storage medium.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only show some embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A divergence-based multi-agent cooperative learning method is characterized by comprising the following steps:
initializing a value network, a policy network and a target policy network;
changing the updating modes of the value network, the strategy network and the target strategy network according to a preset regular term based on divergence to obtain a latest updating mode;
training the plurality of agents according to the value network, the strategy network and the target strategy network to obtain experience data, and updating the value network, the strategy network and the target strategy network according to the experience data and the latest updating mode;
and a plurality of agents acquire observation data from the environment, and make a decision by combining the experience data and the updated strategy network to obtain action data.
2. The method of claim 1, wherein before changing the updating modes of the value network, the policy network and the target policy network according to a preset regular term based on divergence, the method further comprises:
and constructing a maximized objective function of the multi-agent according to the divergence-based regularization item.
4. The method of claim 1, wherein changing the update mode of the value network according to a preset regularization term based on divergence comprises:
the value network is updated according to the maximized objective function:
where λ is the regular term coefficient, π represents the policy network, ρ represents the target policy network, φ is a parameter of the value network,is a parameter of the network of target values, s represents all information in the environment, a represents actions, y represents target values to be fitted, r represents global rewards, E represents mathematical expectations, γ represents a discount factor, s 'represents a new state to which the environment is transferred after the agent makes a decision, a' represents actions made by the agent in the new state,a loss function representing a network of values,representing a network of target values, QφAnd is used for representing a value network, and tau is used for representing a parameter for performing moving average updating of the target value network.
5. The method of claim 1, wherein changing the update mode of the policy network according to a preset divergence-based regularization term comprises:
the policy network updates according to the minimization objective function:
wherein, thetaiRepresenting a policy network piiIs determined by the parameters of (a) and (b),representing the loss function of the policy network, E representing the mathematical expectation, aiRepresenting an action, s representing a state,representing a function of values given a strategy pi and a target strategy p, lambda representing the regular term coefficient, piRepresenting a target policy network, DKLIndicating KL divergence.
6. The method of claim 1, wherein changing the update mode of the target policy network according to a preset divergence-based regularization term comprises:
the target policy network updates according to the running average of the policy network:
7. The method of claim 1, wherein training agents according to the value network, the policy network, and the target policy network to obtain experience data, and updating the value network, the policy network, and the target policy network according to the experience data and the latest update method comprises:
the intelligent agent obtains observation data from the environment, and makes a decision according to a policy network by using experience data to obtain action data;
the environment gives a reward according to the current state and the joint action, and moves to the next state;
storing a multi-element group as experience data into a cache database, wherein the multi-element group comprises a current environment state, a current action, a next environment state, observation data and global rewards;
and after a period of experience is finished, acquiring a plurality of pieces of experience data from the cache database for training, and updating the value network, the strategy network and the target strategy network.
8. A divergence-based multi-agent cooperative learning apparatus, comprising:
the initialization module is used for initializing a value network, a strategy network and a target strategy network;
the change module is used for changing the update modes of the value network, the strategy network and the target strategy network according to a preset regular term based on divergence to obtain a latest update mode;
the training module is used for training the plurality of agents according to the value network, the strategy network and the target strategy network to obtain experience data, and updating the value network, the strategy network and the target strategy network according to the experience data and the latest updating mode;
and the execution module is used for acquiring observation data from the environment by a plurality of agents, and making a decision by combining the experience data and the updated strategy network to obtain action data.
9. A divergence-based multi-agent cooperative learning apparatus, comprising a processor and a memory storing program instructions, the processor being configured to, upon execution of the program instructions, perform a divergence-based multi-agent cooperative learning method as claimed in any one of claims 1 to 7.
10. A computer readable medium having computer readable instructions stored thereon which are executable by a processor to implement a divergence-based multi-agent cooperative learning method as claimed in any one of claims 1 to 7.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110315995.0A CN113095498B (en) | 2021-03-24 | 2021-03-24 | Divergence-based multi-agent cooperative learning method, divergence-based multi-agent cooperative learning device, divergence-based multi-agent cooperative learning equipment and divergence-based multi-agent cooperative learning medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110315995.0A CN113095498B (en) | 2021-03-24 | 2021-03-24 | Divergence-based multi-agent cooperative learning method, divergence-based multi-agent cooperative learning device, divergence-based multi-agent cooperative learning equipment and divergence-based multi-agent cooperative learning medium |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113095498A true CN113095498A (en) | 2021-07-09 |
CN113095498B CN113095498B (en) | 2022-11-18 |
Family
ID=76669465
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110315995.0A Active CN113095498B (en) | 2021-03-24 | 2021-03-24 | Divergence-based multi-agent cooperative learning method, divergence-based multi-agent cooperative learning device, divergence-based multi-agent cooperative learning equipment and divergence-based multi-agent cooperative learning medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113095498B (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113391556A (en) * | 2021-08-12 | 2021-09-14 | 中国科学院自动化研究所 | Group distributed control method and device based on role distribution |
CN113780577A (en) * | 2021-09-07 | 2021-12-10 | 中国船舶重工集团公司第七0九研究所 | Layered decision-making complete cooperation multi-agent reinforcement learning method and system |
CN114418128A (en) * | 2022-03-25 | 2022-04-29 | 新华三人工智能科技有限公司 | Model deployment method and device |
CN115494844A (en) * | 2022-09-26 | 2022-12-20 | 成都朴为科技有限公司 | Multi-robot searching method and system |
CN115660110A (en) * | 2022-12-26 | 2023-01-31 | 中国科学院自动化研究所 | Multi-agent credit allocation method, device, readable storage medium and agent |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180032863A1 (en) * | 2016-07-27 | 2018-02-01 | Google Inc. | Training a policy neural network and a value neural network |
US20200125957A1 (en) * | 2018-10-17 | 2020-04-23 | Peking University | Multi-agent cooperation decision-making and training method |
CN111514585A (en) * | 2020-03-17 | 2020-08-11 | 清华大学 | Method and system for controlling agent, computer device, and storage medium |
CN111612126A (en) * | 2020-04-18 | 2020-09-01 | 华为技术有限公司 | Method and device for reinforcement learning |
-
2021
- 2021-03-24 CN CN202110315995.0A patent/CN113095498B/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180032863A1 (en) * | 2016-07-27 | 2018-02-01 | Google Inc. | Training a policy neural network and a value neural network |
US20200125957A1 (en) * | 2018-10-17 | 2020-04-23 | Peking University | Multi-agent cooperation decision-making and training method |
CN111514585A (en) * | 2020-03-17 | 2020-08-11 | 清华大学 | Method and system for controlling agent, computer device, and storage medium |
CN111612126A (en) * | 2020-04-18 | 2020-09-01 | 华为技术有限公司 | Method and device for reinforcement learning |
Non-Patent Citations (5)
Title |
---|
DEMING YUAN等: "Distributed Mirror Descent for Online Composite Optimization", 《 IEEE TRANSACTIONS ON AUTOMATIC CONTROL》 * |
JIECHUAN JIANG和ZONGQING LU: "learning fairness in multi-agent systems", 《ARXIV.ORG/ABS/1910.14472》 * |
ZHANG-WEI HONG等: "A Deep Policy Inference Q-Network for Multi-Agent Systems", 《PROCEEDINGS OF THE 17TH INTERNATIONAL CONFERENCE ON AUTONOMOUS AGENTS AND MULTIAGENT SYSTEMS》 * |
孙长银等: "多智能体深度强化学习的若干关键科学问题", 《自动化学报》 * |
曲昭伟等: "考虑博弈的多智能体强化学习分布式信号控制", 《交通运输***工程与信息》 * |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113391556A (en) * | 2021-08-12 | 2021-09-14 | 中国科学院自动化研究所 | Group distributed control method and device based on role distribution |
CN113391556B (en) * | 2021-08-12 | 2021-12-07 | 中国科学院自动化研究所 | Group distributed control method and device based on role distribution |
CN113780577A (en) * | 2021-09-07 | 2021-12-10 | 中国船舶重工集团公司第七0九研究所 | Layered decision-making complete cooperation multi-agent reinforcement learning method and system |
CN113780577B (en) * | 2021-09-07 | 2023-09-05 | 中国船舶重工集团公司第七0九研究所 | Hierarchical decision complete cooperation multi-agent reinforcement learning method and system |
CN114418128A (en) * | 2022-03-25 | 2022-04-29 | 新华三人工智能科技有限公司 | Model deployment method and device |
CN114418128B (en) * | 2022-03-25 | 2022-07-29 | 新华三人工智能科技有限公司 | Model deployment method and device |
CN115494844A (en) * | 2022-09-26 | 2022-12-20 | 成都朴为科技有限公司 | Multi-robot searching method and system |
CN115660110A (en) * | 2022-12-26 | 2023-01-31 | 中国科学院自动化研究所 | Multi-agent credit allocation method, device, readable storage medium and agent |
Also Published As
Publication number | Publication date |
---|---|
CN113095498B (en) | 2022-11-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN113095498B (en) | Divergence-based multi-agent cooperative learning method, divergence-based multi-agent cooperative learning device, divergence-based multi-agent cooperative learning equipment and divergence-based multi-agent cooperative learning medium | |
Subramanian et al. | Multi type mean field reinforcement learning | |
Hou et al. | An evolutionary transfer reinforcement learning framework for multiagent systems | |
US20190220750A1 (en) | Solution search processing apparatus and solution search processing method | |
CN113093727A (en) | Robot map-free navigation method based on deep security reinforcement learning | |
CN111898770B (en) | Multi-agent reinforcement learning method, electronic equipment and storage medium | |
Sinapov et al. | Learning inter-task transferability in the absence of target task samples | |
WO2016107426A1 (en) | Systems and methods to adaptively select execution modes | |
US20190354100A1 (en) | Bayesian control methodology for the solution of graphical games with incomplete information | |
CN116560239B (en) | Multi-agent reinforcement learning method, device and medium | |
CN116136945A (en) | Unmanned aerial vehicle cluster countermeasure game simulation method based on anti-facts base line | |
CN116776929A (en) | Multi-agent task decision method based on PF-MADDPG | |
Park et al. | Predictable mdp abstraction for unsupervised model-based rl | |
Shen | Bionic communication network and binary pigeon-inspired optimization for multiagent cooperative task allocation | |
US20150150011A1 (en) | Self-splitting of workload in parallel computation | |
CN116088586A (en) | Method for planning on-line tasks in unmanned aerial vehicle combat process | |
Voss et al. | Playing a strategy game with knowledge-based reinforcement learning | |
CN113599832A (en) | Adversary modeling method, apparatus, device and storage medium based on environment model | |
Kim et al. | Disentangling successor features for coordination in multi-agent reinforcement learning | |
Xu et al. | Cascade attribute learning network | |
CN116489193B (en) | Combat network self-adaptive combination method, device, equipment and medium | |
Lu et al. | Optimal Cost Constrained Adversarial Attacks for Multiple Agent Systems | |
Liu et al. | Soft-actor-attention-critic based on unknown agent action prediction for multi-agent collaborative confrontation | |
Perepu et al. | DSDF: Coordinated look-ahead strategy in multi-agent reinforcement learning with noisy agents | |
Jensen et al. | Industrial policy for advanced ai: Compute pricing and the safety tax |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |