CN113363997B - Reactive voltage control method based on multi-time scale and multi-agent deep reinforcement learning - Google Patents

Abstract

The invention relates to a power system operation and optimization technology, and aims to provide a reactive voltage control method based on multi-time scale and multi-agent deep reinforcement learning. The invention defines the on-load tap changing, the capacitor bank and the energy storage in the photovoltaic, the fan and the load as the intelligent agent, applies the reinforcement learning-based method to the reactive power optimization problem, and allows the controller to learn the control strategy through the interaction with the simulation model of the similar system. The action variables of the reactive power adjusting equipment are interacted with the power distribution network environment, and the intelligent agent can finally achieve the optimal response to the external environment, so that the maximum return value is obtained. The strategic function and the action value function of the intelligent agent are analyzed and fitted by a neural network method, and the training process does not depend on the prediction data result and the accurate trend modeling; by using the reactive power optimization method of two time scales, the network loss is smaller, the voltage stabilizing effect is better, and the effect of improving the safety and reliability of the distribution network is more remarkable.

Description

Reactive voltage control method based on multi-time scale and multi-agent deep reinforcement learning

Technical Field

The invention relates to the technical field of operation and optimization of power systems, in particular to a reactive voltage optimization method based on multi-time scale multi-agent deep reinforcement learning.

Background

With the access of a large number of renewable distributed power sources to a power distribution network, the random fluctuation of the output of wind power equipment and photovoltaic equipment and the uncertain fluctuation of loads can cause the problems of large voltage fluctuation, line crossing of voltage, improvement of network loss and the like of the operation of the power distribution network, and the quality of electric energy is influenced.

The reactive power optimization of the power distribution network aims to effectively ensure the stability of the voltage of each node, reduce the voltage fluctuation and reduce the network loss of the power distribution network under the condition of fully meeting the safe operation constraint of the power distribution network. Reactive power optimization of a power distribution network often involves a number of different variables, a constraint, often considered as a problem in non-linear planning. Although most of the existing model-based reactive voltage optimization methods can effectively inhibit the voltage out-of-limit problem, the existing model-based reactive voltage optimization methods depend on accurate models and prediction data to a great extent, are slow in calculation speed and are prone to falling into local optimization.

Disclosure of Invention

The invention aims to solve the problem of overcoming the defects in the prior art and provides a reactive voltage control method based on multi-time scale and multi-agent deep reinforcement learning.

In order to solve the technical problem, the solution of the invention is as follows:

the reactive voltage control method based on multi-time scale and multi-agent deep reinforcement learning is provided and comprises the following steps:

(1) dividing a reactive voltage control process of a power distribution network accessed to a renewable distributed power supply into a first time scale stage and a second time scale stage; wherein, the first 1 hour is used as the scheduling period of the first time scale stage; the control process is a second time scale stage, and every 1 minute is taken as a scheduling period in the stage;

(2) defining an on-load voltage regulation tap (OLTC), a Capacitor Bank (CB) and an Energy Storage (ES) as an intelligent agent, and building an interactive training environment of a Markov decision process of an environment and intelligent agent interaction at a first time scale stage; in the interactive training of the process, prediction data of photovoltaic, fan and load are input, and a DDQN algorithm (Double Q Network) is adopted to perform offline training of a reactive power optimization discrete action strategy; after training is finished, obtaining scheduling strategies of the OLTC, CB and ES of the intelligent agent, and taking the optimal reactive power control strategy generated by training as the input of a second time scale scheduling stage;

in a first time scale scheduling phase, describing a markov process of interaction between an environment and an agent OLTC, CB and ES as < S, R, a, P, γ >; wherein S represents a system state space and is a set of all states that the intelligent can sense; r is a return space, which is a collection of returns to the agent according to the state actions; a is an action space which is a collection of actions of a decision-making main body on the environment; p is the state transition probability; γ is the discount rate in return, representing the conversion factor for future returns;

(3) in a second time scale stage, establishing an interactive training environment based on a Markov decision process in the same way based on the optimal reactive power control strategy obtained in the step (2); in the training process, a multi-agent deep reinforcement learning algorithm (MA-SAC) based on maximum entropy is adopted for training and optimizing an offline reactive voltage model;

in a second time scale scheduling phase, an interactive training environment based on a Markov process is built in the same way, and the process is described as the interactive training environment by tuples<N，S，a₁，a₂，…，a_N，T，γ，r₁，…，r_N>(ii) a Wherein N represents the number of agents, S represents the system state, a₁，a₂，…，a_NAn action set for an agent; t represents a state transition function, T: s x a₁×…a_N×S→[0，1]Giving the probability of the next state according to the current system state and the joint action; γ is a discount factor; in addition, with r_i(s^t，a₁，…，a_N，s^t+1) Indicating the reward resulting from performing the join action s' while agent i is in state s.

(4) And deploying the reactive voltage model trained by the MA-SAC algorithm to a power distribution network with an online decision, and generating reactive output of intelligent agents in a photovoltaic system, a fan and an energy storage inverter in real-time scheduling by taking 1 minute as a time scale, so as to correct a scheduling result of reactive optimization in a first time scale scheduling stage, and further to relieve voltage fluctuation.

In the present invention, the step (2) specifically includes:

(2.1) under the condition of meeting the constraint conditions of the voltage of the power grid and the operation of the reactive compensation equipment, the reactive optimization objective function of the power distribution network is defined as the total active power loss P of the power distribution network by adjusting the output of an inverter of an intelligent agent of the reactive compensation equipment_lossThe minimum, and the constraint condition includes the upper and lower limit constraints of node voltage, reactive power and action quantity change and the constraint of a power flow equation;

(2.2) establishing an interactive training environment of a Markov decision process according to the reactive voltage optimization objective function and the model of the constraint condition; and taking the intelligent agent optimal small-scale control strategy of the trained DDQN deep reinforcement learning reactive voltage optimization scheduling model as the input of the real-time scheduling strategy of the second-stage time scale.

The step (2.2) specifically comprises:

(2.2.1) in the process of training the intelligent agent, according to the state adjustment strategy function in the power distribution system, taking control measures aiming at given operating conditions to realize reactive power optimization; for a given action by a multi-agent, the environment provides the voltage on all buses in the power distribution system as the state of the DDQN model; the concrete expression is as follows:

s＝{U_i，W_i，C_wi}

wherein, U_iA node voltage matrix of the power distribution network in the ith decision stage; w_iSwitching gears of each regulating device in the ith scheduling period; c_wiAdjusting the actions completed by each device in i scheduling periods;

(2.2.2) constructing motion vectors of the on-load tap changer (OLTC), the Capacitor Bank (CB) and the Energy Storage (ES):

a＝{T_ol，T_cb，T_es}

wherein, T_ol，T_cb，T_esThe tap gear of the OLTC, the switching group number of the capacitor group CB and the reactive power output of the ES are respectively;

(2.2.3) adopting a DDQN algorithm to train a reactive power optimization model in an off-line manner, taking the state as the input of a neural network, calculating all action value functions by using the neural network, and directly outputting a corresponding action-value Q value for evaluating the expected value of a certain action behavior in the current given state;

(2.2.4) defining the target function of the DDQN algorithm reinforcement learning as follows:

Q^*(s，a)＝E_s′～e[r+γmax_a′Q^*(s′，a′)|s，a]

wherein the action cost function Q^*(s, a) is the maximum value of all action values; r is the corresponding reward observed, γ is the discount factor for each step in the future, and s 'and a' are the next state and actions that may be taken, respectively;

(2.2.5) initializing the capacity of the multiplexing pool D to be N; the Q value corresponding to the initialization action is random, and an optimal reactive power compensation strategy is generated according to the reactive power optimization model; the method specifically comprises the following steps:

a. initialization sequence s₁＝{x₁And the preprocessing sequence of the first state phi₁＝φ(s₁) (ii) a Wherein x is₁Is in a first state;

b. randomly selecting an action a with a probability of epsilon_tOtherwise, select action a_t＝max_a Q^*(s', a; θ); wherein, theta is the weight of the neural network function, s' is the next state, and a is the current action;

c. performing actions in an electrical distribution networkAs a_tAnd observe the corresponding r_tAnd state s_t+1(ii) a Let the next state s_t+1＝s_t，a_t，x_t+1Next preprocessing sequence phi_t+1＝φ(s_t+1)；

d. Samples are stored in an empirical multiplexing pool D and small batches of samples (phi) are randomly drawn therefrom_j，a_j，r_j，φ_j+1) (ii) a Order to

Wherein r is_jGamma is the discount factor per step in the future, s, for observing the corresponding reward_t+1For the next observation state, θ_tIs a parameter of the current neural network, θ'_tParameters of the neural network in the next step;

e. to pair

Performing gradient descent and repeating the step b;

and (2.2.6) taking the intelligent agent optimal small-scale control strategy of the reactive voltage optimization scheduling model trained in the first time scale scheduling stage as the input of the real-time scheduling strategy in the second time scale scheduling stage.

In the present invention, the reactive voltage model in step (3) specifically includes:

under the condition of meeting the constraint conditions of the voltage of a power grid and the operation of reactive compensation equipment, the reactive optimization objective function f of the power distribution network is realized by adjusting the intelligent agent of the reactive compensation equipment₁Defined as the sum of active power losses P of the distribution network_lossAnd the constraint conditions comprise upper and lower limit constraints of node voltage, reactive power and action quantity change and constraints of a power flow equation, and the specific formula is defined as follows:

U_min≤U_i≤U_max

φ_min≤φ_i≤φ_max

wherein, U_min，U_maxRespectively, node i voltage U_iLower and upper limits of (d); phi is a_min，φ_maxRespectively is the lower limit and the upper limit of the output of the CB action frequency of the node i; l_ij＝|I_ij|²，u_j＝|U_j|²，I_ijCurrent for branch ij; i: i → j indicates that in the branch ij, nodes including i flow from the node i to the node j; k: j → k indicates that in the branch jk, the node beginning with j flows from the node j to the node k; p_ij、Q_ijRespectively the active and reactive power flowing through branch ij; r is_ij、x_ijThe resistance and reactance of the branch ij are respectively; p is a radical of_jAnd q is_jRespectively the active power and the reactive power injected into the node j.

In the invention, the step (3) of training the offline reactive voltage optimization model by adopting an MA-SAC algorithm specifically comprises the following steps:

under the condition that the operation constraint conditions of the power grid voltage and reactive power compensation equipment are met, the reactive power output of the photovoltaic, the fan and the energy storage inverter is adjusted, the scheduling result of reactive power optimization in the first time scale stage is corrected in the second time scale stage, the objective function is defined as the minimum node voltage deviation of the power distribution network, and the constraint conditions comprise node voltage, photovoltaic fan output upper and lower limit constraints and the constraint of a power flow equation; the specific formula is defined as follows:

U_min≤U_i≤U_max

wherein, U_min，U_maxRespectively, node i voltage U_iLower and upper limits of (d); u shape_i，baseIs the reference voltage amplitude of node i; p_PV，Q_PV，S_PVRespectively the active power, the reactive power and the apparent power of the photovoltaic equipment; p_WD，Q_WD，S_WDRespectively the active power, the reactive power and the apparent power of the photovoltaic equipment;

and establishing an interactive training environment of the Markov process according to the reactive voltage optimization objective function and the constraint condition model.

In the invention, in the step (3), an MA-SAC algorithm is adopted to train an offline reactive voltage optimization model in the training process; the process specifically comprises the following steps:

(3.1) for a given action by the multi-agent, the environment provides the voltage on all buses in the power distribution system as the state of the MA-SAC model; s, which is formulated as:

s＝{U_t，P_PV，t，P_WD，t}

wherein, U_tA node voltage matrix of the power distribution network in the tth decision stage; p is_PV，tPhotovoltaic active power in the t scheduling period; p_WD，tThe active power of the fan in the t scheduling period is set;

(3.2) in the off-line training process, adopting an MA-SAC off-line training reactive power optimization model, in the entropy regularization expression, each agent can obtain a positive reward which is in direct proportion to the current strategy entropy when training, and the target can be defined as

x represents information of the entire environment,

is the ith centralized evaluation function whose input is the action a taken by each agent_iAnd environment information x, the output of which is the Q value of the ith agent, where entropy H is expressed as

H(π(·|s_t))＝-∑_aπ(a|s_t)lnπ(a|s_t)

(3.3) initializing a random process B for action exploration; initializing an environment state x, and generating an optimal photovoltaic and fan real-time dynamic strategy according to the intraday reactive power optimization model, wherein the method comprises the following steps:

a. selecting action a for each agent_iExecuting action a ═ a₁，…，a_NObserving the reward r and the next state x ', and storing the conversion quadruple (x, a, r, x') into an experience playback pool D;

b. for each agent, K samples (x) are extracted from D_j，a_j，r_j，x′_j) Let y_j＝r_j+γQ^μ(x′_j，a′₁，…，a′_N) (ii) a It is composed ofMiddle r_jTo observe the corresponding reward, γ is the discount factor, x ', for each step in the future'_jIs in the next observation state, a'₁，…，a′_NActions made by agents 1-N, respectively;

c. update the merit function by minimizing the loss function:

updating the action function by gradient descent:

and repeating step a;

d. updating parameters of the target neural network of each agent: theta'_i＝τθ_i+(1-τ)θ′_i(ii) a Wherein tau is an update weight;

and (3.4) scheduling the reactive power output of the photovoltaic, the fan and the energy storage inverter by the trained reactive voltage optimization scheduling model in the second time scale stage in a 1-minute level, so as to correct the reactive power strategy in the day ahead and achieve better reactive voltage control effect.

Compared with the prior art, the invention has the beneficial effects that:

(1) the invention designs a reactive voltage optimization method under two time scales. In the first time scale scheduling stage, 1 hour is used as a time scale to schedule OLTC, CB and ES; and in the second time scale scheduling stage, the reactive power output of PV, WD and ES is scheduled by adopting 1 minute as the time scale, and the reactive power optimization strategy in the first time scale scheduling stage is modified.

(2) Compared with the traditional reactive power optimization model, the deep reinforcement learning model is adopted in the first and second time scale scheduling stages, so that the strategy scheduling can be carried out more quickly and in real time, and the accurate load flow modeling is not needed and the accurate load prediction data is not depended on.

(3) According to the invention, OLTC tap positions, CB switching and ES reactive power output are taken as discrete action variables in a first time scale dispatching stage, and PV, WD and ES reactive power output are taken as continuous action variables in a second time scale dispatching stage. In the second time scale scheduling stage, a centralized training and decentralized execution mode is adopted, the deep neural network training is completed in the process of interaction with the power distribution network environment, and the trained network can be quickly decided, so that the voltage deviation is effectively reduced.

Drawings

FIG. 1 is a first time scale scheduling stage deep reinforcement learning DDQN algorithm training framework;

fig. 2 is a second time scale scheduling phase MA-SAC algorithm centralized training-decentralized execution framework.

Detailed Description

The applicant's group of inventors was inspired by a data-driven approach, defining on-load tap-changing (OLTC), Capacitor Bank (CB) and Energy Storage (ES) in photovoltaic, wind and load as agents, applying reinforcement learning based approach to reactive power optimization problem, allowing controllers to learn control strategies by interacting with simulation models of similar systems. The action variables of the reactive power regulation equipment are interacted with the power distribution network environment, the interaction process is described into a process (markov decision process) called Markov decision through a time sequence in mathematics, and the intelligent agent can finally achieve the optimal response to the external environment, so that the maximum return value is obtained. The strategic function and the action value function of each agent are analyzed and fitted by a neural network method, the training process does not depend on a predicted data result and accurate trend modeling, and the two time-scale reactive power optimization methods based on deep reinforcement learning can reduce the network loss and achieve better voltage stabilizing effect and more remarkable effect on improving the safety and reliability of the distribution network.

The multi-agent deep reinforcement learning reactive voltage optimization method based on multi-time scale is exemplified by taking an n-node power distribution network as an example.

Every the node all can have the state of measurationing, wherein, a node is equipped with on-load voltage regulation tap (OLTC), and at least one node is equipped with Capacitor Bank (CB), and at least one node is equipped with photovoltaic equipment (PV), and at least one node equipment is equipped with fan equipment, and at least one node is equipped with energy storage Equipment (ES). The optimization target is set to reduce the system grid loss and the voltage deviation, a Centralized Training and Decentralized Execution (CTDE) framework is utilized, the time scale of a first time scale scheduling stage is 1 hour, a DDQN algorithm is adopted to train an optimization model, the gear of an on-load tap changer (OLTC), the number of switching groups of a Capacitor Bank (CB) and the reactive power output of Energy Storage (ES) are scheduled, and the optimization process can be described as a Markov game process; the time scale of the second time scale scheduling stage is 1 minute, a multi-agent deep reinforcement learning algorithm (MA-SAC) based on maximum entropy is adopted to train a reactive voltage optimization model in an off-line mode, reactive power output of photovoltaic and fan power generation equipment and an energy storage inverter is dynamically adjusted in real time, the scheduling result of reactive power optimization in the first time scale scheduling stage is corrected, and rapid voltage fluctuation is relieved.

First time scale scheduling stage

As shown in FIG. 1, the Markov process of the distribution grid environment and the agent interaction process may be described as<S，R，A，P，γ，>. Under the condition of meeting the constraint conditions of the voltage of the power grid and the operation of reactive compensation equipment, the reactive optimization objective function of the power distribution network can be defined as the sum P of the active power loss of the power distribution network by adjusting the reactive equipment OLTC, CB and ES_lossMinimum, and constraint conditions including node voltage U_iUpper and lower limits of variation of motion amount_iAnd the constraint of the power flow equation, wherein the specific formula is defined as follows:

U_min≤U_i≤U_max

φ_min≤φ_i≤φ_max

wherein, U_min，U_maxRespectively, node i voltage U_iLower and upper limits of (d); phi is a_min，φ_maxThe lower limit and the upper limit of the output force of the CB action frequency of the node i are respectively, for example, the upper limit and the lower limit of the voltage of each node can be set to be 0.95 and 1.05; the number of CB actions in a day does not exceed 5.

According to the reactive voltage optimization objective function and the constraint condition model, establishing an interactive training environment of a Markov process, wherein the process comprises the following steps:

1. during the training process of the intelligent agent, strategy functions can be adjusted according to states in the power distribution system, and control measures are taken according to given operating conditions so as to achieve reactive power optimization. For a given action by a multi-agent, the environment provides the voltage on all buses in the power distribution system as the state of the DDQN model. Can be expressed as s ═ U_i，W_i，C_wiIn which U is_iA node voltage matrix of the power distribution network in the ith decision stage; w_iSwitching gears of each regulating device in the ith scheduling period; c_wiThe actions that the device has completed are adjusted for each of the i scheduling periods.

2. Constructing motion vector a ═ { T of OLTC, CB and ES_ol，T_cb，T_esIn which T is_ol，T_cb，T_esThe tap gear of the OLTC, the switching group number of the capacitor group CB and the reactive power output of the ES are respectively.

3. And (3) performing offline training of a reactive power optimization model by adopting a DDQN (double Q network) algorithm, taking the state as the input of a neural network, calculating all action value functions by using the neural network, and directly outputting corresponding Q values.

4. Defining an objective function of DDQN reinforcement learning as

Q^*(s，a)＝E_s′～e[r+γmax_a′Q^*(s′，a′)|s，a]

Wherein r is the corresponding reward for observation, γ is the discount factor for each step in the future, and s 'and a' are the next state and actions that may be taken, respectively;

5. initializing the capacity of a multiplexing pool D to be N; the Q value corresponding to the initialization action is random, and an optimal reactive power compensation strategy is generated according to the reactive power optimization model, and the method comprises the following steps:

1) initialization sequence s₁＝{x₁And the preprocessing sequence of the first state phi₁＝φ(s₁) (ii) a Wherein x is₁Is in a first state;

2) randomly selecting an action a with a probability of epsilon_tOtherwise, select action a_t＝max_a Q^*(s ', a; θ), where θ is the weight of the neural network function, s' is the next state, and a is the current action;

3) performing an action a in a power distribution network_tAnd observe the corresponding r_tAnd state s_t+1(ii) a Let the next state s_t+1＝s_t，a_t，x_t+1(ii) a Next preprocessing sequence phi_t+1＝φ(s_t+1)；

4) Samples are stored in an empirical multiplexing pool D and small batches of samples (phi) are randomly drawn therefrom_j，a_j，r_j，φ_j+1) (ii) a Order to

Wherein r is_jGamma is the discount factor per step in the future, s, for observing the corresponding reward_t+1For the next observation state, θ_tIs a parameter of the current neural network, θ'_tParameters of the neural network in the next step;

5) to pair

Performing gradient descent and repeating the step (1).

6. And taking the optimal OLTC, CB and ES hour-scale control strategy of the reactive voltage optimized scheduling model trained in the first time scale scheduling stage as the input of the real-time scheduling strategy of the second time scale scheduling stage.

(II) second time scale scheduling phase

As shown in FIG. 2, an interactive training environment based on a Markov process is also built, which can be described as a tuple<N，S，a₁，a₂，…，a_N，T，γ，r₁，…，r_N>Where N represents the number of agents, S represents the system state, a₁，a₂，…，a_NIs a set of actions for the agent. T represents a state transition function, T: s x a₁×…a_N×S→[0，1]And giving the probability of the next state according to the current system state and the joint action. Gamma is a discount factor. In addition, with r_i(s^t，a₁，…，a_N，s^{t +1}) Representing the reward resulting from performing the join action s' while agent i is in state s.

The reactive voltage model of the power distribution network comprises:

under the condition that the constraint conditions of grid voltage and reactive power compensation equipment operation are met, the reactive power output of photovoltaic fans and the reactive power output of fans are adjusted, the second stage corrects the scheduling result of reactive power optimization in the day-ahead scheduling stage, the objective function is defined as the minimum node voltage deviation of the power distribution network, the constraint conditions comprise node voltage, photovoltaic fan output upper and lower limit constraints and the constraints of a power flow equation, and the specific formula is defined as follows:

U_min≤U_i≤U_max

wherein, U_min，U_maxRespectively, node i voltage U_iThe lower and upper limits of (d); u shape_i，baseIs the reference voltage amplitude of node i; p_PV，Q_PV，S_PVRespectively the active power, the reactive power and the apparent power of the photovoltaic equipment; p_WD，Q_WD，S_WDRespectively the active power, the reactive power and the apparent power of the photovoltaic equipment;

according to the reactive voltage optimization objective function and the constraint condition model, establishing an interactive training environment of a Markov process, wherein the process comprises the following steps:

1. for a given action by a multi-agent, the environment provides the voltage on all buses in the power distribution system as a state of the MA-SAC model. Can be expressed as s ═ U_t，P_PV，t，P_WD，tIn which U is_tA node voltage matrix of the power distribution network in the tth decision stage; p_PV，tPhotovoltaic active power in the t scheduling period; p_WD，tAnd the active power of the fan in the t scheduling period.

2. In the off-line training process, an MA-SAC off-line training reactive power optimization model is adopted, in an entropy regularization expression, each agent can obtain a forward reward which is in direct proportion to the current strategy entropy during training, and the target can be defined as:

x represents information of the entire environment,

is the ith centralized evaluation function whose input is the action a taken by each agent_iAnd environment information x, the output of which is the Q value of the ith agent, where entropy H is expressed as

3. Initializing a random process B for action exploration; initializing an environment state x, and generating an optimal photovoltaic and fan real-time dynamic strategy according to the intraday reactive power optimization model, wherein the method comprises the following steps:

1) selecting action a for each agent_iExecuting action a ═ a₁，…，a_NObserving the reward r and the next state x ', and storing the conversion quadruple (x, a, r, x') into an experience playback pool D;

2) for each agent, K samples (x) are extracted from D_j，a_j，r_j，x′_j) Let y_j＝r_j+γQ^μ(x′_j，a′₁，…，a′_N) (ii) a Wherein r is_jTo observe the corresponding reward, γ is the discount factor, x ', for each step in the future'_jIs in the next observation state, a'₁，…，a′_NActions made by agents 1-N, respectively;

3) update the merit function by minimizing the loss function:

updating the action function by gradient descent:

and repeating the step (1).

4) Updating parameters of the target neural network of each agent: theta'_i＝τθ_i+(1-τ)θ′_i. Wherein, tau is an updating weight;

4. and scheduling the reactive power output of the photovoltaic, the fan and the energy storage inverter by the trained reactive power voltage optimization scheduling model in the second time scale stage in a 1-minute level, so as to correct the reactive power strategy in the day ahead and achieve better reactive power voltage control effect.

Claims (7)

1. A reactive voltage control method based on multi-time scale and multi-agent deep reinforcement learning is characterized by comprising the following steps:

(1) dividing a reactive voltage control process of a power distribution network accessed to a renewable distributed power supply into a first time scale stage and a second time scale stage; wherein, the first 1 hour is used as the scheduling period of the first time scale stage; the control process is a second time scale stage, and every 1 minute is taken as a scheduling period in the stage;

(2) defining an on-load tap changing OLTC, a capacitor bank CB and an energy storage ES as an intelligent agent, and building an interactive training environment of a Markov decision process of an environment and intelligent agent interaction at a first time scale stage; in the interactive training of the process, prediction data of photovoltaic, a fan and a load are input, and a DDQN algorithm Double Q Network is adopted to perform offline training of a reactive power optimization discrete action strategy; after training is finished, obtaining scheduling strategies of the OLTC, CB and ES of the intelligent agent, and taking the optimal reactive power control strategy generated by training as the input of a second time scale scheduling stage;

(3) in a second time scale stage, establishing an interactive training environment based on a Markov decision process in the same way based on the optimal reactive power control strategy obtained in the step (2); in the training process, a multi-agent deep reinforcement learning algorithm MA-SAC based on maximum entropy is adopted for training and optimizing an offline reactive voltage model;

(4) and deploying the reactive voltage model trained by the MA-SAC algorithm to a power distribution network with an online decision, and generating reactive output of intelligent agents in a photovoltaic system, a fan and an energy storage inverter in real-time scheduling by taking 1 minute as a time scale, so as to correct a scheduling result of reactive optimization in a first time scale scheduling stage, and further to relieve voltage fluctuation.

2. The method of claim 1, wherein in the first time scale scheduling phase of step (2), the markov process of interaction between the environment and the agents OLTC, CB and ES is described as < S, R, a, P, γ >; wherein S represents a system state space and is a set of all states that the intelligent can sense; r is a reward space, which is a collection of even rewards that the environment returns to the agent according to state actions; a is an action space which is a collection of actions of a decision-making main body on the environment; p is the state transition probability; γ is the discount rate in return, representing the conversion factor for future returns;

in the second time scale scheduling stage in the step (3), an interactive training environment based on the Markov process is built in the same way, and the process is described as an interactive training environment by a tuple<N，S，a₁，a₂，…，a_N，T，γ，r₁，…，r_N>(ii) a Wherein N represents the number of agents, S represents the system state, a₁，a₂，…，a_NAn action set for an agent; t represents a state transition function, T: s x a₁×…aa_N×S→[0，1]Giving the probability of the next state according to the current system state and the joint action; γ is a discount factor; in addition, with r_i(s^t，a₁，…，a_N，s^t+1) Representing the reward resulting from performing the join action s' while agent i is in state s.

3. The method according to claim 1, wherein the step (2) comprises in particular:

(2.1) under the condition of meeting the constraint conditions of the voltage of the power grid and the operation of reactive compensation equipment, defining a reactive optimization objective function of the power distribution network as the total active power loss P of the power distribution network by adjusting the output of an inverter of an agent of the reactive compensation equipment_lossMinimum, and aboutThe beam condition comprises upper and lower limit constraints of node voltage, reactive power and action quantity change and constraints of a power flow equation;

(2.2) establishing an interactive training environment of a Markov decision process according to the reactive voltage optimization objective function and the model of the constraint condition; and taking the intelligent agent optimal small-scale control strategy of the trained DDQN deep reinforcement learning reactive voltage optimization scheduling model as the input of the real-time scheduling strategy of the second-stage time scale.

4. The method according to claim 3, characterized in that said step (2.2) comprises in particular:

(2.2.1) in the training process of the intelligent agent, according to the state adjustment strategy function in the power distribution system, taking control measures aiming at given operating conditions to realize reactive power optimization; for a given action by a multi-agent, the environment provides the voltage on all buses in the power distribution system as the state of the DDQN model; the concrete expression is as follows:

s＝{U_i，W_i，C_wi}

wherein, U_iA node voltage matrix of the power distribution network in the ith decision stage; w_iSwitching gears of each regulating device in the ith scheduling period; c_wiAdjusting the actions completed by each device in i scheduling periods;

(2.2.2) constructing a motion vector of the agent:

a＝{T_ol，T_cb，T_es}

wherein, T_ol，T_cb，T_esThe on-off tap gear of the OLTC, the switching group number of the capacitor group CB and the reactive power output of the ES are respectively;

(2.2.3) adopting a DDQN algorithm to train a reactive power optimization model in an off-line manner, taking the state as the input of a neural network, calculating all action value functions by using the neural network, and directly outputting a corresponding action-value Q value for evaluating the expected value of a certain action behavior in the current given state;

(2.2.4) defining an objective function of the DDQN algorithm reinforcement learning as follows:

Q^*(s，a)＝E_s′～e[r+γmax_a′Q^*(s′，a′)|s，a]

wherein the action cost function Q^*(s, a) is the maximum value of all action values; r is the corresponding reward observed, γ is the discount factor for each step in the future, and s 'and a' are the next state and actions that may be taken, respectively;

(2.2.5) initializing the capacity of the multiplexing pool D to be N; the Q value corresponding to the initialization action is random, and an optimal reactive power compensation strategy is generated according to the reactive power optimization model; the method specifically comprises the following steps:

a. initialization sequence s₁＝{x₁And the preprocessing sequence of the first state phi₁＝φ(s₁) (ii) a Wherein x is₁Is in a first state;

b. randomly selecting an action a with a probability of epsilon_tOtherwise, select action a_t＝max_a Q^*(s', a; θ); wherein, theta is the weight of the neural network function, s' is the next state, and a is the current action;

c. performing an action a in a power distribution network_tAnd observe the corresponding r_tAnd state s_t+1(ii) a Let the next state s_t+1＝s_t，a_t，x_t+1Next preprocessing sequence phi_t+1＝φ(s_t+1)；

d. Samples are stored in an empirical multiplexing pool D and small batches of samples (phi) are randomly drawn therefrom_j，a_j，r_j，φ_j+1) (ii) a Order to

Wherein r is_jGamma is the discount factor per step in the future, s, for observing the corresponding reward_t+1For the next observation state, θ_tIs a parameter of the current neural network, θ'_tParameters of the neural network in the next step;

e. to pair

Performing gradient descent and repeating the step b;

and (2.2.6) taking the intelligent agent optimal small-scale control strategy of the reactive voltage optimization scheduling model trained in the first time scale scheduling stage as the input of the real-time scheduling strategy in the second time scale scheduling stage.

5. The method according to claim 1, wherein the reactive voltage model in step (3) comprises in particular:

under the condition of meeting the constraint conditions of the voltage of a power grid and the operation of reactive compensation equipment, the reactive optimization objective function f of the power distribution network is realized by adjusting the intelligent agent of the reactive compensation equipment₁Defined as the sum of active power losses P of the distribution network_lossAnd the constraint conditions comprise upper and lower limit constraints of node voltage, reactive power and action quantity change and constraints of a power flow equation, and the specific formula is defined as follows:

U_min≤U_i≤U_max

φ_min≤φ_i≤φ_max

wherein, U_min，U_maxRespectively, node i voltage U_iLower and upper limits of (d); phi is a_min，φ_maxCB of nodes i respectivelyThe lower limit and the upper limit of the output of the action times; l_ij＝|I_ij|²，u_j＝|U_j|²，I_ijCurrent for branch ij; i: i → j indicates that in the branch ij, nodes including i flow from the node i to the node j; k: j → k indicates that in the branch jk, the node beginning with j flows from the node j to the node k; p_ij、Q_ijRespectively the active and reactive power flowing through branch ij; r is_ij、x_ijRespectively the resistance and reactance of the branch ij; p is a radical of_jAnd q is_jRespectively the active power and the reactive power injected into the node j.

6. The method according to claim 1, wherein the step (3) of training the offline reactive voltage optimization model by using an MA-SAC algorithm specifically comprises the following steps:

under the condition that the operation constraint conditions of the power grid voltage and reactive power compensation equipment are met, the reactive power output of the photovoltaic, the fan and the energy storage inverter is adjusted, the scheduling result of reactive power optimization in the first time scale stage is corrected in the second time scale stage, the objective function is defined as the minimum node voltage deviation of the power distribution network, and the constraint conditions comprise node voltage, photovoltaic fan output upper and lower limit constraints and the constraint of a power flow equation; the specific formula is defined as follows:

U_min≤U_i≤U_max

wherein, U_min，U_maxRespectively, node i voltage U_iLower and upper limits of (d); u shape_i，baseIs the reference voltage amplitude of node i; p_PV，Q_PV，S_PVRespectively the active power, the reactive power and the apparent power of the photovoltaic equipment; p_WD，Q_WD，S_WDRespectively the active power, the reactive power and the apparent power of the photovoltaic equipment;

wherein l_ij＝|I_ij|²，u_j＝|U_j|²，I_ijCurrent for branch ij; i: i → j indicates that in the branch ij, nodes including i flow from the node i to the node j; k: j → k indicates that in the branch jk, the node beginning with j flows from the node j to the node k; p_ij、Q_ijRespectively the active and reactive power flowing through branch ij; r is_ij、x_ijThe resistance and reactance of the branch ij are respectively; p is a radical of_jAnd q is_jRespectively injecting active power and reactive power of the node j;

and establishing an interactive training environment of the Markov process according to the reactive voltage optimization objective function and the constraint condition model.

7. The method according to claim 1, wherein in the step (3), an MA-SAC algorithm is adopted to train an offline reactive voltage optimization model in the training process; the process specifically comprises the following steps:

(3.1) for a given action of the multi-agent, the environment provides the voltage on all buses in the power distribution system as the current environment state s of the MA-SAC model, which is formulated as:

s＝{U_t，P_PV，t，P_WD，t}

wherein, U_tA node voltage matrix of the power distribution network at the t decision stage; p_PV，tPhotovoltaic active power in the t scheduling period; p_WD，tThe active power of the fan in the t scheduling period is set;

(3.2) in the off-line training process, adopting an MA-SAC off-line training reactive power optimization model, in the entropy regularization expression, each agent can obtain a positive reward which is in direct proportion to the current strategy entropy when training, and the target of the positive reward is defined as

x represents information of the entire environment,

is the ith centralized evaluation function whose input is the action a taken by each agent_iAnd environment information x, the output of which is the Q value of the ith agent, where entropy H is expressed as

H(π(·|s_t))＝-∑_aπ(a|s_t)lnπ(a|s_t)

(3.3) initializing a random process B for action exploration; initializing an environment state x, and generating an optimal photovoltaic and fan real-time dynamic strategy according to a reactive power optimization model in a day, wherein the method comprises the following steps:

a. selecting action a for each agent_iExecuting action a ═ a₁，…，a_NObserving the reward r and the next state x ', and storing the converted quadruple (x, a, r, x') into an experience simultaneous storage pool D;

b. for each agent, K samples (x) are extracted from D_j，a_j，r_j，x′_j) Let y_j＝r_j+γQ^μ(x′_j，a′₁，…，a′_N) (ii) a Wherein r is_jGamma is the discount factor, x ', for each step in the future to observe the corresponding reward'_jIs in the next observation state, a'₁，…，a′_NActions made by agents 1-N, respectively;

c. update the merit function by minimizing the loss function:

updating the action function by gradient descent:

and repeating step a;

d. updating parameters of the target neural network of each agent: theta'_i＝τθ_i+(1-τ)θ′_i(ii) a Wherein, tau is an updating weight;

and (3.4) scheduling the reactive power output of the photovoltaic, the fan and the energy storage inverter by the trained reactive voltage optimization scheduling model in the second time scale stage in a 1-minute level, so as to correct the reactive power strategy in the day ahead and achieve better reactive voltage control effect.

Images