CN112632860A - Power transmission system model parameter identification method based on reinforcement learning - Google Patents
Power transmission system model parameter identification method based on reinforcement learning Download PDFInfo
- Publication number
- CN112632860A CN112632860A CN202110002104.6A CN202110002104A CN112632860A CN 112632860 A CN112632860 A CN 112632860A CN 202110002104 A CN202110002104 A CN 202110002104A CN 112632860 A CN112632860 A CN 112632860A
- Authority
- CN
- China
- Prior art keywords
- value
- parameter
- parameters
- reinforcement learning
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000002787 reinforcement Effects 0.000 title claims abstract description 57
- 230000005540 biological transmission Effects 0.000 title claims abstract description 31
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 29
- 238000004088 simulation Methods 0.000 claims abstract description 27
- 230000008569 process Effects 0.000 claims abstract description 21
- 230000035945 sensitivity Effects 0.000 claims abstract description 15
- 238000005457 optimization Methods 0.000 claims abstract description 14
- 230000006870 function Effects 0.000 claims description 44
- 239000002245 particle Substances 0.000 claims description 38
- 230000009471 action Effects 0.000 claims description 32
- 239000003795 chemical substances by application Substances 0.000 claims description 30
- 239000013598 vector Substances 0.000 claims description 28
- 238000012549 training Methods 0.000 claims description 26
- 239000011159 matrix material Substances 0.000 claims description 23
- 238000010206 sensitivity analysis Methods 0.000 claims description 11
- 230000006399 behavior Effects 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 5
- 230000008859 change Effects 0.000 claims description 4
- 238000012614 Monte-Carlo sampling Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 2
- 238000005094 computer simulation Methods 0.000 abstract description 5
- 230000008901 benefit Effects 0.000 abstract description 4
- 230000002265 prevention Effects 0.000 abstract description 2
- 239000003638 chemical reducing agent Substances 0.000 description 3
- 238000013016 damping Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 239000006096 absorbing agent Substances 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001364 causal effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000002068 genetic effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000035939 shock Effects 0.000 description 1
- 230000005477 standard model Effects 0.000 description 1
- 238000005728 strengthening Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/25—Design optimisation, verification or simulation using particle-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2111/00—Details relating to CAD techniques
- G06F2111/08—Probabilistic or stochastic CAD
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2119/00—Details relating to the type or aim of the analysis or the optimisation
- G06F2119/14—Force analysis or force optimisation, e.g. static or dynamic forces
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Geometry (AREA)
- Computer Hardware Design (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Feedback Control In General (AREA)
Abstract
The invention discloses a method for identifying model parameters of a power transmission system based on reinforcement learning, and belongs to the field of system modeling simulation. Aiming at the problems of inconsistent sensitivity of model parameters of the power transmission system, low convergence rate and higher search range requirement of the existing identification algorithm, the invention constructs a reinforcement learning framework for identifying the model parameters of the power transmission system, and has the advantages of local optimization prevention, high convergence rate and large search range; the method adopts a staged identification process, the rough adjustment stage utilizes the characteristics of high convergence speed and large search range of reinforcement learning to quickly find the optimal subinterval of each parameter, and the fine adjustment stage utilizes the characteristics of high precision and strong global search capability of a heuristic algorithm to determine the final identification result in the optimal subinterval.
Description
Technical Field
The invention belongs to the field of system modeling simulation, and particularly relates to a method for identifying parameters of a power transmission system model based on reinforcement learning.
Background
With the continuous development of the multi-field system modeling simulation technology, the Modelica language gradually becomes an industry standard for the development of the multi-field and multi-subject system simulation field, and the aim of the Modelica language is to define a universal programming language for complex system modeling. The Modelica model is a program which is written by using a Modelica language and has the capabilities of object-oriented modeling, multi-field unified modeling, non-causal statement type modeling and continuous discrete hybrid modeling, and can be used for engineers in different industries to build own simulation model systems and develop corresponding dynamic simulation. For the Modelica model, the model is required to be consistent with a physical prototype as much as possible, and the key point is the setting of model parameters, the process of building the model is only used for determining the basic form of the model, and accurate model parameters are required to be set for obtaining the optimal simulation performance.
The power transmission system is a device which obtains energy from a power source in the form of speed/angular velocity and force/moment under the control of a controller and transmits the energy to the next link of the system, and mainly comprises components such as a motor, a speed reducer and the like, and the states of energy loss, speed change and the like exist in the power transmission process.
The existing model parameter identification method has various types, the most common method is the least square method at present, and the method is a default parameter optimization method in modeling software. However, the least square estimation has too high dependency on data, the identification result is easily affected by noise, and the requirement on the initialization range of the parameters is strict. In addition, the commonly used identification method is an evolutionary Algorithm, such as Particle Swarm Optimization (PSO), Genetic Algorithm (GA), and the like, which has a strong ability to approach global optimum, but has a slow convergence rate and a precision that needs to be improved. Therefore, an identification method with high convergence rate and low requirement on the initial range of parameters is urgently needed to identify the parameters of the Modelica-based power transmission system model.
Disclosure of Invention
In view of the above defects or improvement requirements of the prior art, the present invention provides a method for identifying parameters of a model of a powertrain system based on reinforcement learning, and aims to provide an identification method with high convergence rate and low requirement for an initial range of parameters to identify parameters of a model lia-based model of a powertrain system.
In order to achieve the above object, the present invention provides a method for identifying parameters of a powertrain system model based on reinforcement learning, comprising:
s1, constructing a dynamic model of a power transmission system based on a multi-field unified modeling language Modelica;
s2, carrying out sensitivity analysis on the parameters to be identified of the model;
s3, roughly adjusting the parameters to be identified based on a reinforcement learning algorithm:
constructing a reinforcement learning framework for identifying the Modelica power transmission system model parameters;
performing iterative training by using a reinforcement learning framework to obtain an optimal subinterval of each parameter to be identified;
s4, parameter fine adjustment:
and (3) taking the mean square error of the measured data and the model estimation value as an objective function, and iteratively searching for optimization in a solution space formed by the model parameters to be identified to obtain each corresponding parameter value when the objective function value is minimum, wherein the parameter value is used as a final identification result.
Further, step S2 is specifically to perform parameter sensitivity analysis on the powertrain model with undetermined parameters by using a sobol method, and the specific steps are as follows:
01. respectively carrying out Monte Carlo sampling in the possible value range of N parameters to be identified to generate an initial sample matrix I A, an initial sample matrix II B and a cross sample matrixWhere i ═ {1, 2, …, N };
02. the sample matrix A, B,As input, the power transmission system model is simulated and solved to respectively obtain an initial sample matrix A, an initial sample matrix B and a cross sample matrixThe model simulation result vectors f (A), f (B),
03. Calculating the global influence index S of each parameter based on the simulation result and the following formulaTi:
Wherein Y represents f (A), f (B) anda set of vectors is formed; va ((Y) represents the variance of the powertrain model output;
04. performing sensitivity sorting on the parameters to be identified according to the size of the parameter global influence index; larger impact factors indicate more sensitivity;
05. and combining the parameters to be identified with the sensitivity lower than a set threshold value.
Further, the reinforcement learning framework construction process specifically includes:
(1) model estimation value YestAnd measured value YmeaAs an reinforcement learning objective function f (x);
(2) constructing a single step reward:
r=min(1,max(0,(F(Xmean)-F(Xbest))/(F(Xmean)-F(Xcur))))
wherein r represents a single step reward value, F (X)cur) Denotes the value of the objective function under the current parameter, F (X)best) Denotes the value of the objective function under the optimum parameter, F (X)mean) Representing parametersObjective function value at average;
(3) minimum variation G according to parameteri(i ═ 1,2, …, N) and the range setting action for each parameter:
splitting the search range of the ith parameter intoSelecting one subinterval, and randomly acquiring a value in the subinterval as an action; wherein,is the maximum value of the ith parameter,is the ith parameter minimum; minimum change GiThe variable quantity of the ith parameter which is increased or decreased in each step in the identification process is referred to;
(4) constructing an action selection strategy:
01. selecting a search path:
determining whether the action of the next round of selection is on the left or the right of the current action, wherein the selection index is Lp(i, j) the calculation formula is:
wherein k represents the number of parameter transformation combinations;is the first path and the current action ai,jThe nth largest Q value, λ, among the adjacent k actions1Is a path weight coefficient;
obtain a [0,1 ]]Random number of epsilon in between1The search path/is determined using the following formula:
rand (1,2) represents a random probability distribution within the interval 1-2;
02. determining an action:
obtain a [0,1 ]]Random number of epsilon in between2Action a is determined using the following formula:
q (i, m) represents the Q value of the ith parameter to be identified, epsilon1And ε2Is a random number used to ensure reinforcement learning exploratory properties;
(5) and (3) constructing an updating strategy of a Q value function:
the Q value function updating formula corresponding to the ith parameter is as follows:
Qr+1(i,j)+=α(r+(1-λ2)max(Lp(i,j))+λ2min(Lp(i,j))-Qr(i,j))
where α is a hyperparameter controlling the learning rate, r is a single step reward, λ2To control the over-parameters of the update amplitude.
Further, in the parameter coarse tuning stage, a reinforcement Learning framework based on a Q-Learning algorithm is used, and the iterative training specifically comprises the following steps:
(1) randomly initializing all parameters to be identified, substituting the parameters into a power transmission system model for calculation, and comparing the parameters with actually measured data to obtain a mean square error serving as an initial value of an optimal objective function F (X);
(2) the plurality of agents execute serial learning from large to small according to the sensitivity of the parameters to be identified; the link of adjusting the parameters to be identified according to the Q value table in the reinforcement learning frame is called an agent, and each parameter to be identified uniquely corresponds to one agent; the process of the intelligent agent adjusting the parameters to be identified is called the learning behavior of the intelligent agent;
the learning process comprises the following steps: randomly selecting an action a in the range of possible value intervals of the corresponding parameters of the current agentrand(i) Fixing other parameters, applying the parameters to the power transmission system model to obtain the current parametersThe objective function value F (X)cur) And a single step reward value r, updating the Q value table of the current agent according to the Q value updating strategy; if F (X)cur)≤F(Xbest) If the current parameter has a search value, the step (3) is carried out, otherwise, the learning process is repeated;
(3) selecting an action a according to an action selection policyiter(i) Fixing other parameters, applying the parameters to FMU to obtain the value of target function F (X)cur) And a reward and punishment value r, updating the Q value table;
(4) iteratively executing steps (2) and (3) for the ith parameterSecondly, completing one identification of the ith parameter, and switching to the identification of the (i + 1) th parameter;
(5) all the parameters to be identified are serially executed in the steps (2) to (4) to complete a training period, and if the current completion period number is less than the given training timesAnd (2) entering the next training period, otherwise, finishing the training.
Further, step S4 is specifically to use a PSO optimization algorithm to fine tune the parameters to obtain a final identification result, and the specific steps are as follows:
setting a fine tuning range to take into account recognition errors of reinforcement learningWherein the mu value is set according to the searching capability of the particle swarm algorithm,in order to strengthen the optimal subinterval obtained by learning,the parameter is the optimal value of the ith parameter after reinforcement learning identification;
an N-dimensional space and a population of particles are initialized,creating a velocity vector v for each particleiPosition vector xiHistorical optimal position vector piAnd a historical optimal position vector p for the entire particle swarmg;
Establishing a fitness function G (X) by using the mean square error of a model predicted value and an actually measured value, wherein the model predicted value is obtained by substituting the current position of particles into an FMU for simulation and solving; the goal of the search over the entire population is to minimize G (X);
starting iterative search:
in one iteration, calculating the fitness value of each particle, and updating p of single particleiValue and p of populationgThe value, then according to the following formula update each particle velocity and position vector;
where ω is the inertial weight and is 0.6, c by default1、c2Is a learning factor, r1、r2Is [0,1 ]]A random number in between;
after the maximum iteration times are reached, the historical optimal position vector p of the particle swarmgThe final result of the parameter identification is obtained.
In general, the above technical solutions contemplated by the present invention can achieve the following advantageous effects compared to the prior art.
(1) Aiming at the problems that the Modelica model parameter sensitivities are inconsistent, the existing identification algorithm is low in convergence speed and high in search range requirement, a reinforcement learning framework for identifying the Modelica power transmission system model parameters is constructed, and the method has the advantages of local optimization prevention, high convergence speed and large search range; the invention adopts a staged identification process, the rough adjustment stage utilizes the characteristics of fast convergence speed and large search range of reinforcement learning to quickly find the optimal subinterval of each parameter, the fine adjustment stage utilizes the characteristics of high precision and strong global search capability of a heuristic algorithm to determine the final identification result in the optimal subinterval, compared with the method only using a single algorithm, the staged identification effect is better,
(2) aiming at the problem of huge search space caused by excessive parameters, the invention also carries out sensitivity analysis on the parameters to be identified, combines the parameters to be identified with lower sensitivity, is beneficial to reducing the number of the parameters to be identified and saves the expense of computing resources in the process of strengthening learning and identifying the parameters; meanwhile, the sensitivity is used for setting the parameter identification priority, so that a basis can be provided for the learning rate setting in the parameter identification process of the reinforcement learning method, and the parameter identification precision is improved.
(3) The method designs a sensitivity analysis scheme based on a Sobol method, simultaneously explores all parameters of the model in a possible value space based on the variance, has no limitation on the nonlinearity and monotonicity of the model, and is suitable for carrying out parameter global sensitivity analysis on any model.
Drawings
FIG. 1 is a flow chart of a method for identifying parameters of a powertrain model based on reinforcement learning according to the present invention;
FIG. 2 is a model of the powertrain created based on Modelica;
FIG. 3 is a global impact index for inertia2.J/spring. c/damming/and Jmotor parameters;
FIG. 4 is a Modelica-based power transmission model established for a numerically-controlled machine tool feeding system, parameter identification is carried out by adopting the method of the invention, and the output result of a physical prototype and the simulation result of the model are obtained under the same input condition.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
The invention provides a Modelica model parameter identification method based on reinforcement learning, which can effectively identify key parameters of a Modelica model so as to improve the simulation performance of the model. In order to meet the requirements of high convergence speed and high identification precision, the invention takes the actually measured data of a physical machine corresponding to a model as a training sample, adopts a reinforcement Learning framework improved based on a Q-Learning algorithm in the parameter coarse adjustment stage, and carries out serial iterative training on an intelligent agent represented by each parameter to be identified; and in the fine adjustment stage, a PSO/GA algorithm is adopted to determine the final value of the parameter to be identified. The method has the characteristics of high convergence rate and high identification precision.
Reinforcement learning is a machine learning method based on a trial and error mechanism,origin of originResearch in the field of animal intelligence has subsequently evolved into an important branch of the field of machine learning. The reinforcement learning considers the interaction process between the intelligent agent and the unknown environment guided by the target, accumulates experience in the process of continuous trial and error and continuously updates the action selection strategy, and has the advantages of high convergence speed, easy avoidance of local optimization, no need of priori knowledge and the like. Currently, reinforcement learning is mostly used in the field of combinatorial optimization (sequence decision) and has been successful, but the function optimization field to which parameter identification belongs has not been applied yet. In consideration of the requirements of fast convergence, global optimum and the like required by the Modelica model parameter identification, the invention provides a parameter identification method based on reinforcement learning to solve the problem.
The invention adopts the overall technical scheme that: analyzing the parameter sensitivity of the Modelica model to be identified by using a Sobol-based sensitivity analysis method, sequencing the parameters according to the size of the global influence factor of the parameters, and determining the training sequence and the discretization degree (namely the number of lattices split in the initialization range) of the parameters in the coarse adjustment stage; the Q-Learning-based improved reinforcement Learning framework is used, the mean square error between actual measurement data and model simulation data on a physical machine is taken as an objective function and an environment state, the adjustment of parameters in a feasible range is taken as an action, the relation between a current objective function value and a historical optimal/average value is a single-step reward, the parameters to be identified of the Modelica model are coarsely adjusted, and an optimal subinterval of each parameter is obtained; and (3) based on the optimal subintervals of all the parameters, performing iterative training by adopting a PSO/GA algorithm to obtain the optimal identification value of each parameter, namely the final identification result of the whole technical scheme.
S1, Modelica model parameter sensitivity analysis based on a Sobol method comprises the following steps:
(1) an FMU (Functional Module-up) of a Modelica model to be identified is derived, a Functional model unit comprises a C code, a solver and the like of the model, can be directly called and independently completes Simulation solving of the model, and is set to be a 'Co-Simulation' mode;
(2) respectively carrying out Monte Carlo sampling in the range of N parameters to be identified to generate a sample matrix A, B and ABiAnd (i-1, 2, …, N), performing simulation solution on the FMU by taking the sample matrix as input to obtain a Y-value matrix, obtaining a global influence factor of each parameter to be identified based on the Y-value matrix and an influence index formula, and sequencing. Determining a training sequence according to the size of the influence factors of the parameters to be identified, wherein the parameters with larger influence factors (namely more sensitive) are optimized in a single-round training;
(3) obtaining a random value in the search range of each parameter, substituting the random value into FMU to calculate the result as a reference value, and sequentially determining the minimum subinterval size G of each parameteri(i 1,2, …, N) to ensure that the ith parameter is increasing or decreasing GiAnd the simulation result is influenced obviously.
And S2, a reinforcement Learning framework based on the Q-Learning algorithm is used for the parameter coarse adjustment stage. The device is characterized by comprising the following components:
(1) setting the environmental state as the model estimated value Y of the current wheel numberestMeasured value Y of physical machinemeaThe mean square error (i.e., the reinforcement learning objective function f (x)) of (a) is used to represent the difference between the current model parameter combination and the actual parameter combination;
(2) the single step reward is used for measuring the quality of the action selected in the current state and is an expression of an objective function value, and the single step reward has the following two modes:
ii r=min(1,max(0,(F(Xmean)-F(Xbest))/(F(Xmean)-F(Xcur))))
r represents a reward or punishment value, F (X)mean) Denotes the value of the objective function at the mean value of the parameter, F (X)best) Denotes the value of the objective function under the optimum parameter, F (X)cur) Representing the value of an objective function at the current parameter
(3) Action according to parameter minimum sub-interval Gi(i-1, 2, …, N) and range settings for each parameter, splitting the search range for the ith parameter intoSub-interval (Is the maximum value of the parameter, and the maximum value of the parameter,the parameter is the minimum value), after one subinterval is selected, a value is randomly acquired in the subinterval to serve as an action;
(4) the action selection strategy is divided into two phases: selecting a search path and determining behavior. The former determines whether the next round of selected behavior is on the left or right of the current behavior, and the selection index is Lp(i, j), the calculation formula is as follows:
Lp(i, j) … …, k the number of parameter transformation combinations
Is the current behavior a on the ith pathi,jThe nth largest Q value, lambda, of the adjacent k behaviors1Is a pathA weight coefficient.
Obtain a [0,1 ]]Random number of epsilon in between1The search path is determined using the following formula:
rand (1,2) denotes a random probability distribution within the interval 1-2
The determined behavior is to select one of k behaviors adjacent to the current behavior on the search path to obtain a [0,1 ]]Random number of epsilon in between2Behavior is determined using the following formula:
ε1and ε2Are random numbers that are used to ensure reinforcement learning exploratory.
(5) Designing an updating strategy of a Q value function based on a Q-Learning algorithm, wherein a Q value function updating formula corresponding to the ith parameter is as follows:
Qr+1(i,j)+=α(r+(1-λ2)max(Lp(i,j))+λ2min(Lp(i,j))-Qr(i,j))
where α is a hyperparameter controlling the learning rate, r is a single step reward, λ2The smaller the value of the hyper-parameter for controlling the update amplitude, the larger the update amplitude of the Q value table.
S3, a Modelica model parameter identification algorithm based on a reinforcement learning framework is used for a parameter coarse adjustment stage, and the method specifically comprises the following steps:
(1) all the parameters to be identified are initialized randomly and substituted into FMU for calculation, and compared with measured data, the mean square error is obtained and used as the initial value of the optimal objective function, and the initial value is recorded;
(2) entering a training period, wherein a plurality of agents perform serial learning, namely, each parameter is optimized in turn, and the learning process of a single agent is called as a 'one-turn';
(3) enter intoA round, randomly selecting an action a within the search range of the parameterrand(i) Fixing other parameters, applying the parameters to FMU to obtain the objective function value F (X) of current roundcur) And a current reward and punishment value r, wherein the Q value updating strategy updates the Q value table of the current agent according to the Q value updating strategy of claim 3. If F (X)cur)≤F(Xbest) If the current parameter has the search value, turning to (4), otherwise, entering the next round;
(4) entering a search cycle in which execution is performedEach iteration selects an action a according to an action selection strategyiter(i) Fixing other parameters, applying the parameters to FMU to obtain the objective function value F (X) of current roundcur) Updating the Q value table of the current agent according to the current reward and punishment value r;
(5) after the search period of one agent is finished, entering a 'one-turn' of the next agent until all agents go through one-turn, finishing a training period, and if the current period number is less than the current period numberAnd (2) entering the next training period, otherwise, finishing the training.
S4, a parameter fine-tuning scheme based on a heuristic optimization algorithm PSO is formed as follows:
(1) and determining the fine tuning range of each parameter to be identified according to the operation result of the reinforcement learning algorithm. If the optimal subinterval obtained by reinforcement learning isSetting a fine tuning range to take into account recognition errors of reinforcement learningThe value of μ is set according to the search capability of the particle swarm algorithm,the parameter is the optimal value of the ith parameter after reinforcement learning identification;
(2) initializing an N-dimensional space and a group of particles, creating a velocity vector v for each particleiPosition vector xiHistorical optimal position vector piAnd a historical optimal position vector p for the entire particle swarmg;
(3) And establishing a fitness function F (X) by using the mean square error of the model predicted value and the measured value, wherein the model predicted value is obtained by substituting the current position of the particle into an FMU for simulation and solving. The goal of the search over the entire population is to minimize F (X);
(4) starting iterative search, calculating the fitness value of each particle in one iteration, and updating p of single particleiValue and p of populationgThe value, then according to the following formula update each particle velocity and position vector,
in the formula (1), omega is an inertia weight and is 0.6 as default, c1、c2For learning factors, in general c1=c2=2,r1、 r2Is [0,1 ]]A random number in between;
after the maximum iteration times are reached, the historical optimal position vector p of the particle swarmgThe final result of the parameter fine tuning is also the final result of the entire parameter identification process.
The embodiment of the invention takes a Modelica-based power transmission system model (a numerically-controlled machine tool single-shaft servo feeding system) as shown in figure 2 as an example to describe the method in detail, the system simulates a sinusoidal signal to drive a gear reducer and drive a load to run, and two components for simulating inertia and springs and shock absorbers for influencing the whole system are arranged in the middle of the system. The input of the whole system model is a sinusoidal signal, the output is the absolute angular velocity of the component inertia3, and the parameters to be identified and their attached information are shown in table 1.
TABLE 1
It should be explained that this embodiment is for explaining the technical solution of the present invention in detail and proving its effectiveness and feasibility, not a real engineering case, so in this embodiment, the measured data of a real physical machine is not used as a sample set, but the output of a standard model is used as a sample set, and then all the parameters to be identified are initialized randomly, and further, the technical solution of the present invention is used for parameter identification so that the parameters to be identified are as close to the parameter standard value as possible, and then the identification is completed. The complete identification process comprises the following steps:
1. deriving an FMU (functional model unit) of the Modelica model of the transmission system, and setting simulation parameters such as step length, starting time, ending time, solver and the like of simulation solution;
2. and setting the parameter to be identified in the FMU as a standard value, and performing simulation solving to obtain an output as a sample set. Sampling 1000 points on an output curve in a fixed period to serve as a data sample set for parameter identification at this time, wherein a subsequent reinforcement learning objective function and a PSO algorithm fitness function are mean square errors between the ordinate of the 1000 points and a model predicted value;
3. and carrying out parameter sensitivity analysis on the Modelica model. As described above, the 4 parameters to be identified are respectively reducer load moment of inertia2. j/torsion spring stiffness coefficient spring. c/damper damping coefficient damping/and motor output shaft moment of inertia Jmotor, and the sensitivity analysis steps are as follows:
(1) setting a sample: dimension 4 (i.e. 4 input variables), number 10000 (i.e. 10000 input samples), generating a 10000 × 8 Sobol matrix based on the range of each parameter, and processing the Sobol matrix according to the Sobol method to obtain a matrix A, B and AB (i)(i=1,2,…,8);
(2) Mixing the aboveInputting the sample matrix into FMU for simulation solution to obtain output, and sorting the output into Y value matrix Y according to Sobol methodA、YBAnd YABWith f (A), f (B) and f (A) respectivelyB i) Represents;
(3) the global influence index S of each parameter is obtained according to the following formulaTi:
The results are shown in fig. 3, in terms of sensitivity, Jmotor > spring.c > damming > inertia 2.j;
(4) determining the learning sequence to be Jmotor, spring.c, damming and inertia2.J according to the sensitivity sequence obtained in the step (3), and then determining G corresponding to each parameteriAnd determining that the number of the split lattices corresponding to each parameter is 200/1000/100/30 respectively.
4. And carrying out rough parameter adjustment on the Modelica model. Using Q-Learning based reinforcement Learning, a search range needs to be set for each parameter, in this case 4 parameters to be identified, and the search range is obtained from a look-up manual or experience, as also shown in table 1. Simultaneously, parameters in the algorithm are set, and the learning rate alpha of the reinforcement learning algorithm is made to be 0.7 and lambda10.5 is the path weight coefficient, λ20.25 is a hyper-parameter for controlling the update amplitude, k 4 is the number of parameter transformation combinations, and then the parameters are preliminarily identified by the following steps:
(1) randomly initializing all parameters to be identified, substituting the parameters into a model based on a Modelica power transmission system for calculation, comparing the parameters with a data sample set to obtain a mean square error as an initial value of an optimal objective function, and recording the mean square error;
(2) entering a training period, wherein a plurality of agents perform serial learning, namely, each parameter is optimized in turn, and the learning process of a single agent is called as a 'one-turn';
(3) entering a round, and randomly selecting an action a in the search range of the parameterrand(i) Fixing other parameters, and applying the sameThe parameters are applied to the FMU to obtain the objective function value F (X) of the current roundcur) Using the formulaObtaining a single step reward r, using equation Qr+1(i,j)+=α(r+(1-λ2)max(Lp(i,j))+λ2min(Lp(i,j))-Qr(i, j)) updates the Q value table of the current agent. If F (X)cur)≤F(Xbest) If the current parameter has the search value, turning to (4), otherwise, entering the next round;
(4) entering a search cycle in which execution is performedEach iteration selects an action a according to an action selection strategyiter(i) Fixing other parameters, applying the parameters to FMU to obtain the objective function value F (X) of current roundcur) Updating the Q value table of the current agent according to the current reward and punishment value r;
(5) after the search period of one agent is finished, entering a 'one-turn' of the next agent until all agents go through one-turn, finishing a training period, and if the current period number is less than the current period numberAnd (2) entering the next training period, otherwise, finishing the training.
The initial identification result obtained at this stage is the optimal subinterval of each parameter, as shown in table 2, on the basis of which an accurate solution needs to be determined;
TABLE 2
5. And carrying out parameter fine adjustment on the Modelica model. Step 4, only the optimal subinterval where each parameter to be identified is located is obtained, and a final identification result is determined by using a PSO algorithm in the interval, and the specific steps are as follows:
(1) the optimal subinterval obtained in step 4 isSetting a fine tuning range to take into account recognition errors of reinforcement learningSetting the mu values to be 2 based on the searching capability of the particle swarm optimization;
(2) initializing a 4-dimensional space and a group of particles, creating a velocity vector v for each particleiPosition vector xiHistorical optimal position vector piAnd a historical optimal position vector p for the entire particle swarmg;
(3) And establishing a fitness function F (X) by using the model predicted value and the mean square error of the data sample set, wherein the model predicted value is obtained by substituting the current position of the particle into an FMU for simulation and solving. The goal of the search over the entire population is to minimize F (X);
(4) starting iterative search, calculating the fitness value of each particle in one iteration, and updating p of single particleiValue and p of populationgThe value, then according to the following formula update each particle velocity and position vector,
in the formula (1), omega is an inertia weight and is 0.6 as default, c1、c2For learning factors, in general c1=c2=2,r1、 r2Is [0,1 ]]A random number in between;
(5) after the maximum iteration times are reached, the historical optimal position vector p of the particle swarmgThe final result of the parameter fine tuning is the most important result of the whole parameter identification processAnd (4) final results.
The comparison between the final identification result and the parameter standard value is shown in Table 3, and the errors between the parameter identification result and the parameter standard value are less than 5%, so that the engineering use requirement is met.
TABLE 3
A Modelica-based power transmission model is established for a feeding system of a numerical control machine tool, parameter identification is carried out by adopting the method, the output result of a physical prototype and the model simulation result are shown in figure 4 under the same input condition, and it can be seen that the transformation trend and the numerical value of the model simulation curve are basically consistent with the output curve of the physical prototype, thereby explaining the correctness and the effectiveness of the parameter identification method.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (7)
1. A method for identifying parameters of a power transmission system model based on reinforcement learning is characterized by comprising the following steps:
s1, constructing a dynamic model of a power transmission system based on a multi-field unified modeling language Modelica;
s2, carrying out sensitivity analysis on the parameters to be identified of the model;
s3, roughly adjusting the parameters to be identified based on a reinforcement learning algorithm:
constructing a reinforcement learning framework for identifying the Modelica power transmission system model parameters;
performing iterative training by using a reinforcement learning framework to obtain an optimal subinterval of each parameter to be identified;
s4, parameter fine adjustment:
and (3) taking the mean square error of the measured data and the model estimation value as an objective function, and iteratively searching for optimization in a solution space formed by the model parameters to be identified to obtain each corresponding parameter value when the objective function value is minimum, wherein the parameter value is used as a final identification result.
2. The method for identifying the parameters of the power transmission system model based on the reinforcement learning as claimed in claim 1, wherein the step S2 is to perform parameter sensitivity analysis on the power transmission system model with undetermined parameters by using a sobol method, and comprises the following steps:
01. respectively carrying out Monte Carlo sampling in the possible value range of N parameters to be identified to generate an initial sample matrix I A, an initial sample matrix II B and a cross sample matrixWhere i ═ {1, 2, …, N };
02. the sample matrix A, B,As input, the power transmission system model is simulated and solved to respectively obtain an initial sample matrix A, an initial sample matrix B and a cross sample matrixThe model simulation result vectors f (A), f (B),
03. Calculating the global influence index S of each parameter based on the simulation result and the following formulaTi:
Wherein Y represents f (A), f (B) andform aA set of vectors of (a); var (Y) represents the variance of the model output of the powertrain;
04. performing sensitivity sorting on the parameters to be identified according to the size of the parameter global influence index; larger impact factors indicate more sensitivity;
05. and combining the parameters to be identified with the sensitivity lower than a set threshold value.
3. The reinforcement learning-based power transmission system model parameter identification method according to claim 1, wherein the reinforcement learning framework construction process specifically comprises:
(1) model estimation value YestAnd measured value YmeaAs an reinforcement learning objective function f (x);
(2) constructing a single step reward:
r=min(1,max(0,(F(Xmean)-F(Xbest))/(F(Xmean)-F(Xcur))))
wherein r represents a single step reward value, F (X)cur) Denotes the value of the objective function under the current parameter, F (X)best) Denotes the value of the objective function under the optimum parameter, F (X)mean) Representing the value of the objective function at the mean value of the parameter;
(3) minimum variation G according to parameteri(i ═ 1, 2.., N) and the range setting actions for each parameter:
splitting the search range of the ith parameter intoSelecting one subinterval, and randomly acquiring a value in the subinterval as an action; wherein,is the maximum value of the ith parameter,is the ith parameter minimum; minimum change GiMeans that the ith parameter is being identifiedThe amount of change in each step of the process;
(4) constructing an action selection strategy:
01. selecting a search path:
determining whether the action of the next round of selection is on the left or the right of the current action, wherein the selection index is Lp(i, j) the calculation formula is:
wherein k represents the number of parameter transformation combinations;is the first path and the current action ai,jThe nth largest Q value, λ, among the adjacent k actions1Is a path weight coefficient;
obtain a [0,1 ]]Random number of epsilon in between1The search path/is determined using the following formula:
rand (1,2) represents a random probability distribution within the interval 1-2;
02. determining an action:
obtain a [0,1 ]]Random number of epsilon in between2Action a is determined using the following formula:
q (i, m) represents the Q value of the ith parameter to be identified, epsilon1And ε2Random numbers are used to ensure reinforcement learning exploratory properties;
(5) and (3) constructing an updating strategy of a Q value function:
the Q value function updating formula corresponding to the ith parameter is as follows:
Qr+1(i,j)+=α(r+(1-λ2)max(Lp(i,j))+λ2min(Lp(i,j))-Qr(i,j))
where α is a hyperparameter controlling the learning rate, r is a single step reward, λ2To control the over-parameters of the update amplitude.
4. The method for identifying the parameters of the model of the power transmission system based on the reinforcement Learning as claimed in claim 1, wherein in the rough parameter adjusting stage, the reinforcement Learning framework based on the Q-Learning algorithm is used, and the iterative training comprises the following specific steps:
(1) randomly initializing all parameters to be identified, substituting the parameters into a power transmission system model for calculation, and comparing the parameters with actually measured data to obtain a mean square error serving as an initial value of an optimal objective function F (X);
(2) the plurality of agents execute serial learning from large to small according to the sensitivity of the parameters to be identified; the link of adjusting the parameters to be identified according to the Q value table in the reinforcement learning frame is called an agent, and each parameter to be identified uniquely corresponds to one agent; the process of the intelligent agent adjusting the parameters to be identified is called the learning behavior of the intelligent agent;
the learning process comprises the following steps: randomly selecting an action a in the range of possible value intervals of the corresponding parameters of the current agentrand(i) Fixing other parameters, applying the parameters to the power transmission system model to obtain the objective function value F (X) under the current parameterscur) And a single step reward value r for updating the Q value table of the current agent according to the Q value updating strategy of claim 3; if F (X)cur)≤F(Xbest) If the current parameter has a search value, the step (3) is carried out, otherwise, the learning process is repeated;
(3) the action selection policy of claim 3 selecting an action aiter(i) Fixing other parameters, applying the parameters to FMU to obtain the value of target function F (X)cur) And a reward and punishment value r, updating the Q value table;
(4) iteratively executing steps (2) and (3) for the ith parameterSecondly, completing one identification of the ith parameter, and switching to the identification of the (i + 1) th parameter;
5. The method as claimed in claim 1, wherein the step S4 is to use PSO optimization algorithm to fine-tune the parameters to obtain the final recognition result.
6. The reinforcement learning-based power transmission system model parameter identification method according to claim 1, wherein the parameters are fine-tuned by using a PSO optimization algorithm, and the method comprises the following specific steps:
setting the fine tuning range toWherein the mu value is set according to the searching capability of the particle swarm algorithm,in order to strengthen the optimal subinterval obtained by learning,the parameter is the optimal value of the ith parameter after reinforcement learning identification;
initializing an N-dimensional space and a group of particles, creating a velocity vector v for each particleiPosition vector xiHistorical optimal position vector piAnd a historical optimal position vector p for the entire particle swarmg;
Establishing a fitness function G (X) by using the mean square error of a model predicted value and an actually measured value, wherein the model predicted value is obtained by substituting the current position of particles into an FMU for simulation and solving; the goal of the search of the entire population of particles is to minimize g (x);
starting iterative search: in one iteration, calculating the fitness value of each particle, and updating p of single particleiValue and p of populationgThe value, then according to the following formula update each particle velocity and position vector;
where ω is the inertial weight, c1、c2Is a learning factor, r1、r2Is [0,1 ]]A random number in between;
after the maximum iteration times are reached, the historical optimal position vector p of the particle swarmgThe final result of the parameter identification is obtained.
7. A powertrain system model parameter identification system based on reinforcement learning, comprising: a computer-readable storage medium and a processor;
the computer-readable storage medium is used for storing executable instructions;
the processor is configured to read executable instructions stored in the computer readable storage medium and execute the reinforcement learning-based powertrain system model parameter identification method of any one of claims 1-6.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110002104.6A CN112632860B (en) | 2021-01-04 | 2021-01-04 | Power transmission system model parameter identification method based on reinforcement learning |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110002104.6A CN112632860B (en) | 2021-01-04 | 2021-01-04 | Power transmission system model parameter identification method based on reinforcement learning |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112632860A true CN112632860A (en) | 2021-04-09 |
CN112632860B CN112632860B (en) | 2024-06-04 |
Family
ID=75290870
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110002104.6A Active CN112632860B (en) | 2021-01-04 | 2021-01-04 | Power transmission system model parameter identification method based on reinforcement learning |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112632860B (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113406434A (en) * | 2021-05-14 | 2021-09-17 | 杭州电子科技大学 | SVG dynamic parameter segmentation optimization identification method based on parameter fault characteristics |
CN113836788A (en) * | 2021-08-24 | 2021-12-24 | 浙江大学 | Acceleration method for flow industry reinforcement learning control based on local data enhancement |
CN114676572A (en) * | 2022-03-25 | 2022-06-28 | 中国航空发动机研究院 | Parameter determination method and device and computer readable storage medium |
CN114860388A (en) * | 2022-07-07 | 2022-08-05 | 中国汽车技术研究中心有限公司 | Combined simulation method for converting FMU (failure mode reporting) model into Modelica model |
DE102022104313A1 (en) | 2022-02-23 | 2023-08-24 | Dr. Ing. H.C. F. Porsche Aktiengesellschaft | Method, system and computer program product for autonomously calibrating an electric powertrain |
CN117312808A (en) * | 2023-11-30 | 2023-12-29 | 山东省科学院海洋仪器仪表研究所 | Calculation method for sea surface aerodynamic roughness |
CN118096085A (en) * | 2024-04-24 | 2024-05-28 | 山东冠县鑫恒祥面业有限公司 | Flour production line equipment operation and maintenance management method based on Internet of things |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107341294A (en) * | 2017-06-15 | 2017-11-10 | 苏州同元软控信息技术有限公司 | Spacecraft Information System Modeling emulation mode based on Modelica language |
CN109522602A (en) * | 2018-10-18 | 2019-03-26 | 北京航空航天大学 | A kind of Modelica Model Parameter Optimization method based on agent model |
US20190347370A1 (en) * | 2018-05-09 | 2019-11-14 | Palo Alto Research Center Incorporated | Learning constitutive equations of physical components with constraints discovery |
-
2021
- 2021-01-04 CN CN202110002104.6A patent/CN112632860B/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107341294A (en) * | 2017-06-15 | 2017-11-10 | 苏州同元软控信息技术有限公司 | Spacecraft Information System Modeling emulation mode based on Modelica language |
US20190347370A1 (en) * | 2018-05-09 | 2019-11-14 | Palo Alto Research Center Incorporated | Learning constitutive equations of physical components with constraints discovery |
CN109522602A (en) * | 2018-10-18 | 2019-03-26 | 北京航空航天大学 | A kind of Modelica Model Parameter Optimization method based on agent model |
Non-Patent Citations (1)
Title |
---|
吴义忠;蒋占四;陈立平;: "基于Modelica语言的多领域模型仿真优化研究", ***仿真学报, no. 12, 20 June 2009 (2009-06-20) * |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113406434A (en) * | 2021-05-14 | 2021-09-17 | 杭州电子科技大学 | SVG dynamic parameter segmentation optimization identification method based on parameter fault characteristics |
CN113406434B (en) * | 2021-05-14 | 2022-05-31 | 杭州电子科技大学 | SVG dynamic parameter segmentation optimization identification method based on parameter fault characteristics |
CN113836788A (en) * | 2021-08-24 | 2021-12-24 | 浙江大学 | Acceleration method for flow industry reinforcement learning control based on local data enhancement |
CN113836788B (en) * | 2021-08-24 | 2023-10-27 | 浙江大学 | Acceleration method for flow industrial reinforcement learning control based on local data enhancement |
DE102022104313A1 (en) | 2022-02-23 | 2023-08-24 | Dr. Ing. H.C. F. Porsche Aktiengesellschaft | Method, system and computer program product for autonomously calibrating an electric powertrain |
CN114676572A (en) * | 2022-03-25 | 2022-06-28 | 中国航空发动机研究院 | Parameter determination method and device and computer readable storage medium |
CN114676572B (en) * | 2022-03-25 | 2023-02-17 | 中国航空发动机研究院 | Parameter determination method and device and computer readable storage medium |
CN114860388A (en) * | 2022-07-07 | 2022-08-05 | 中国汽车技术研究中心有限公司 | Combined simulation method for converting FMU (failure mode reporting) model into Modelica model |
CN117312808A (en) * | 2023-11-30 | 2023-12-29 | 山东省科学院海洋仪器仪表研究所 | Calculation method for sea surface aerodynamic roughness |
CN117312808B (en) * | 2023-11-30 | 2024-02-06 | 山东省科学院海洋仪器仪表研究所 | Calculation method for sea surface aerodynamic roughness |
CN118096085A (en) * | 2024-04-24 | 2024-05-28 | 山东冠县鑫恒祥面业有限公司 | Flour production line equipment operation and maintenance management method based on Internet of things |
Also Published As
Publication number | Publication date |
---|---|
CN112632860B (en) | 2024-06-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112632860B (en) | Power transmission system model parameter identification method based on reinforcement learning | |
CN108621159B (en) | Robot dynamics modeling method based on deep learning | |
Vrabie et al. | Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems | |
CN111428849A (en) | Improved particle swarm algorithm-based transfer function model parameter identification method and device | |
US20220326664A1 (en) | Improved machine learning for technical systems | |
KR20170052344A (en) | Method and apparatus for searching new material | |
CN112395777B (en) | Engine calibration parameter optimization method based on automobile exhaust emission simulation environment | |
CN115291513A (en) | Boiler reheat steam temperature prediction method and device, terminal equipment and storage medium | |
CN114880806A (en) | New energy automobile sales prediction model parameter optimization method based on particle swarm optimization | |
Picotti et al. | Data-driven tuning of a nmpc controller for a virtual motorcycle through genetic algorithm | |
CN116933948A (en) | Prediction method and system based on improved seagull algorithm and back propagation neural network | |
CN115167102A (en) | Reinforced learning self-adaptive PID control method based on parallel dominant motion evaluation | |
CN114740710A (en) | Random nonlinear multi-agent reinforcement learning optimization formation control method | |
Menner et al. | Automated controller calibration by Kalman filtering | |
Stenger et al. | Benchmark of bayesian optimization and metaheuristics for control engineering tuning problems with crash constraints | |
Xu et al. | Meta-learning via weighted gradient update | |
Arshad et al. | Deep Deterministic Policy Gradient to Regulate Feedback Control Systems Using Reinforcement Learning. | |
Liu et al. | System identification based on generalized orthonormal basis function for unmanned helicopters: A reinforcement learning approach | |
CN114967472A (en) | Unmanned aerial vehicle trajectory tracking state compensation depth certainty strategy gradient control method | |
CN114139937A (en) | Indoor thermal comfort data generation method, system, equipment and medium | |
del Rio Ruiz et al. | Towards the development of a CAD tool for the implementation of high-speed embedded MPCs on FPGAs | |
Brasch et al. | Lateral control of a vehicle using reinforcement learning | |
CN113657604A (en) | Device and method for operating an inspection table | |
He et al. | A novel tuning method for predictive control of vav air conditioning system based on machine learning and improved PSO | |
CN111077769A (en) | Method for controlling or regulating a technical system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |