CN117991647A

CN117991647A - Performance monitoring and self-healing control method and device for industrial control system

Info

Publication number: CN117991647A
Application number: CN202410396935.XA
Authority: CN
Inventors: 杨旭; 高峰; 高晶晶; 崔家瑞; 李擎; 黄健; 胡章权
Original assignee: University of Science and Technology Beijing USTB
Current assignee: University of Science and Technology Beijing USTB
Priority date: 2024-04-03
Filing date: 2024-04-03
Publication date: 2024-05-07
Anticipated expiration: 2044-04-03
Also published as: CN117991647B

Abstract

The invention relates to the field of operation monitoring and self-healing control, in particular to a method and a device for monitoring and self-healing performance of an industrial control system, wherein the method comprises the following steps: acquiring real-time operation data of an industrial control system to be controlled; obtaining a real-time performance evaluation result of the industrial control system according to the real-time operation data and a performance degradation monitoring module based on a Bellman equation, and obtaining a performance degradation monitoring result of the industrial control system according to the real-time performance evaluation result; and according to the performance degradation monitoring result and the self-optimizing feedback supplement self-healing control module based on the reinforcement learning algorithm, obtaining the self-healing control result of the industrial control system. The invention realizes real-time performance monitoring by utilizing the data of the industrial operation process under the condition of not depending on the controlled object and the mathematical model of the control system; the original controller structural parameters are not changed, and only the data and reinforcement learning method is used for designing feedback compensation to recover performance degradation, so that self-healing control is realized.

Description

Performance monitoring and self-healing control method and device for industrial control system

Technical Field

The invention relates to the field of operation monitoring and self-healing control, in particular to a method and a device for monitoring and self-healing performance of an industrial control system.

Background

In recent years, the rapid development of social economy promotes the rapid increase of the demand of various industrial products, and simultaneously, higher requirements are also put forward on the quality of the products, and the modern industrial production gradually develops towards refinement and intensification. The degree of automation of industrial production process is continuously improved, and new challenges are presented to the design of safety and reliability of the industrial control system, so that the performance monitoring and self-healing control technology of the industrial control system is increasingly paid attention to.

The production environment of the industrial process is bad, the operation condition is complex, the limitation of materials and equipment is caused, the controlled object, the actuating mechanism and the sensor of the industrial control system can be failed, and the system performance is reduced; meanwhile, due to factors such as frequent switching of certain production conditions, abrasion and aging of equipment, improper operation and the like, abnormal operation conditions of a control system are easy to cause, performance degradation of the control system is further caused, normal operation of the control system is affected, product quality and yield are reduced, safety accidents are also caused due to serious conditions, and irrecoverable loss is caused. Therefore, it is necessary to establish a set of abnormal operation monitoring scheme and a self-healing recovery mechanism for coping with performance degradation of the control system for the industrial control system, so as to ensure safe and stable operation of the industrial process.

However, the actual industrial production process has complex structure, high process continuity, serious coupling condition between variables and unclear mechanism, and it is difficult to obtain an accurate mathematical model for designing a performance monitoring scheme and a self-healing control method. However, with the development of sensor technology and network communication technology, massive real-time data about complex industrial production operation processes are stored, and the data contains abundant industrial control system operation states and information. Therefore, how to utilize industrial operation data to effectively monitor the performance of the industrial production process control system and design a self-healing control mechanism with performance fading gradually becomes a technical problem worthy of research in the industry and academia.

The current popular performance evaluation method of the industrial control system is based on minimum variance performance evaluation, the method focuses on product quality, variance fluctuation of system output is utilized to measure the performance of the control system, but in the actual production process, the product quality is ensured, cost consumption is also required to be focused, and control cost also influences the evaluation of the performance of the control system, so that control input and process output of the industrial process need to be considered simultaneously in the process of designing the performance evaluation standard of the industrial control system.

In order to solve the problem, the LQG (Linear Quadratic Gaussian ) standard is put forward, so that not only is the variance of the output considered, but also the control input is added into the standard of system performance measurement, the information used in the evaluation process is enriched, and the performance evaluation standard is further more reasonable. However, the LQG-based reference has a high dependency on the model, and is constructed only after obtaining a relatively accurate mathematical model, so that in order to overcome the disadvantage, an LQG performance evaluation method based on subspace identification technology is proposed, but the process still needs to identify a parameter matrix of a controlled system, the required calculation cost is high, and performance monitoring is performed after the identification process, wherein errors also affect the performance evaluation effect.

In order to ensure that the industrial production process can complete corresponding control tasks under abnormal operation conditions and ensure the safety and reliability of system operation, fault-tolerant control technology is developed in industrial control systems. General fault-tolerant control techniques are classified into passive fault-tolerant and active fault-tolerant techniques according to whether they rely on fault diagnosis units. Among them, the more advantageous active fault-tolerant control approaches are roughly classified into control law reconstruction and control parameter rescheduling. Most industrial control systems, however, write parameters or programs in the control unit and are not easily modified. Meanwhile, most fault-oriented fault-tolerant control technologies are designed aiming at control units, performance-oriented fault-tolerant control system designs are in a development stage, and fault tolerance focusing on performance recovery of the control system gradually becomes a hot spot. With the appearance of self-healing control ideas, the research of self-healing control of the operation process from abnormal working conditions to normal working conditions is realized through control means, a flight control system, a power grid and an internet system are taken as research objects in the early stage, however, the research of the performance recovery self-healing means of the industrial process control system with environmental complexity and process dynamics is still in a starting stage. For industrial processes, the self-healing control means that the input and output of a control loop are regulated by an active control means under abnormal working conditions of the industrial process deviating from safe operation. At present, most of self-healing control methods aiming at abnormal working conditions are self-healing control methods taking an optimal control loop set value as a means, and lack of intelligent self-healing decisions aiming at complexity and dynamics of industrial processes, fusion of data and knowledge and real-time feedback of an execution process.

Disclosure of Invention

In order to solve the technical problems of performance monitoring and self-healing recovery of an industrial control system in a complex operating environment, the embodiment of the invention provides a method and a device for performance monitoring and self-healing control of the industrial control system. The technical scheme is as follows:

In one aspect, there is provided an industrial control system performance monitoring and self-healing control method implemented by an industrial control system performance monitoring and self-healing control device, the method comprising:

s1, acquiring real-time operation data of an industrial control system to be controlled.

S2, obtaining a real-time performance evaluation result of the industrial control system according to the real-time operation data and the performance degradation monitoring module based on the Bellman equation, and obtaining a performance degradation monitoring result of the industrial control system according to the real-time performance evaluation result.

And S3, supplementing a self-healing control module according to the performance degradation monitoring result and the self-optimization feedback based on the reinforcement learning algorithm to obtain a self-healing control result of the industrial control system.

Optionally, the performance degradation monitoring module in S2, which is based on the real-time operation data and Bellman equation, obtains a real-time performance evaluation result of the industrial control system, and obtains a performance degradation monitoring result of the industrial control system according to the real-time performance evaluation result, where the performance degradation monitoring module includes:

S21, acquiring input and output data in the operation process of the industrial control system.

S22, building performance indexes of the industrial control system according to the input and output data.

S23, obtaining a performance evaluation matrix of the industrial control system according to the mathematical description of the controlled object of the industrial control system, the mathematical description of the controller, the mathematical description of the reference input generator, the Bellman equation and the performance index.

S24, designing an identification method of a performance monitoring matrix of the industrial control system according to the performance index and the performance evaluation matrix.

S25, obtaining the performance monitoring reference matrix of the industrial control system according to the input and output data and the identification method of the performance monitoring matrix.

S26, obtaining a real-time performance evaluation result of the industrial control system according to the real-time operation data and the identification method of the performance monitoring matrix.

And S27, obtaining performance degradation monitoring indexes of the industrial control system according to the performance monitoring reference matrix and the real-time performance evaluation result, and obtaining the performance degradation monitoring result of the industrial control system according to the performance degradation monitoring indexes and the designed monitoring logic.

Optionally, the performance index of the industrial control system in S22 is as shown in the following formula (1):

（1）

In the method, in the process of the invention, Representing performance index of industrial control system,/>Representing discount factors,/>Representing the sampling instant of the data,/>，/>Tracking error matrix representing system output,/>Representing the transpose of the matrix,/>Representing system output data,/>Representing reference input,/>Representing a tracking error weight matrix,/>Representing a matrix of control input weights,/>Representing system input data.

Optionally, obtaining the performance evaluation matrix of the industrial control system according to the mathematical description of the controlled object of the industrial control system, the mathematical description of the controller, the mathematical description of the reference input generator, the Bellman equation, and the performance index in S23 includes:

S231, constructing an augmentation system according to the mathematical description of the controlled object of the industrial control system, the mathematical description of the controller and the mathematical description of the reference input generator.

S232, rewriting the performance index into a recursive form of a Bellman equation.

S233, obtaining a parameterized form of the performance index of the industrial control system according to the augmentation system, the recursion form and the Bellman optimality principle.

S234, obtaining a performance evaluation matrix of the industrial control system according to the parameterized form, wherein the performance evaluation matrix is shown in the following formula (2):

（2）

In the method, in the process of the invention, Performance evaluation matrix representing an industrial control system,/>Representing a matrix of weights of the tracking error,Representing an output-related parameter matrix of an augmentation system,/>，Representing the reference input state quantity,/>Representing the state quantity of the system,/>Representing a matrix of control input weights,Representing an augmented system control input related parameter matrix,/>Output parameter matrix representing reference input,/>Representing feedback control gain,/>Representing the output parameter matrix of the system,/>Representing the discount factor(s),，/>，/>State parameter matrix representing reference input,/>Representing a state parameter matrix of the system,/>，/>Representing a matrix of input parameters of the system.

Optionally, the identifying method of the performance monitoring matrix of the industrial control system is designed according to the performance index and the performance evaluation matrix in S24, including:

S241, obtaining the rewritten performance index according to the performance evaluation matrix and the performance index.

S242, obtaining the replaced performance index according to the rewritten performance index and the Kalman filter type data model.

S243, vectorizing the replaced performance index to obtain the vectorized performance index.

S244, obtaining a performance monitoring matrix of the industrial control system according to the input and output data, the vectorized performance index and the recursive least square identification method.

Optionally, the self-optimizing feedback supplementing self-healing control module based on the performance degradation monitoring result and the reinforcement learning algorithm in S3 obtains a self-healing control result of the industrial control system, which includes:

s31, acquiring input and output data in the operation process of the industrial control system.

S32, designing a representation form of a self-healing control law.

S33, defining a Q value function according to the performance index of the industrial control system and the parameterized form of the performance index, and rewriting the Q value function according to the representation form of the self-healing control law to obtain the Q value function of the self-healing control law.

S34, minimizing the Q value function of the self-healing control law to obtain the self-healing control law parameter representation form.

S35, obtaining the self-healing control law according to the input and output data, the Q value function of the self-healing control law, the self-healing control law parameter representation form and the Q-learning algorithm.

S36, obtaining a self-healing control result of the industrial control system according to the performance degradation monitoring result and the self-healing control law.

Optionally, obtaining the self-healing control law according to the input/output data, the Q value function of the self-healing control law, the parameter representation form of the self-healing control law and the Q-learning algorithm in S35 includes:

s351, rewriting the Q value function of the self-healing control law to obtain the Q value function of the self-healing control law after rewriting.

S352, obtaining the self-healing control law representation according to the representation form, the parameter representation form and the Q value function of the rewritten self-healing control law.

S353, constructing a time sequence difference TD error representation form, and vectorizing the TD error representation form to obtain the vectorized TD error representation form.

S354, obtaining the self-healing control law according to the input and output data, the vectorized TD error representation form, the self-healing control law representation and the Q-learning algorithm.

In another aspect, there is provided an apparatus for monitoring and self-healing performance of an industrial control system, the apparatus being applied to a method for monitoring and self-healing performance of an industrial control system, the apparatus comprising:

And the acquisition module is used for acquiring real-time operation data of the industrial control system to be controlled.

And the performance monitoring module is used for obtaining a real-time performance evaluation result of the industrial control system according to the real-time operation data and the performance degradation monitoring module based on the Bellman equation and obtaining a performance degradation monitoring result of the industrial control system according to the real-time performance evaluation result.

And the self-healing control module is used for supplementing the self-healing control module according to the performance degradation monitoring result and the self-optimization feedback based on the reinforcement learning algorithm to obtain the self-healing control result of the industrial control system.

Optionally, the performance monitoring module is further configured to:

Optionally, the performance index of the industrial control system is as shown in the following formula (1):

（1）

Optionally, the performance monitoring module is further configured to:

（2）

Optionally, the performance monitoring module is further configured to:

Optionally, the self-healing control module is further configured to:

S32, designing a representation form of a self-healing control law.

Optionally, the self-healing control module is further configured to:

In another aspect, there is provided an industrial control system performance monitoring and self-healing control apparatus, the industrial control system performance monitoring and self-healing control apparatus comprising: a processor; and a memory having stored thereon computer readable instructions which, when executed by the processor, implement any one of the industrial control system performance monitoring and self-healing control methods described above.

In another aspect, a computer readable storage medium having stored therein at least one instruction that is loaded and executed by a processor to implement any of the above-described industrial control system performance monitoring and self-healing control methods is provided.

The technical scheme provided by the embodiment of the invention has the beneficial effects that at least:

In the embodiment of the invention, only input and output data are utilized, the performance of the control system in the industrial process is monitored under the condition of not depending on the controlled object and the accurate mathematical model of the control system, and meanwhile, the performance of the control system in an abnormal running state is recovered to a certain extent by a data driving feedback mode based on a reinforcement learning method under the premise of not changing a preset controller, so that self-healing control is realized.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a method for monitoring and self-healing control of the performance of an industrial control system according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of monitoring abnormal performance and self-healing control of performance degradation of an industrial control system according to an embodiment of the present invention;

FIG. 3 is a block diagram of an industrial control system performance monitoring and self-healing control device according to an embodiment of the present invention;

Fig. 4 is a schematic structural diagram of an industrial control system performance monitoring and self-healing control device according to an embodiment of the present invention.

Detailed Description

The technical scheme of the invention is described below with reference to the accompanying drawings.

In embodiments of the invention, words such as "exemplary," "such as" and the like are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, the term use of an example is intended to present concepts in a concrete fashion. Furthermore, in embodiments of the present invention, the meaning of "and/or" may be that of both, or may be that of either, optionally one of both.

In the embodiments of the present invention, "image" and "picture" may be sometimes used in combination, and it should be noted that the meaning of the expression is consistent when the distinction is not emphasized. "of", "corresponding (corresponding, relevant)" and "corresponding (corresponding)" are sometimes used in combination, and it should be noted that the meaning of the expression is consistent when the distinction is not emphasized.

In embodiments of the present invention, sometimes a subscript such as W ₁ may be expressed in a non-subscript form such as W1, and the meaning of the expression is consistent when de-emphasizing the distinction.

In order to make the technical problems, technical solutions and advantages to be solved more apparent, the following detailed description will be given with reference to the accompanying drawings and specific embodiments.

The embodiment of the invention provides an industrial control system performance monitoring and self-healing control method, which can be realized by industrial control system performance monitoring and self-healing control equipment, wherein the industrial control system performance monitoring and self-healing control equipment can be a terminal or a server. The flow chart of the industrial control system performance monitoring and self-healing control method shown in fig. 1, the process flow of the method can comprise the following steps:

In a feasible implementation manner, in order to solve the problems of abnormal monitoring and self-healing control of performance degradation of an industrial control system in a complex operation environment, the invention designs a performance degradation monitoring module based on a Bellman equation and a self-optimizing feedback supplementing self-healing control module based on a reinforcement learning algorithm, which is triggered according to the monitoring result of the performance degradation monitoring module, as shown in fig. 2.

In the present invention, control performance monitoring (Control Performance Monitoring): the method is used for monitoring the trend and the state of the statistical value which can reflect the control performance of the system and change along with the time.

Self-healing Control (Self-healing Control): the system can timely detect abnormal states and take reasonable measures to inhibit or eliminate the influence of the abnormality on the system performance so as to improve the running reliability of the system.

Optionally, the step S2 may include the following steps S21 to S27:

s21, acquiring, transmitting and storing input/output data (I/O data) in the operation process through an industrial sensor and a data storage device, wherein the input/output data (I/O data) comprises a control input And system output/>。

In a possible embodiment, the performance index form in step S22 is defined as follows, taking into account both the system output and the control input:

（1）

In the method, in the process of the invention, Representing performance index of industrial control system,/>Representing discount factors, in order to make the cost function bounded, the value range is/>，/>Representing the sampling instant of the data, the single step cost is represented as，/>Tracking error matrix representing system output,/>Representing the transpose of the matrix,/>Representing system output data,/>Representing reference input,/>Representing a tracking error weight matrix,/>Representing a matrix of control input weights,/>Representing system input data.

S23, obtaining a representation form of a closed-loop control system performance evaluation matrix by using a Bellman equation according to the controlled object, the controller and the general mathematical description of the reference input generator.

Optionally, the step S23 may include the following steps S231 to S2314:

In a possible embodiment, a general mathematical description of the controlled object, the controller and the reference input generator is given.

Specifically, assume that the state space of an industrial control closed loop system is described as:

（2）

In the method, in the process of the invention, 、/>And/>Is a system parameter matrix,/>、/>And/>Respectively expressed at/>Control input, state input and state output of moment; /(I)Representing feedback control gain,/>Representing a reference input.

Giving reference inputThe form of (2): /(I)。

S232, rewriting the performance index into a recursive form of a Bellman equation:

（3）

Wherein, 。

In a possible embodiment, the performance index is parameterized according to the principle of optimality in Bellman and the recursive representation (3), wherein an augmentation system is constructed consisting of the controlled object and the reference input, wherein the state of the augmentation system is represented asI.e. the closed loop control system is described as:

（4）

Wherein the method comprises the steps of ，/>，/>，/>。

Thereafter, the parameterized form of the performance index can be expressed as an augmented state quadratic form as follows:

（5）

Bringing form (5) into recursive form (3), there are;

（6）

a parameterized performance evaluation matrix can be obtained In the above, the ratio of/>Performance evaluation matrix representing an industrial control system,/>Representing a tracking error weight matrix,/>Representing a matrix of parameters related to the output of the augmentation system,，/>Representing the reference input state quantity,/>Representing the state quantity of the system,/>Representing a matrix of control input weights,/>Representing an augmented system control input related parameter matrix,/>Output parameter matrix representing reference input,/>Representing feedback control gain,/>Representing the output parameter matrix of the system,/>Representing discount factors,/>，/>，/>State parameter matrix representing reference input,/>Representing a state parameter matrix of the system,/>，/>Representing a matrix of input parameters of the system.

In one possible implementation, the system performance monitoring matrix is directly identified using data identification techniques.

Optionally, the step S24 may include the following steps S241 to S244:

In one possible implementation, the performance evaluation matrix may be equivalently represented asFurther, the performance index can be rewritten as follows:

（7）

s242, in order to identify a performance evaluation matrix by using input/output data, a Kalman filter type data model (I/O model) is introduced, specifically:

（8）

Wherein the method comprises the steps of ，/>If step size/>Large enough (larger than the system order), there is/>I.e. can be approximated by:

（9）

Parameter matrix:

Data matrix:

，/>，。

s243 using the formula (9) Substitution/>The method comprises the following steps: /(I)

（10）

Wherein a performance monitoring matrix is defined。

S244 to enable performance monitoring matrix by direct recognitionAfter vectorization of the above formula, it is possible to obtain:

（11）

And (3) making:

；

The method comprises the following steps:

（12）

the data collected in step S21 can be further utilized to obtain the data through a recursive least squares identification method Thereby being capable of directly obtaining the performance monitoring matrix/>。

S25, collecting stored historical operation data based on the step S21, and obtaining a performance evaluation benchmark based on a Bellman equation through the step S24 by utilizing the data of normal operation of the industrial process in a stable state.

S26, acquiring real-time operation data on line, and acquiring real-time performance evaluation results of the industrial control system on line by utilizing the data according to the identification method in the fourth step.

And S27, comparing the performance evaluation standard obtained in the step S25 with the real-time performance evaluation result obtained in the step S26 to obtain a performance degradation monitoring index of the industrial control system, and giving a performance degradation monitoring result according to the designed monitoring logic.

In a possible embodiment, according to step S25, a performance monitoring reference matrix is obtainedAccording to S26, a real-time performance monitoring matrix/>, is obtained。

Further, defining performance monitoring indexAnd the index threshold value of normal operation is/>。

Further, an industrial control system performance degradation monitoring logic is designed:

（13）

Optionally, the step S3 may include the following steps S31 to S36:

In a possible implementation, step S31 is the same as step S21, and input/output data during operation is collected and stored.

S32, designing a feedback supplement type self-healing control law which does not change the performance recovery of the industrial control system of the original controller.

In one possible implementation, the design performance recovery self-healing control law is in the form of:

（14）

Original controller Unchanged and define self-healing control law/>Parameterized form.

S33, further obtaining a Q value function related to a self-healing control law in a performance recovery mechanism of the industrial control system according to the performance index of the industrial control system defined in the step S22 of the first functional module and the state space representation form of the closed-loop control system described in the step S23.

In a possible embodiment, the following Q-value function is defined:

（15）

further, it is rewritten as a result of And/>The form represented: /(I)

（16）

Wherein,，/>，，/>，/>。

To this end, a control law about self-healing recovery is obtainedExpressed as a Q function of (c).

S34, obtaining a self-healing control law parameter representation form according to the Q value function obtained in the step three to minimize the value of the Q value function.

In a possible embodiment, the control law is recovered with respect to self-healing obtained according to step S33In order to minimize its value, the Q-function expression is required for/>Is zero, i.e。

Further, according to the correlation matrix in step S33The self-healing control law expression form is obtained as follows:

（17）

S35, in a model-free mode, based on the historical operation data collected and stored in the first step, the self-healing control law related parameters are solved through a Q-learning algorithm.

Optionally, the step S35 may include the following steps S351-S355:

In a possible implementation, to implement model-free solution, according to equation (9) of step S24 in the functional module one, the Q-value function in step three is expressed as follows:

（18）

In a possible implementation manner, according to the self-healing control law parameterized form designed in step S32 and the self-healing control law expression in step S34, the parameters of the self-healing control law are obtained according to the description form of (18):，/>。

Namely, the self-healing control law is expressed as:

（19）

s353, constructing a representation form of the time sequence difference TD error:

（20）

vectorizing the obtained product to obtain the product:

（21）

Wherein,

，。

S354, minimizing TD error according to Q-learning algorithm, and performing recursive least square identification according to the operation data collected in step S31 and data with a certain step length to obtainThe expression is as follows:

（22）

when TD (Temporal Difference) errors meet the minimum, output Further can obtain the self-healing recovery control law/>。

S36, triggering a performance degradation recovery mechanism in time according to the monitoring result of the first functional module, and calling the self-healing control law parameter to realize performance recovery.

The invention constructs the performance index of the closed-loop control system based on the Belman equation, designs the control performance evaluation matrix, and constructs reasonable performance degradation monitoring logic; the performance evaluation is performed after the recognition system model is replaced by a direct recognition evaluation matrix, so that errors caused by redundant recognition processes are avoided; and further designing a control performance degradation recovery method based on the reinforcement learning theory by utilizing the performance index in the form of the Bellman equation.

Aiming at the situation that the common industrial control system performance monitoring method is conservative in designing and evaluating the benchmark and the situation that part of the evaluating benchmark depends on an accurate mathematical model of the industrial process, the control input is added to the design process of the benchmark, and meanwhile, the industrial control system performance evaluating benchmark is built by using only measured data. Furthermore, the performance evaluation matrix of the closed loop control system is directly identified, so that identification errors caused by a performance evaluation mode of identifying system parameters and then performing LQG reference design are reduced, and performance evaluation accuracy is improved.

Aiming at the problem of control performance decline caused by abnormal operation conditions of an industrial process in a complex environment, the design of a feedback supplement based on a reinforcement learning method is realized by considering that a pre-designed controller or an algorithm program is solidified in hardware equipment and is not easy to modify and adjust, and the performance recovery of a control system is realized by using data only on the premise of not changing the original controller.

The invention aims to solve the problems of industrial control system performance monitoring and self-healing recovery in a complex operating environment. Comprising the following steps: a control system performance evaluation matrix and monitoring scheme based on a Bellman equation and a control system performance self-healing control method based on a reinforcement learning method are designed. Real-time performance monitoring is realized by utilizing the data of the industrial operation process under the condition of not depending on the controlled object and the mathematical model of the control system; the original controller structural parameters are not changed, and only the data and reinforcement learning method is used for designing feedback compensation to recover performance degradation, so that self-healing control is realized.

FIG. 3 is a block diagram of an industrial control system performance monitoring and self-healing control device for an industrial control system performance monitoring and self-healing control method, according to an exemplary embodiment. Referring to fig. 3, the apparatus includes an acquisition module 310, a performance monitoring module 320, and a self-healing control module 330. Wherein:

An acquisition module 310 is configured to acquire real-time operation data of an industrial control system to be controlled.

The performance monitoring module 320 is configured to obtain a real-time performance evaluation result of the industrial control system according to the real-time operation data and the performance degradation monitoring module based on the Bellman equation, and obtain a performance degradation monitoring result of the industrial control system according to the real-time performance evaluation result.

The self-healing control module 330 is configured to supplement the self-healing control module with self-optimization feedback based on the reinforcement learning algorithm according to the performance degradation monitoring result, and obtain a self-healing control result of the industrial control system.

Optionally, the performance monitoring module 320 is further configured to:

（1）

Optionally, the performance monitoring module 320 is further configured to:

（2）

Optionally, the performance monitoring module 320 is further configured to:

Optionally, the self-healing control module 330 is further configured to:

S32, designing a representation form of a self-healing control law.

Optionally, the self-healing control module 330 is further configured to:

Fig. 4 is a schematic structural diagram of an industrial control system performance monitoring and self-healing control device according to an embodiment of the present invention, where as shown in fig. 4, the industrial control system performance monitoring and self-healing control device may include the industrial control system performance monitoring and self-healing control device shown in fig. 3. Alternatively, the industrial control system performance monitoring and self-healing control device 410 may include the first processor 2001.

Optionally, the industrial control system performance monitoring and self-healing control device 410 may also include a memory 2002 and a transceiver 2003.

The first processor 2001 may be connected to the memory 2002 and the transceiver 2003, for example, via a communication bus.

The various constituent elements of the industrial control system performance monitoring and self-healing control device 410 are described in detail below in conjunction with FIG. 4:

The first processor 2001 is a control center of the industrial control system performance monitoring and self-healing control device 410, and may be one processor or a generic name of a plurality of processing elements. For example, the first processor 2001 is one or more central processing units (central processing unit, CPU), may be an Application SPECIFIC INTEGRATED Circuit (ASIC), or may be one or more integrated circuits configured to implement embodiments of the present invention, such as: one or more microprocessors (DIGITAL SIGNAL processors, DSPs), or one or more field programmable gate arrays (field programmable GATE ARRAY, FPGAs).

Alternatively, the first processor 2001 may perform various functions of the industrial control system performance monitoring and self-healing control device 410 by running or executing a software program stored in the memory 2002 and invoking data stored in the memory 2002.

In a specific implementation, first processor 2001 may include one or more CPUs, such as CPU0 and CPU1 shown in fig. 4, as an example.

In a particular implementation, as one example, the industrial control system performance monitoring and self-healing control device 410 may also include a plurality of processors, such as the first processor 2001 and the second processor 2004 shown in FIG. 4. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).

The memory 2002 is used for storing a software program for executing the solution of the present invention, and is controlled by the first processor 2001 to execute the solution, and the specific implementation may refer to the above method embodiment, which is not described herein.

Alternatively, memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (random access memory, RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (ELECTRICALLY ERASABLE PROGRAMMABLE READ-only memory, EEPROM), compact disc read-only memory (compact disc read-only memory) or other optical disk storage, optical disk storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, without limitation. The memory 2002 may be integrated with the first processor 2001, may be present independently, and may be coupled to the first processor 2001 through an interface circuit (not shown in fig. 4) of the industrial control system performance monitoring and self-healing control device 410, as embodiments of the present invention are not specifically limited in this regard.

A transceiver 2003 for communicating with a network device or with a terminal device.

Alternatively, transceiver 2003 may include a receiver and a transmitter (not separately shown in fig. 4). The receiver is used for realizing the receiving function, and the transmitter is used for realizing the transmitting function.

Alternatively, the transceiver 2003 may be integrated with the first processor 2001, or may exist separately, and be coupled to the first processor 2001 through an interface circuit (not shown in fig. 4) of the industrial control system performance monitoring and self-healing control device 410, as embodiments of the present invention are not limited in this regard.

It should be noted that the structure of the industrial control system performance monitoring and self-healing control device 410 shown in fig. 4 is not limited to this router, and an actual knowledge structure recognition device may include more or less components than illustrated, or may combine certain components, or may have a different arrangement of components.

In addition, the technical effects of the industrial control system performance monitoring and self-healing control device 410 may refer to the technical effects of the industrial control system performance monitoring and self-healing control method described in the above method embodiments, and are not described herein again.

It is to be appreciated that the first processor 2001 in embodiments of the invention may be a central processing unit (central processing unit, CPU) which may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSPs), application Specific Integrated Circuits (ASICs), off-the-shelf programmable gate arrays (field programmable GATE ARRAY, FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It should also be appreciated that the memory in embodiments of the present invention may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an erasable programmable ROM (erasable PROM), an electrically erasable programmable EPROM (EEPROM), or a flash memory. The volatile memory may be random access memory (random access memory, RAM) which acts as external cache memory. By way of example, and not limitation, many forms of random access memory (random access memory, RAM) are available, such as static random access memory (STATIC RAM, SRAM), dynamic Random Access Memory (DRAM), synchronous Dynamic Random Access Memory (SDRAM), double data rate synchronous dynamic random access memory (double DATA RATE SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (ENHANCED SDRAM, ESDRAM), synchronous link dynamic random access memory (SYNCHLINK DRAM, SLDRAM), and direct memory bus random access memory (direct rambus RAM, DR RAM).

The above embodiments may be implemented in whole or in part by software, hardware (e.g., circuitry), firmware, or any other combination. When implemented in software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with embodiments of the present invention are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more sets of available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium may be a solid state disk.

It should be understood that the term "and/or" is merely an association relationship describing the associated object, and means that three relationships may exist, for example, a and/or B may mean: there are three cases, a alone, a and B together, and B alone, wherein a, B may be singular or plural. In addition, the character "/" herein generally indicates that the associated object is an "or" relationship, but may also indicate an "and/or" relationship, and may be understood by referring to the context.

In the present invention, "at least one" means one or more, and "a plurality" means two or more. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b, or c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or plural.

It should be understood that, in various embodiments of the present invention, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

It will be clearly understood by those skilled in the art that, for convenience and brevity of description, specific working procedures of the apparatus, device and unit described above may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus, device and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another device, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An industrial control system performance monitoring and self-healing control method, which is characterized by comprising the following steps:

s1, acquiring real-time operation data of an industrial control system to be controlled;

S2, obtaining a real-time performance evaluation result of the industrial control system according to the real-time operation data and a performance degradation monitoring module based on a Bellman equation, and obtaining a performance degradation monitoring result of the industrial control system according to the real-time performance evaluation result;

2. The method for monitoring and self-healing control of performance of an industrial control system according to claim 1, wherein the performance degradation monitoring module in S2 obtains a real-time performance evaluation result of the industrial control system according to the real-time operation data and Bellman equation, and obtains a performance degradation monitoring result of the industrial control system according to the real-time performance evaluation result, comprising:

S21, acquiring input and output data in the operation process of the industrial control system;

s22, building performance indexes of the industrial control system according to the input and output data;

s23, obtaining a performance evaluation matrix of the industrial control system according to the mathematical description of the controlled object of the industrial control system, the mathematical description of the controller, the mathematical description of the reference input generator, the Bellman equation and the performance index;

s24, designing an identification method of a performance monitoring matrix of the industrial control system according to the performance index and the performance evaluation matrix;

s25, obtaining a performance monitoring reference matrix of the industrial control system according to the input and output data and the identification method of the performance monitoring matrix;

s26, obtaining a real-time performance evaluation result of the industrial control system according to the real-time operation data and the identification method of the performance monitoring matrix;

And S27, obtaining a performance degradation monitoring index of the industrial control system according to the performance monitoring reference matrix and the real-time performance evaluation result, and obtaining a performance degradation monitoring result of the industrial control system according to the performance degradation monitoring index and the designed monitoring logic.

3. The method for monitoring and self-healing control of performance of an industrial control system according to claim 2, wherein the performance index of the industrial control system in S22 is represented by the following formula (1):

（1）

4. The method for monitoring and self-healing control of performance of an industrial control system according to claim 2, wherein the obtaining the performance evaluation matrix of the industrial control system according to the mathematical description of the controlled object of the industrial control system, the mathematical description of the controller, the mathematical description of the reference input generator, the Bellman equation, and the performance index in S23 includes:

s231, constructing an augmentation system according to mathematical description of a controlled object of the industrial control system, mathematical description of a controller and mathematical description of a reference input generator;

s232, rewriting the performance index into a recursive form of a Bellman equation;

s233, obtaining a parameterized form of the performance index of the industrial control system according to the augmentation system, the recursion form and the Bellman optimality principle;

（2）

5. The method for monitoring and self-healing control of performance of an industrial control system according to claim 2, wherein the identifying method for designing the performance monitoring matrix of the industrial control system according to the performance index and the performance evaluation matrix in S24 comprises:

s241, obtaining rewritten performance indexes according to the performance evaluation matrix and the performance indexes;

s242, obtaining a replaced performance index according to the rewritten performance index and a Kalman filter form data model;

S243, vectorizing the replaced performance index to obtain a vectorized performance index;

6. The method for monitoring and self-healing control of industrial control system according to claim 1, wherein the step of obtaining the self-healing control result of the industrial control system according to the performance degradation monitoring result and the self-optimization feedback supplementary self-healing control module based on the reinforcement learning algorithm in step S3 comprises the following steps:

s31, acquiring input and output data in the operation process of the industrial control system;

s32, designing a representation form of a self-healing control law;

S33, defining a Q value function according to the performance index of the industrial control system and the parameterized form of the performance index, and rewriting the Q value function according to the representation form of the self-healing control law to obtain the Q value function of the self-healing control law;

s34, minimizing the Q value function of the self-healing control law to obtain a self-healing control law parameter representation form;

S35, obtaining a self-healing control law according to the input and output data, the Q value function of the self-healing control law, the self-healing control law parameter representation form and the Q-learning algorithm;

7. The method for monitoring and self-healing control of industrial control system according to claim 6, wherein the step of obtaining the self-healing control law according to the input/output data, the Q-value function of the self-healing control law, the parameter representation of the self-healing control law and the Q-learning algorithm in S35 comprises:

S351, rewriting the Q value function of the self-healing control law to obtain the Q value function of the rewritten self-healing control law;

S352, obtaining a self-healing control law representation according to the representation form of the self-healing control law, the representation form of the self-healing control law parameter and the Q value function of the self-healing control law after rewriting;

s353, constructing a time sequence difference TD error representation form, and vectorizing the TD error representation form to obtain a vectorized TD error representation form;

S354, obtaining the self-healing control law according to the input and output data, the vectorized TD error expression form, the self-healing control law expression and the Q-learning algorithm.

8. An industrial control system performance monitoring and self-healing control device for implementing the industrial control system performance monitoring and self-healing control method according to any one of claims 1 to 7, characterized in that the device comprises:

the acquisition module is used for acquiring real-time operation data of the industrial control system to be controlled;

The performance monitoring module is used for obtaining a real-time performance evaluation result of the industrial control system according to the real-time operation data and the performance degradation monitoring module based on the Bellman equation, and obtaining a performance degradation monitoring result of the industrial control system according to the real-time performance evaluation result;

9. An industrial control system performance monitoring and self-healing control device, characterized in that the industrial control system performance monitoring and self-healing control device comprises:

A processor;

a memory having stored thereon computer readable instructions which, when executed by the processor, implement the method of any of claims 1 to 7.

10. A computer readable storage medium having stored therein program code which is callable by a processor to perform the method of any one of claims 1 to 7.