CN114329936B - Virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning - Google Patents
Virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning Download PDFInfo
- Publication number
- CN114329936B CN114329936B CN202111577141.6A CN202111577141A CN114329936B CN 114329936 B CN114329936 B CN 114329936B CN 202111577141 A CN202111577141 A CN 202111577141A CN 114329936 B CN114329936 B CN 114329936B
- Authority
- CN
- China
- Prior art keywords
- coal
- intelligent
- information
- mining
- virtual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 151
- 238000004519 manufacturing process Methods 0.000 title claims abstract description 41
- 230000002787 reinforcement Effects 0.000 title claims abstract description 30
- 239000003245 coal Substances 0.000 claims abstract description 247
- 238000005065 mining Methods 0.000 claims abstract description 178
- 239000003795 chemical substances by application Substances 0.000 claims abstract description 142
- 230000008569 process Effects 0.000 claims abstract description 90
- 238000005520 cutting process Methods 0.000 claims abstract description 37
- 238000012549 training Methods 0.000 claims abstract description 18
- 230000008878 coupling Effects 0.000 claims abstract description 11
- 238000010168 coupling process Methods 0.000 claims abstract description 11
- 238000005859 coupling reaction Methods 0.000 claims abstract description 11
- 238000001514 detection method Methods 0.000 claims abstract description 6
- 230000009471 action Effects 0.000 claims description 43
- 230000007246 mechanism Effects 0.000 claims description 27
- 230000006870 function Effects 0.000 claims description 16
- 230000033001 locomotion Effects 0.000 claims description 15
- 230000000694 effects Effects 0.000 claims description 13
- 230000009916 joint effect Effects 0.000 claims description 11
- 238000005452 bending Methods 0.000 claims description 9
- 230000003993 interaction Effects 0.000 claims description 9
- 238000013486 operation strategy Methods 0.000 claims description 9
- 230000006399 behavior Effects 0.000 claims description 8
- 238000005457 optimization Methods 0.000 claims description 7
- 239000011435 rock Substances 0.000 claims description 7
- 230000033228 biological regulation Effects 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 6
- 230000007704 transition Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000003631 expected effect Effects 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims description 3
- 230000014759 maintenance of location Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 10
- 238000004458 analytical method Methods 0.000 description 9
- 238000012544 monitoring process Methods 0.000 description 7
- 230000036544 posture Effects 0.000 description 7
- 238000012360 testing method Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000004088 simulation Methods 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000005465 channeling Effects 0.000 description 3
- 230000000875 corresponding effect Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000005553 drilling Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000036541 health Effects 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 230000008447 perception Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000001276 controlling effect Effects 0.000 description 2
- 230000002079 cooperative effect Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 208000028752 abnormal posture Diseases 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000000540 analysis of variance Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000010998 test method Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to a deduction method of a virtual fully-mechanized production system based on multi-agent deep reinforcement learning, which is used for carrying out three-dimensional modeling on an agent body of a coal machine equipment according to respective structural parameters and degree of freedom characteristics of a fully-mechanized working face, and carrying out modeling on a virtual coal seam according to coal seam detection information and coal cutter cutting information; importing the established model into Unity3D as an initial scene of three machines of the whole fully mechanized mining face, accessing historical operation information in the equipment mining process into the scene, and extracting key operation information of coal mill equipment as a data driving source of initial operation; and carrying out centralized iterative training on the equipment operation data, obtaining decision information of equipment operation in a distributed control mode, and finally realizing 'three-machine' collaborative propulsion of the fully mechanized mining face and dynamic coupling with a working space under the control of a decision result so as to provide a technical platform support for the efficient operation of a fully mechanized mining face production system.
Description
Technical Field
The invention relates to the technical field of fully-mechanized coal face simulation, in particular to a virtual fully-mechanized coal face production system deduction method based on multi-agent deep reinforcement learning.
Background
Along with the advancement of intelligent national manufacturing, the digital twin technology gradually increases in the manufacturing process, and pursues a more efficient, safer and more transparent production and manufacturing mode, so that the realization of the above contents cannot be separated from the mapping effect of the virtual scene, and a virtual scene completely equivalent to a physical production and manufacturing scene needs to be established. Coal mining belongs to a deep working space, and the mining specificity makes the development of coal mine intelligentization slower than the development of other industries, so that a virtual mining process for establishing a complete mapping of the coal mining process is needed.
The mining operation of the fully-mechanized mining face is taken as an important ring of the coal mining process, more equipment is involved in the mining process, the cooperative requirements on equipment in the mining process are high, and the requirements on the process and straightness are also high in the mining process, so that the challenge in realizing the cooperative propulsion of the full life cycle of the fully-mechanized mining face is relatively large.
The invention patent of publication number CN111140231A discloses a coal seam roof and floor path virtual planning method for fully mechanized mining equipment space-time kinematics, and an inherent coal seam roof and floor is established through Unity3D software; and constructing a space-time kinematic relation between fully mechanized coal mining equipment and a coal seam roof and floor by using a physical engine, dynamically generating a single-cycle coal seam roof and floor by using a mesh assembly, and cooperatively propelling a scraper conveyor and a hydraulic support along with the guidance of a coal mining machine. The invention patent of publication number CN109783962A discloses a comprehensive mechanized coal mining equipment collaborative propulsion simulation method based on a virtual reality physical engine, which is characterized in that virtual comprehensive mechanized coal mining equipment is subjected to model rigid repair and then is virtually contacted with a virtual coal seam, so that underground operation information of the equipment is simulated, a virtual real-time updated coal seam is realized by real-time recording front and rear roller cutting tracks of a coal mining machine and performing Mesh grid collision reconstruction in Unity3D software, and the self-adaptive propulsion process of the equipment in an underground coal seam environment is truly reproduced by controlling the existence and display of inherent coal seam information and virtual real-time updated coal seam attributes. The invention patent of publication number CN108643884A discloses a jumbolter propulsion and rotation system and a cooperative self-adaptive control method thereof, which are used for implementing active disturbance rejection control on a single loop of the propulsion system and the rotation system, and enhancing the robustness of the drilling process; and determining the optimal propelling force and the optimal rotation speed of the drilling machine when drilling according to the estimated rock hardness coefficient by adopting a compound control mode, so as to realize the cooperative action of the anchor rod rotation and the propelling system. The invention patent of publication number CN112392485A discloses a transparent digital twin self-adaptive mining system and method for a fully mechanized mining face of a coal mine, a digital twin three-dimensional virtual mining scene based on a unified geodetic coordinate system is constructed, and equipment-side information is acquired in real time by utilizing an intelligent perception technology; the information drives the twin model in the virtual scene, so that transparent perception of the mining environment of the underground coal mining operation place, intelligent monitoring of equipment, visual self-adaptive mining, fault prediction and the like are realized, coal mining operators are reduced, and the intelligent mining level of the coal mine is improved. The invention patent of publication No. CN111208759A discloses a method for performing perception analysis, simulation, iterative optimization and decision control by utilizing a convolutional network deep learning algorithm based on a three-dimensional visual virtual scene by constructing a digital twin model. Based on data twinning and data driving, the real-time monitoring, intelligent sensing, accurate positioning and health prediction of the unmanned fully-mechanized mining working face of the remote physical space mine are realized through the virtual space digital twinning unmanned fully-mechanized mining working face. The invention patent of publication number CN111210359A discloses a digital twin evolution mechanism and method for an intelligent mine scene, and realizes data image and information interaction between a digital twin and a physical entity by constructing a digital twin model, and object twin, process twin and performance twin of the physical space physical entity and a virtual space digital twin; according to the method, the remote visual monitoring of the intelligent mine scene in the physical space is realized in the intelligent mine scene in the virtual space by a digital twin evolution mechanism and method. The invention patent with publication number of CN112945160A provides a relative pose test platform and a test method between virtual and actual fused hydraulic supports, wherein the virtual and actual fused method is adopted to simulate the working flow and supporting scene of adjacent hydraulic supports in a real coal seam environment, the multi-support virtual test scene simulates the action flow, the real-time relative pose state and the coal seam inclination condition of the adjacent supports in a real well, and the real-time pose picture, the relative pose data and the real-time pressure data of the actual tested hydraulic supports between the hydraulic supports are displayed through the alternate actions of the adjacent virtual hydraulic supports and the actual tested hydraulic supports; the invention patent of publication number CN109989751A discloses a cross-platform remote real-time motion tracking method for fully-mechanized three machines, constructs a fully-mechanized three-machine virtual model driving module, realizes real-time driving of a three-machine virtual model, dynamically displays the real-time running state of fully-mechanized three-machine equipment, and displays the real-time running state on a client computer. The invention patent of publication number CN113128109A discloses a testing and evaluating method for an intelligent fully-mechanized mining robot production system, wherein an AI robot system constructs a deep reinforcement learning model of virtual equipment based on sensing error analysis and error analysis of execution error, determines simulation initial data and virtual scene operation data, inputs equipment and geological detection means according to future intelligent development operation parameters, and performs virtual previewing and iterative optimization according to the input parameters; and constructing comprehensive evaluation indexes considering the cutting track, straightness, working space and dynamic coal seam, simulating the running of the fully mechanized mining robot in the future, determining the development trend and testing the running performance of the robot.
From the above research contents, it can be seen that the establishment of the virtual model and the healthy operation thereof play an important role in realizing the digital twin technology, the construction accuracy of the virtual model has a great influence on the reliability of the scene operation, but the following problems still exist in the current research: (1) The influence of the environment on the operation of equipment in the current realization process of the collaborative propulsion of the virtual fully-mechanized mining face is mainly limited by the additional function of a virtual simulation engine, and the influence of the operation conditions of other equipment on the operation of the equipment is not comprehensively considered; (2) The data driving effect is mainly reflected in the construction process of the model, the effect of the historical exploitation data is supposed to penetrate through the whole virtual operation process of the equipment, namely, key information of the equipment operation is extracted from the historical data, and the key information needs to be combined with the virtual operation of the equipment; (3) In the aspect of application of deep reinforcement learning knowledge on equipment, a global target is limited to engineering problems, and only single equipment operation information and historical exploitation information are utilized in iterative training, so that the influence of interaction and joint action between the equipment is not considered.
In summary, the real-time update of the virtual scene is required in terms of virtual debugging, process design and later service in the operation process of the digital twin fully-mechanized coal mining face, and in the prior art, the virtual reconstruction flexibility of the three-machine operation process of the fully-mechanized coal mining face in the virtual environment is poor, so that the operation and maintenance requirements of the current digital twin fully-mechanized coal mining face are not met, and the three-machine autonomous coordination capability of the virtual fully-mechanized coal mining face is not enough.
Therefore, on the basis of establishing the height mapping of the physical fully-mechanized coal mining face under the virtual environment, a coupling operation mechanism which is based on data driving and gives consideration to the mutual influence between the coal seam environment and the equipment in the presence Jing Yun is required to be established in the operation process of the equipment.
Disclosure of Invention
The invention aims to provide a virtual fully mechanized coal mining production system deduction method based on multi-agent deep reinforcement learning, so as to improve the flexibility of virtual scene operation, consider the interaction and interaction among equipment, equipment and coal layers, ensure that virtual coal machine equipment operates efficiently and stably under the working space constructed by the virtual coal seam and the equipment together, and provide guidance and service for the production operation of an actual fully mechanized coal mining working face.
In order to achieve the above purpose, the invention adopts the following technical scheme: the method comprises the steps of carrying out three-dimensional modeling on a coal machine equipment intelligent body according to respective structural parameters and the characteristics of degrees of freedom of a fully mechanized coal face, and carrying out modeling on a virtual coal seam according to coal seam detection information and coal cutter cutting information; importing the established model into Unity3D as an initial scene of three machines of the whole fully mechanized mining face, accessing historical operation information in the equipment mining process into the scene, and extracting key operation information of coal mill equipment as a data driving source of initial operation;
Carrying out centralized iterative training on equipment operation data, obtaining decision information of equipment operation in a distributed control mode, finally realizing speed regulation, roller height regulation and pushing of the intelligent body of the coal mining machine under the control of decision results, self-adaptively bending and deducting a next cycle exploitation track of the intelligent body of the scraper conveyor, pushing, moving a frame, lifting/lowering a column, extending/retracting a mutual aid plate and correcting frames among the intelligent bodies of the intelligent body of the hydraulic support, and updating a coal seam according to cutting information of the coal mining machine;
the three machines are hydraulic supports, scraper conveyors and coal mining machines, and the built coal machine equipment intelligent bodies are coal mining machine intelligent bodies, hydraulic support intelligent body groups and scraper conveyor intelligent bodies.
Further, in the virtual environment, decision is made on the movement of the intelligent agent of the coal machine equipment, and the three-machine collaborative propulsion process of the fully mechanized coal mining face is realized, and the method comprises the following steps:
(1) Constructing an operation environment by using the Ml-Agents plug-in of the Unity3D, and determining information of each intelligent agent about the intelligent agent and information of other intelligent Agents, namely joint information of the intelligent agent, azimuth information on a virtual coal seam, key operation action information of the intelligent agent, relative azimuth information among other intelligent Agents, position information of the intelligent agent and key operation action information of the intelligent agent;
(2) The coal machine equipment intelligent body selects and executes respective actions according to the current coal seam environment and the relative state information among the equipment, thereby influencing the transfer and update of the environment state, and the process is realized by<S,A 1 ,…,A n ,T,R 1 ,…,R n >Respectively represent a state set (S), an action set (A) i ) Reward set (R) i ) Probability of state transition (T);
(3) Using machine-learned "curiosity options" in Unity3D, a single agent jackpot mechanism is established by combining s t ，a t ，s t+1 The output isThrough s of input t ，a t Predicting the next state +.>And s t+1 The larger the gap is, a t The greater the curiosity for the unknown state, the greater the reward; by training a filter, some characteristics irrelevant to the behavior of the coal machine equipment intelligent agent are filtered, and the filtered behavior state is input into another network to obtain +.>
(4) The expected effect of the whole mining process, namely the expected of realizing the maximum mining rate under the premise of ensuring the safe mining, is recorded as Q (s, a), and is decomposed into weighted sums of local Qi (si, ai), the intelligent bodies of the coal machine equipment all have respective local value functions, and the mining target of the fully-mechanized mining face 'three machines' is decomposed into the operation target of single coal machine equipment;
(5) Integrating operation decision processes of single coal machine equipment, determining the maximum expected value of three-machine operation of a fully mechanized mining face, combining the decision processes of single coal machine equipment intelligent bodies by adopting a hybrid network module QMIX method, wherein the operation action corresponding to the maximized global Qtot value is the combination of all local Qa values;
(6) Establishing a monotonicity constraint relation between a global Q value of a fully mechanized mining face running on a virtual coal seam and a local Q value of single equipment by using a three-machine multi-agent, solving the problem of uncertainty in the mining process under the complex coal seam condition by adopting a centralized learning method under the premise of considering the combined action effect among the multi-agent, extracting the running strategy of single coal machine equipment agent from the mining process, and further realizing distributed control, wherein the constraint relation is shown in the following formula, and n is the number of the equipment agents:
(7) In the training process, adding global information such as the mining amount, mining time, rock retention amount and flatness of the fully-mechanized mining face to carry out auxiliary training, guiding the optimization of a strategy through a joint action Q value, and simultaneously, the individual can extract local Qi values from the global Q value to finish respective decisions so as to realize the distributed control of multiple intelligent agents; selecting a maximized global Qtot value as an iteration update target, and selecting actions of each agent in each iteration;
(8) Obtain the optimal Q tot And the optimal strategy establishes a coupling propulsion operation mechanism of three machines of the fully mechanized coal mining face, and each coal machine equipment intelligent body realizes coupling deduction of the three machines of the fully mechanized coal mining face according to the respective optimal operation strategy and combined with the guidance of the integral operation strategy.
Further, in the step (6), the joint action effect means that in the process that the coal machine equipment intelligent bodies reach the optimal joint action problem, the motions among the intelligent bodies are modeled mutually, and the potential motion information can be obtained from other intelligent bodies to make a decision, so that a foundation is established for the equipment intelligent body cooperation mechanism.
Further, the equipment intelligent agent cooperation mechanism is a cooperation operation mechanism between the coal cutter intelligent agent and the coal seam, between the scraper conveyor intelligent agent and the hydraulic support intelligent agent, between the scraper conveyor intelligent agent and the coal seam, between the scraper conveyor intelligent agent and the virtual coal seam, between the coal cutter intelligent agent and between the hydraulic support intelligent agent groups.
Further, the equipment intelligent agent cooperative mechanism is based on a virtual coal seam exploitation space in a virtual environment created by Unity3D, the scraper conveyor intelligent agent is adaptively paved on a virtual coal seam bottom plate, the coal mining machine intelligent agent takes the scraper conveyor intelligent agent as a track, front and rear rollers are adaptively heightened to cut coal, and the hydraulic support intelligent agent timely advances and supports according to the cutting condition of the coal seam top and bottom plates.
Further, mutual modeling means that one of the coal machine equipment agents models the operation strategy of other agents based on the historical operation information of the other agents in the combined action learning process.
Further, the mutual modeling means that the coal cutter intelligent body performs coal cutting action according to the coal seam environment information and the posture information of the scraper conveyor intelligent body; the scraper conveyor intelligent body performs interaction with the coal seam environment according to cutting process information of the coal mining machine intelligent body, and bending information of the scraper conveyor intelligent body of the next mining cycle is obtained; the hydraulic support agent group carries out self-adaptive support according to the cutting top and bottom plate conditions of the coal cutter agent, and carries out pushing according to the obtained bending information of the scraper conveyor agent of the next mining cycle.
Further, the distributed control means that the local value function of each coal machine equipment intelligent body only needs to be observed locally under the view angle of the intelligent body, and the action execution with the largest accumulated expected reward is selected through the local value function to perform distributed execution, so that the whole virtual reconstruction system is distributed during execution.
Further, the hybrid network module QMIX is responsible for merging local value functions of the single coal machine equipment agents, and in the hybrid network module, weights of all layers are generated by using a super network and absolute value calculation, so that the weights are always in positive and monotonicity constraint relation.
Compared with the prior art, the virtual fully-mechanized mining face production system deduction method based on multi-agent deep reinforcement learning has the following advantages and outstanding innovation points:
1. on the basis of the influence of the existing coal seam conditions on all equipment intelligent bodies, the mutual influence among the equipment intelligent bodies is considered, the total operation environment based on the coal seam mining space and the operation information of other equipment intelligent bodies relative to the single equipment intelligent body is formed, global information of the fully-mechanized mining face mining process is added for auxiliary training, the unstable problem of fluctuation of the coal seam environment and the influence of the combined action effect among the intelligent bodies in the fully-mechanized mining face advancing process are overcome by adopting a centralized training and distributed decision control mode, the optimal actions of all the equipment intelligent bodies and the optimal cooperation strategy among the equipment intelligent bodies are ensured, and the mutual influence and mutual guidance cooperative advancing mechanism among equipment and coal seams in operation is realized.
2. Based on historical mining data, real-time fusion is carried out on coal seam prediction information, cutting height information and pose information of coal mining machine agents, pose information of scraper conveyor agents and supporting information of hydraulic support agents in the Unity3D, a virtual coal seam dynamic updating mechanism is established, key characteristic information is extracted through concentrated iteration training, updating of the coal seam along with the three-machine promotion of a fully mechanized mining face is achieved, updating of coal-rock capacity proportion and limit is included, whether complex geology is encountered in a future mining process or not can be checked in a virtual environment, and decision is made on cutting track information of the coal mining machine in the mining process according to fluctuation conditions of the virtual coal seam.
3. The coal cutter intelligent body makes an action decision according to the virtual coal rock boundary, forms an optimal cutting scheme according to the virtual coal seam condition and feeds back to the actual coal cutter driving cutting process, so that the maximum exploitation benefit is ensured; in the process of cutting coal walls by the coal cutter intelligent body, the effect of group joint action can be considered, and the decision of other equipment intelligent bodies on the optimal behavior can be guided; under the condition that the inclination angle of the coal seam is large and equipment channeling is easy to occur, the coal mining machine intelligent body calculates the environment state transition probability of the scraper conveyor intelligent body and the adjacent hydraulic support intelligent body group, determines whether the state of the equipment intelligent body is abnormal or not, and timely makes a decision and stops the abnormal intelligent body state.
4. In the operation decision process, the hydraulic support intelligent body optimizes the operation of the hydraulic support intelligent body by learning the coal seam environment and the operation state of other equipment intelligent bodies, and in the supporting process, the hydraulic support intelligent body establishes the optimal decision information between the coal seam roof information and the intelligent body supporting posture, so as to analyze the support-surrounding rock coupling relation; when the hydraulic support intelligent body and the scraper conveyor intelligent body cooperatively operate, the behavior decision of the pushing mechanism is influenced by the pose characteristics of the scraper conveyor intelligent body and the fluctuation of the coal seam, and an optimal pushing path is determined; in view of discrete action spaces among the hydraulic support intelligent bodies, interference conditions of the action spaces are analyzed, so that a decision is made on whether the action of biting and extruding the frames occurs among the intelligent bodies, and the cutting action of the coal cutter intelligent bodies is modeled, so that healthy operation of the hydraulic support intelligent body group in the subsequent propelling process is ensured.
5. The intelligent agent of the scraper conveyor can perform strategy integration according to the running states of the intelligent agent of the coal mining machine and the intelligent agent group of the hydraulic support and the fluctuation condition of the coal seam, so as to realize self-adaptive bending; the pose information of the scraper conveyor agent can guide modeling of pushing action of the hydraulic support agent, and the straightness control of the hydraulic support agent group and the walking track of the coal mining machine agent are decided, so that the straightness problem of the whole fully-mechanized mining working face is controlled; according to the gap between the width of the cutting bottom plate and the width of the middle groove, the paving posture of the scraper conveyor intelligent body can invert the fluctuation of the coal seam, the posture of the scraper conveyor intelligent body can influence the updating reliability of the virtual coal seam bottom plate, and the paving posture information of the scraper conveyor intelligent body and the cutting bottom plate information of the coal mining machine intelligent body can be combined and then used as prior information to be provided for the updating process of the coal seam.
6. The method can be applied to the production, design and service processes of the actual fully-mechanized mining face, the formulation and evaluation of the solution to the common engineering problems in the mining process and the identification and analysis to the equipment operation information and fault information in the advancing process of the fully-mechanized mining face, so as to achieve the purposes of running prediction and monitoring for the whole mining process, realizing the active discovery of problems and providing a solution strategy.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention.
FIG. 1 is a frame diagram of a virtual fully-mechanized face production system coupling deduction method based on multi-agent deep reinforcement learning;
FIG. 2 is a schematic illustration of fully mechanized face propulsion;
FIG. 3 is a framework diagram of a virtual reconstruction method of a fully mechanized face production system based on multi-agent deep reinforcement learning;
FIG. 4 is a virtual reconstructed Q network architecture for a fully mechanized face production system;
FIG. 5 is a schematic diagram of a virtual reconstruction decision and control method for a fully mechanized coal mining face production system;
FIG. 6 is a diagram of the functions that can be achieved after the deduction method of the virtual fully mechanized face production system based on multi-agent deep reinforcement learning is applied;
FIG. 7 is a model selection and optimization of equipment for a virtual fully-mechanized face production system deduction method based on multi-agent deep reinforcement learning;
FIG. 8 is a diagram of the functions that can be achieved after the deduction method of the virtual fully mechanized face production system based on multi-agent deep reinforcement learning is applied;
FIG. 9 is a schematic representation of virtual fully mechanized face production system deduction method based on multi-agent deep reinforcement learning for model selection and optimization;
FIG. 10 is a diagram of a scraper conveyor straightness adjustment for a virtual fully mechanized face production system deduction method based on multi-agent deep reinforcement learning;
FIG. 11 is a diagram of a monitoring study of slip-up and slip-down of a scraper conveyor based on a deduction method of a virtual fully-mechanized face production system based on multi-agent deep reinforcement learning.
Detailed Description
The invention adopts multi-agent deep reinforcement learning as the technical support of three-machine cooperation of the fully mechanized mining face, and realizes self-adaptive cooperation propulsion on a virtual coal seam and mutual cooperation propulsion among equipment in the process of three-machine operation propulsion of the fully mechanized mining face. According to the operation characteristics of the coal machine equipment in the propelling process, the coal machine equipment is regarded as the coal machine equipment intelligent body, so that the cooperative propelling process of the three machines of the fully mechanized mining face on the virtual coal seam can be regarded as that under the driving of the historical mining information, under the environment of the virtual mining space, the combined action effect of each equipment intelligent body and other intelligent bodies is considered in the working process, and the mining process is completed in a mutual cooperation mode.
FIG. 1 is a block diagram of a method for deducting a virtual fully-mechanized coal mining face production system for multi-agent deep reinforcement learning. According to the characteristics of the structural parameters and degrees of freedom of three machines (namely, a hydraulic support, a scraper conveyor and a coal mining machine) of the fully mechanized mining face, three-dimensional modeling of an intelligent body is carried out, modeling of a virtual coal seam is carried out according to coal seam detection information and coal mining machine cutting information, the established three-dimensional model is imported into Unity3D to serve as an initial scene of the whole fully mechanized mining face, historical operation information in the equipment mining process is accessed into the scene, and coal extraction equipment operation key information serves as a data driving source of initial operation. The equipment operation data is subjected to centralized iterative training, decision information of equipment operation is obtained in a distributed control mode, finally, under the control of decision results, speed regulation, roller height regulation and pushing of the coal mining machine intelligent body in a virtual environment can be realized, the scraper conveyor intelligent body is adaptively bent and the next cycle mining track is deduced, the hydraulic support intelligent body is pushed, moved, lifted/lowered, extended/retracted, mutually assisted plates and the correction frame among the intelligent bodies are adopted, and coal seam updating is carried out according to cutting information of the coal mining machine.
As shown in fig. 2, which is a schematic diagram of the fully mechanized coal face propulsion, the coal cutter uses the scraper conveyor as a running track to perform coal cutting operation, and after the coal cutter cuts a certain distance, the hydraulic support pushes the scraper conveyor, and the hydraulic support timely supports the top plate to avoid collapse. After the coal cutter finishes cutting one cutter, the coal cutter reversely cuts coal, and the coal cutter circularly reciprocates in the way, so that the propulsion of the fully-mechanized coal mining working face is realized.
In the process of realizing the 'three-machine' collaborative propulsion of the virtual fully-mechanized mining face, the method mainly comprises three major contents: the problems of coal cutter cutting operation, scraper conveyor flatness assurance and hydraulic support group support are solved, but the three are mutually related. The method specifically comprises the following steps: 1. the coal cutter uses the scraper conveyor as a track to cut coal, the straightness of a coal wall needs to be ensured in the coal cutting process, the straightness of the scraper conveyor needs to be ensured in the coal cutting process, and the straightness control needs to be realized by controlling the pushing action of the hydraulic support. 2. The height of the roller of the coal cutter needs to be adjusted in the cutting process so as to ensure the maximum production of the fully mechanized coal face, and the goaf after cutting needs to be supported in time by the hydraulic support, so that the supporting posture of the hydraulic support is also determined. 3. When the hydraulic support pushes the scraper conveyor, the specific pushing distance required by guaranteeing the straight line is determined by considering the motion process of each structure of the pushing mechanism, and then the straightness of the scraper conveyor is controlled.
In view of the above, it is necessary to make decisions on the movement of the coal machine equipment agent in a virtual environment to realize a "three-machine" collaborative propulsion process of the fully mechanized coal mining face.
Fig. 3 is a process diagram of a deduction process of a virtual fully-mechanized coal mining face production system for multi-agent deep reinforcement learning, which specifically includes the following steps:
(1) And constructing an operation environment by using the Ml-Agents plug-in of the Unity3D, and determining information of each agent about the agent and information about other Agents, namely joint information of the agent, azimuth information on a virtual coal seam, key operation action information of the agent, relative azimuth information among other Agents, position information of the other Agents and key operation action information of the agent.
(2) The coal machine equipment intelligent body selects and executes respective actions according to the current coal seam environment and the relative state information among the equipment, thereby influencing the transfer and update of the environment state, and the process is realized by<S,A 1 ,…,A n ,T,R 1 ,…,R n >Respectively represent a state set (S), an action set (A) i ) Reward set (R) i ) State transition probability (T).
(3) Using machine-learned "curiosity options" in Unity3D, a single agent jackpot mechanism as shown in FIG. 4 is established by combining s t ，a t ，s t+1 The output isThrough s of input t ，a t Predicting the next state +.>And s t+1 The larger the gap is, a t The greater the curiosity for the unknown state, the greater the reward; by training a filter, some characteristics irrelevant to the behavior of the coal machine equipment intelligent agent are filtered, the filtered behavior state is input into another network, and a Feature extractor Feature Ext is better optimized to obtain +.>The key to the above process is the curiosity module as shown in fig. 5. By adding forward_loss and reverse_loss to the script and using Curiositivity function, network1 and Network2 in the figure are respectively represented by a create_forward_model and a create_reverse_model in the script.
(4) As shown in fig. 6, the expected effect of the whole mining process, namely, the desire of realizing the maximum mining rate on the premise of ensuring the safe mining is recorded as Q (s, a), and is decomposed into weighted sums of local Qi (si, ai), the coal machine equipment intelligent bodies all have respective local value functions, and the mining target of the fully-mechanized mining face 'three machines' is decomposed into the operation target of single coal machine equipment.
(5) The method comprises the steps of integrating operation decision-making processes of single coal machine equipment, determining the maximum expected value of three-machine operation of a fully mechanized coal mining face, combining the decision-making processes of single coal machine equipment intelligent agents by adopting a QMIX (hybrid network module) method, and enabling operation actions corresponding to the maximized global Qtot value to be combinations of local Qa values.
(6) As shown in fig. 7, a monotonicity constraint relation between a global Q value of a fully-mechanized mining face running on a virtual coal seam and a local Q value of a single device is established, a centralized learning method is adopted to solve the problem of uncertainty in the mining process under the complex coal seam condition under the premise of considering the combined action effect among the multiple agents, a running strategy of a single coal machine equipment intelligent agent is extracted from the problem, and further distributed control is realized, wherein the constraint relation is shown in the following formula, and n is the number of the equipment intelligent agents:
(7) In the training process, the global information of the mining amount, the mining time, the rock remaining amount and the flatness of the fully-mechanized mining face is added for auxiliary training, the strategy is guided to be optimized through the joint action Q value, and meanwhile, the individual can extract local Qi values from the global Q value to finish respective decisions, so that the distributed control of multiple intelligent agents is realized. The maximized global Qtot value is selected as the target for iterative updating, with the actions of each agent being selected in each iteration.
(8) Obtain the optimal Q tot And the optimal strategy establishes a coupling propulsion operation mechanism of three machines of the fully mechanized coal mining face, and each coal machine equipment intelligent body realizes coupling deduction of the three machines of the fully mechanized coal mining face according to the respective optimal operation strategy and combined with the guidance of the integral operation strategy.
As shown in fig. 8, the function of the fully mechanized coal mining face three-machine deduction method based on multi-agent deep reinforcement learning after application can be realized, after the high-reliability collaborative operation process of the fully mechanized coal mining face three-machine is realized, the dynamic coupling of the fully mechanized coal mining face three-machine equipment and a working space can be realized, the analysis and the deduction of the whole process of the operation and monitoring of the whole production system are performed, the dynamic characteristics of the coal machine equipment during the operation and the association relation between the equipment and a coal seam are revealed, the identification and the prediction of the health condition of the whole machine of the coal machine equipment during the mining process are realized, a visual analysis platform and a health monitoring platform are provided for coal mining, and a technical platform support is provided for the high-efficiency operation of the fully mechanized coal mining face production system.
After the three-machine collaborative propulsion of the fully mechanized mining face is realized, when the virtual debugging process is realized, the digital equipment is controlled to run in real time in the digital underground environment by means of connecting a real control program, manual button operation and the like. The digital underground environment design needs to comprehensively consider complex geology, equipment faults and common engineering problems, and carry out problem analysis to determine a reasonable scheme. The real description of the problems possibly occurring in the exploitation process in the joint debugging process is realized, so that the purpose of realizing the actual underground debugging in the virtual debugging of the digital space is achieved. Some of the key parameters are set as follows:
(1) The problem of interruption of information transmission is set through program interaction in the Unity3D, and downhole related data are analyzed to obtain an operation rule;
(2) In consideration of sensor failure and abnormal problems, the problems are effectively processed by establishing an early warning mechanism and performing relevant compensation in the Unity3D interaction process.
(3) The equipment is set through relevant data of similar mining areas in a combined debugging mode under complex working conditions such as real arrangement and the like, and the equipment performance and the sensing control elements of the equipment are calibrated.
After the deduction process of 'three machines' high reliability of the fully mechanized coal mining face is realized, corresponding service can be provided for the coal mining process. And transmitting the equipment operation information in the real-time exploitation process to the virtual scene, identifying the equipment operation state and sensing the space fault in the exploitation process by utilizing the visualized virtual scene, and providing a solution.
Wherein, coal machine equipment intelligent agent refers to coal-winning machine intelligent agent, hydraulic support intelligent agent crowd, scraper conveyor intelligent agent.
The coal mining machine intelligent body can be used for adaptively heightening front and rear drums of the coal mining machine according to the fluctuation condition of the coal seam and adaptively walking along the scraper conveyor, so that the whole virtual coal mining process is more efficient, and the actual coal mining process is highly mapped.
The hydraulic support intelligent body group consists of single hydraulic support intelligent bodies, the single hydraulic support intelligent bodies can realize pushing, moving, lifting, lowering and extending/retracting the mutual aid plates, the hydraulic support intelligent bodies can adaptively adjust and correct the frames, and abnormal postures among the hydraulic supports are avoided.
The scraper conveyor intelligent body is capable of being adaptively paved on a virtual coal seam bottom plate in a virtual environment, and can be adaptively bent and propelled on the basis of cooperative action with a hydraulic support group along with the movement of the coal mining machine.
The joint action effect means that the motion among the intelligent agents of the coal machine equipment can be modeled mutually in the process of achieving the optimal joint action problem, and can obtain potential motion information from other intelligent agents to make decisions, thereby laying a foundation for the establishment of a cooperation mechanism of the equipment intelligent agents.
The equipment intelligent agent cooperation mechanism is a cooperation operation mechanism between the coal cutter intelligent agent and the coal seam, between the scraper conveyor intelligent agent and the hydraulic support intelligent agent, between the scraper conveyor intelligent agent and the coal seam, between the scraper conveyor intelligent agent and the virtual coal seam, between the coal cutter intelligent agent and between the hydraulic support intelligent agent groups. Specifically, under the virtual environment created by Unity3D, based on the virtual coal seam exploitation space, the scraper conveyor intelligent body is adaptively paved on the virtual coal seam bottom plate, the coal mining machine intelligent body takes the scraper conveyor intelligent body as a track, the front roller and the rear roller are adaptively heightened to cut coal, and the hydraulic support intelligent body is timely pushed and supported according to the cutting condition of the coal seam top and bottom plates.
The mutual modeling means that one of the coal machine equipment intelligent agents models the operation strategies of other intelligent agents based on the historical operation information of the other intelligent agents in the combined action learning process. Specifically, the coal cutter intelligent body performs coal cutting action according to the coal seam environment information and the posture information of the scraper conveyor intelligent body; the scraper conveyor intelligent body performs interaction with the coal seam environment according to cutting process information of the coal mining machine intelligent body, and bending information of the scraper conveyor intelligent body of the next mining cycle is obtained; the hydraulic support agent group carries out self-adaptive support based on the cutting top and bottom plate conditions of the coal mining machine agent, and the pushing is carried out according to the obtained bending information of the scraper conveyor agent of the next mining cycle.
The distributed control means that each coal machine equipment intelligent body only needs to carry out local observation under the visual angle of the intelligent body, and the action execution with the largest accumulated expected reward is selected through the local value function to carry out distributed execution, so that the whole virtual reconstruction system is distributed during execution.
The mixed network module is responsible for merging local value functions of single coal machine equipment intelligent agents, and in the module, weights of all layers are generated by utilizing a super network and absolute value calculation, so that the weights are always in positive and monotonicity constraint relation.
As shown in fig. 9, taking an equipment model selection stage in the early production stage of the fully mechanized coal mining face as an example, the model selection and optimization are performed by using the established multi-agent deep reinforcement learning fully mechanized coal mining face 'three-machine' deduction method. In the process of selecting the products of the three-machine equipment on the fully mechanized mining face, the typical multi-objective decision-making phenomenon exists because each equipment is more constrained by conditions during the type selection. Selecting the model of a coal mining machine according to the mining height of a coal seam, the mining transportation requirement of a coal mine and the reliability requirement of fully-mechanized coal face equipment, selecting the model of a scraper conveyor according to the maximum cutting amount of the coal mining machine, the width of a roller of the coal mining machine and the length of the fully-mechanized coal face, and selecting the model of a hydraulic support according to the mining height and the fluctuation condition of the coal seam and the requirements of the coal mining machine and the scraper conveyor, thereby preliminarily selecting the model of matched equipment: the hydraulic support is ZY11000/18/38D, the coal mining machine is MG400/920-WD, and the scraper conveyor is SGZ800/2x525. After modeling, the modeling method is applied in a virtual environment, a deduction method of a virtual fully-mechanized mining face production system based on multi-agent deep reinforcement learning is established, the operation condition of the matched equipment is analyzed, and the matched scheme is optimized. The original and optimized protocols obtained are shown in the following table.
Fig. 10 shows a targeted study of the adjustment of the straightness of the scraper conveyor in a virtual environment during fully mechanized mining face operation. The method comprises the steps of intensively training historical pose information of an actual hydraulic support group and data sources of a straightening process based on inversion of coal cutter track information in a virtual environment to obtain track information of a scraper conveyor, obtaining motion decision results of each degree of freedom of a single hydraulic support intelligent pushing mechanism, and predicting obtained track information of the scraper conveyor and decision position information of pushing points of the hydraulic support under a multi-factor condition, wherein the pushing mechanism always keeps connection with the scraper conveyor and moves according to the motion decision results of each degree of freedom according to linkage effects between the hydraulic support intelligent and the scraper conveyor intelligent, so that virtual connection between the hydraulic support intelligent group and the scraper conveyor is completed; when the scraper conveyor intelligent body advances, the hydraulic support intelligent body is pushed in time under the action of decision information, so that the scraper conveyor is straightened under the virtual environment.
Fig. 11 is a schematic view showing virtual monitoring of the upward and downward slip of the scraper conveyor. Based on coal seam detection information and comprehensive mining working face three-machine equipment historical operation information, a multi-agent operation environment is constructed, a virtual comprehensive mining working face three-machine cooperative operation method is established by using the knowledge in fig. 3, the position of a scraper conveyor in the pushing process is monitored virtually, whether upward channeling and downward sliding occur is judged according to the movement quantity of a scraper conveyor head, and early warning is carried out on whether upward channeling and downward sliding occurs in the real comprehensive mining working face pushing process. From two angles of coal seam and equipment, considering the fluctuation condition of a coal seam bottom plate, the inclination angle of the coal seam, the friction factor between the equipment and the bottom plate, the acting force of a coal cutter on a scraper conveyor and the influence of virtual running resistance of each middle groove on the upward-shifting and downward-shifting of the scraper conveyor, the influence of the inclination angle of the A-coal seam, the B-friction factor and the rotating speed of a drum of the C-coal cutter on the upward-shifting and downward-shifting of the scraper conveyor is analyzed on the basis of realizing virtual monitoring, and the analysis results are shown in the following two tables.
Test protocol and results analysis table
Analysis of variance table
From the comparison of the F value and the critical value, factor a is a significant factor. The main and secondary orders of the influence of each factor on the test result are A, C, B, namely the coal seam inclination angle, the drum speed of the coal mining machine and the friction factor.
The scope of the present invention is not limited to the above embodiments, but various modifications and alterations of the present invention will become apparent to those skilled in the art, and any modifications, improvements and equivalents within the spirit and principle of the present invention are intended to be included in the scope of the present invention.
Claims (8)
1. A virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning is characterized by comprising the following steps of:
carrying out three-dimensional modeling on the intelligent body of the coal machine equipment according to the respective structural parameters and the characteristics of the degree of freedom of the three machines of the fully mechanized coal face, and carrying out modeling on the virtual coal bed according to the coal bed detection information and the coal cutter cutting information; importing the established model into Unity3D as an initial scene of three machines of the whole fully mechanized mining face, accessing historical operation information in the equipment mining process into the scene, and extracting key operation information of coal mill equipment as a data driving source of initial operation;
Carrying out centralized iterative training on equipment operation data, obtaining decision information of equipment operation in a distributed control mode, finally realizing speed regulation, roller height regulation and pushing of the intelligent body of the coal mining machine under the control of decision results, self-adaptively bending and deducting a next cycle exploitation track of the intelligent body of the scraper conveyor, pushing, moving a frame, lifting/lowering a column, extending/retracting a mutual aid plate and correcting frames among the intelligent bodies of the intelligent body of the hydraulic support, and updating a coal seam according to cutting information of the coal mining machine;
the three machines are hydraulic supports, scraper conveyors and coal mining machines, and the built coal machine equipment intelligent bodies are coal mining machine intelligent bodies, hydraulic support intelligent body groups and scraper conveyor intelligent bodies;
under a virtual environment, decision is made on the movement of the intelligent agent of the coal machine equipment, and the three-machine collaborative propulsion process of the fully mechanized coal mining face is realized, and the method comprises the following steps:
(1) Constructing an operation environment by using the Ml-Agents plug-in of the Unity3D, and determining information of each intelligent agent about the intelligent agent and information of other intelligent Agents, namely joint information of the intelligent agent, azimuth information on a virtual coal seam, key operation action information of the intelligent agent, relative azimuth information among other intelligent Agents, position information of the intelligent agent and key operation action information of the intelligent agent;
(2) The coal machine equipment intelligent body selects and executes respective actions according to the current coal seam environment and the relative state information among the equipment, thereby influencing the transfer and update of the environment state, and the process is realized by<S,A 1 ,…,A n ,T,R 1 ,…,R n >Respectively represent a state set (S), an action set (A) i ) Reward set (R) i ) Probability of state transition (T);
(3) Using machine-learned "curiosity options" in Unity3D, a single agent jackpot mechanism is established by combining s t ，a t ，s t+1 The output isThrough s of input t ，a t Predicting the next state +.>And s t+1 The larger the gap is, a t The greater the curiosity for the unknown state, the greater the reward; by training a filter, some characteristics irrelevant to the behavior of the coal machine equipment intelligent agent are filtered, and the filtered behavior state is input into another network to obtain +.>
(4) The expected effect of the whole mining process, namely the expected of realizing the maximum mining rate under the premise of ensuring the safe mining, is recorded as Q (s, a), and is decomposed into weighted sums of local Qi (si, ai), the intelligent bodies of the coal machine equipment all have respective local value functions, and the mining target of the fully-mechanized mining face 'three machines' is decomposed into the operation target of single coal machine equipment;
(5) Integrating operation decision processes of single coal machine equipment, determining the maximum expected value of three-machine operation of a fully mechanized mining face, combining the decision processes of single coal machine equipment intelligent bodies by adopting a hybrid network module QMIX method, wherein the operation action corresponding to the maximized global Qtot value is the combination of all local Qa values;
(6) Establishing a monotonicity constraint relation between a global Q value of a fully mechanized mining face running on a virtual coal seam and a local Q value of single equipment by using a three-machine multi-agent, solving the problem of uncertainty in the mining process under the complex coal seam condition by adopting a centralized learning method under the premise of considering the combined action effect among the multi-agent, extracting the running strategy of single coal machine equipment agent from the mining process, and further realizing distributed control, wherein the constraint relation is shown in the following formula, and n is the number of the equipment agents:
(7) In the training process, adding global information such as the mining amount, mining time, rock retention amount and flatness of the fully-mechanized mining face to carry out auxiliary training, guiding the optimization of a strategy through a joint action Q value, and simultaneously, the individual can extract local Qi values from the global Q value to finish respective decisions so as to realize the distributed control of multiple intelligent agents; selecting a maximized global Qtot value as an iteration update target, and selecting actions of each agent in each iteration;
(8) Obtain the optimal Q tot And the optimal strategy establishes a coupling propulsion operation mechanism of three machines of the fully mechanized coal mining face, and each coal machine equipment intelligent body realizes coupling deduction of the three machines of the fully mechanized coal mining face according to the respective optimal operation strategy and combined with the guidance of the integral operation strategy.
2. The virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning of claim 1, wherein the method comprises the following steps: in the step (6), the joint action effect means that in the process that the coal machine equipment intelligent bodies reach the optimal joint action problem, the motions among the intelligent bodies are mutually modeled, and the potential motion information can be obtained from other intelligent bodies to make decisions, so that a foundation is established for the equipment intelligent body cooperation mechanism.
3. The virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning according to claim 2, wherein the method is characterized in that: the equipment intelligent cooperative operation mechanism is a cooperative operation mechanism between the coal cutter intelligent body and the coal seam, between the scraper conveyor intelligent body and the hydraulic support intelligent body, between the scraper conveyor intelligent body and the coal seam, between the scraper conveyor intelligent body and the virtual coal seam, between the coal cutter intelligent body and between the hydraulic support intelligent body groups.
4. The virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning according to claim 3, wherein the method comprises the following steps: the equipment intelligent cooperative mechanism is based on a virtual coal seam exploitation space under a virtual environment created by Unity3D, a scraper conveyor intelligent body is adaptively paved on a virtual coal seam bottom plate, a coal mining machine intelligent body takes the scraper conveyor intelligent body as a track, front and rear rollers are adaptively heightened to cut coal, and a hydraulic support intelligent body timely advances and supports according to cutting conditions of a coal seam top and bottom plate.
5. The virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning according to claim 2, wherein the method is characterized in that: the mutual modeling means that one of the coal machine equipment intelligent agents models the operation strategy of other intelligent agents based on the historical operation information of the other intelligent agents in the combined action learning process.
6. The virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning of claim 5, wherein the method comprises the following steps: the mutual modeling means that the coal cutter intelligent body performs coal cutting action according to the coal seam environment information and the posture information of the scraper conveyor intelligent body; the scraper conveyor intelligent body performs interaction with the coal seam environment according to cutting process information of the coal mining machine intelligent body, and bending information of the scraper conveyor intelligent body of the next mining cycle is obtained; the hydraulic support agent group carries out self-adaptive support according to the cutting top and bottom plate conditions of the coal cutter agent, and carries out pushing according to the obtained bending information of the scraper conveyor agent of the next mining cycle.
7. The virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning according to claim 1, 2, 4 or 6, wherein the method comprises the following steps: the distributed control means that the local value function of each coal machine equipment intelligent body only needs to be observed locally under the visual angle of the intelligent body, and the action execution with the largest accumulated expected reward is selected through the local value function to be performed in a distributed mode, and the whole virtual reconstruction system is distributed in the execution.
8. The virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning of claim 7, wherein the method comprises the following steps: the mixed network module QMIX is responsible for merging local value functions of single coal machine equipment intelligent agents, and in the module, weights of all layers are generated by utilizing a super network and absolute value calculation, so that the weights are always in positive and monotonicity constraint relation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111577141.6A CN114329936B (en) | 2021-12-22 | 2021-12-22 | Virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111577141.6A CN114329936B (en) | 2021-12-22 | 2021-12-22 | Virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114329936A CN114329936A (en) | 2022-04-12 |
| CN114329936B true CN114329936B (en) | 2024-03-29 |
Family
ID=81055291
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111577141.6A Active CN114329936B (en) | 2021-12-22 | 2021-12-22 | Virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114329936B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115017677B (en) * | 2022-04-27 | 2023-07-25 | 中国人民解放军军事科学院战略评估咨询中心 | Action strategy pre-judging method and system for deduction simulation |
| CN116011991B (en) * | 2022-12-30 | 2023-12-19 | 中国电子科技集团公司第三十八研究所 | Multi-user collaborative task guaranteeing method based on agent and backup technology |
| CN116151043B (en) * | 2023-04-20 | 2023-07-21 | 西安华创马科智能控制***有限公司 | Pose inversion method and device for scraper conveyor |
| CN116291659B (en) * | 2023-05-24 | 2023-08-08 | 太原理工大学 | Hydraulic support man-machine cooperative control strategy recommendation method |
| CN117348500B (en) * | 2023-12-04 | 2024-02-02 | 济南华科电气设备有限公司 | Automatic control method and system for fully-mechanized coal mining face |
| CN118226790A (en) * | 2024-05-22 | 2024-06-21 | 山东济矿鲁能煤电股份有限公司阳城煤矿 | Intelligent coal mining equipment cooperative control system and method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102508995A (en) * | 2011-09-26 | 2012-06-20 | 河南理工大学 | Coal mine accident simulating method and system based on multi-intelligent agent |
| CN104317637A (en) * | 2014-10-16 | 2015-01-28 | 安徽理工大学 | Multi-agent-based virtual miner safety behavior modeling and emergency simulation system |
| WO2020000399A1 (en) * | 2018-06-29 | 2020-01-02 | 东莞理工学院 | Multi-agent deep reinforcement learning proxy method based on intelligent grid |
| CN113128109A (en) * | 2021-04-08 | 2021-07-16 | 太原理工大学 | Test and evaluation method for intelligent fully-mechanized mining robot production system |
| WO2021184614A1 (en) * | 2020-03-14 | 2021-09-23 | 天地科技股份有限公司 | Intelligent decision control method and system for fully-mechanized mining equipment used for working surface under complex condition |
-
2021
- 2021-12-22 CN CN202111577141.6A patent/CN114329936B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102508995A (en) * | 2011-09-26 | 2012-06-20 | 河南理工大学 | Coal mine accident simulating method and system based on multi-intelligent agent |
| CN104317637A (en) * | 2014-10-16 | 2015-01-28 | 安徽理工大学 | Multi-agent-based virtual miner safety behavior modeling and emergency simulation system |
| WO2020000399A1 (en) * | 2018-06-29 | 2020-01-02 | 东莞理工学院 | Multi-agent deep reinforcement learning proxy method based on intelligent grid |
| WO2021184614A1 (en) * | 2020-03-14 | 2021-09-23 | 天地科技股份有限公司 | Intelligent decision control method and system for fully-mechanized mining equipment used for working surface under complex condition |
| CN113128109A (en) * | 2021-04-08 | 2021-07-16 | 太原理工大学 | Test and evaluation method for intelligent fully-mechanized mining robot production system |
Non-Patent Citations (2)
| Title |
|---|
| 多智能体深度强化学习的若干关键科学问题;孙长银;穆朝絮;;自动化学报;20200715(07);全文 * |
| 工业互联网驱动的透明综采工作面运行***及关键技术;谢嘉成;王学文;郝尚清;李娟莉;葛星;史恒波;;计算机集成制造***;20191215(12);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114329936A (en) | 2022-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114329936B (en) | Virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning | |
| WO2021184614A1 (en) | Intelligent decision control method and system for fully-mechanized mining equipment used for working surface under complex condition | |
| CN114708393B (en) | Mine stress field twinning modeling assimilation system and method in full-time air mining process | |
| CN109630154B (en) | Tunneling robot for tunneling and remote mobile terminal command system | |
| Wang et al. | The recent technological development of intelligent mining in China | |
| LU501938B1 (en) | Method and system for intelligent analysis of big data on unmanned mining in mine | |
| CN109214076B (en) | Virtual planning method for supporting geographical environment and equipment of fully mechanized coal mining face | |
| Li et al. | Application and outlook of information and intelligence technology for safe and efficient TBM construction | |
| CN112883559B (en) | Planning cutting method and device based on big data system, storage medium and electronic device | |
| Ge et al. | A virtual adjustment method and experimental study of the support attitude of hydraulic support groups in propulsion state | |
| Shi et al. | An operation optimization method of a fully mechanized coal mining face based on semi-physical virtual simulation | |
| CN113128109B (en) | Test and evaluation method for intelligent fully-mechanized mining robot production system | |
| Maruvanchery et al. | Early construction cost and time risk assessment and evaluation of large-scale underground cavern construction projects in Singapore | |
| CN111140231A (en) | Coal seam roof and floor path virtual planning method oriented to space-time kinematics of fully-mechanized mining equipment | |
| Xie et al. | A virtual test and evaluation method for fully mechanized mining production system with different smart levels | |
| CN115455727A (en) | Digital twin-driven fully-mechanized mining equipment and coal bed coupling system optimization decision method | |
| Li et al. | Integrated parameter optimization approach: Just-in-time (JIT) operational control strategy for TBM tunnelling | |
| CN115773127A (en) | Intelligent decision-making method, system, equipment and medium for slurry balance shield | |
| CN109783962B (en) | Fully-mechanized coal mining equipment collaborative propulsion simulation method based on virtual reality physical engine | |
| CN116415479A (en) | Continuous oil pipe fatigue life early warning method based on digital twinning | |
| Fu et al. | Online multi-objective optimization for real-time TBM attitude control with spatio-temporal deep learning model | |
| JP7492482B2 (en) | Shield tunneling machine tunneling prediction model | |
| CN113051756B (en) | Virtual-real fusion coal mining machine memory cutting test system and method | |
| CN116091602A (en) | Digital twin interaction method and system for tunneling robot | |
| CN118192410A (en) | Coal cutter bottom plate cutting track planning method with equipment continuous propulsion as target |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |