CN114329936A - Virtual fully mechanized mining production system deduction method based on multi-agent deep reinforcement learning - Google Patents
Virtual fully mechanized mining production system deduction method based on multi-agent deep reinforcement learning Download PDFInfo
- Publication number
- CN114329936A CN114329936A CN202111577141.6A CN202111577141A CN114329936A CN 114329936 A CN114329936 A CN 114329936A CN 202111577141 A CN202111577141 A CN 202111577141A CN 114329936 A CN114329936 A CN 114329936A
- Authority
- CN
- China
- Prior art keywords
- coal
- mining
- equipment
- intelligent
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000005065 mining Methods 0.000 title claims abstract description 242
- 238000000034 method Methods 0.000 title claims abstract description 149
- 238000004519 manufacturing process Methods 0.000 title claims abstract description 43
- 230000002787 reinforcement Effects 0.000 title claims abstract description 31
- 239000003245 coal Substances 0.000 claims abstract description 285
- 239000003795 chemical substances by application Substances 0.000 claims abstract description 136
- 230000008569 process Effects 0.000 claims abstract description 93
- 238000005520 cutting process Methods 0.000 claims abstract description 37
- 238000012549 training Methods 0.000 claims abstract description 18
- 230000008878 coupling Effects 0.000 claims abstract description 12
- 238000010168 coupling process Methods 0.000 claims abstract description 12
- 238000005859 coupling reaction Methods 0.000 claims abstract description 12
- 238000001514 detection method Methods 0.000 claims abstract description 6
- 230000009471 action Effects 0.000 claims description 45
- 230000007246 mechanism Effects 0.000 claims description 29
- 230000033001 locomotion Effects 0.000 claims description 24
- 230000006870 function Effects 0.000 claims description 16
- 238000005452 bending Methods 0.000 claims description 11
- 230000000694 effects Effects 0.000 claims description 11
- 230000006399 behavior Effects 0.000 claims description 10
- 230000009916 joint effect Effects 0.000 claims description 9
- 238000013486 operation strategy Methods 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 8
- 239000011435 rock Substances 0.000 claims description 7
- 230000007704 transition Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000001186 cumulative effect Effects 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims description 2
- 238000004458 analytical method Methods 0.000 description 8
- 238000012544 monitoring process Methods 0.000 description 7
- 230000036544 posture Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000003993 interaction Effects 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 238000004088 simulation Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000000875 corresponding effect Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000005553 drilling Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000036541 health Effects 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000001276 controlling effect Effects 0.000 description 2
- 230000002079 cooperative effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 208000028752 abnormal posture Diseases 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000002730 additional effect Effects 0.000 description 1
- 238000000540 analysis of variance Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000003631 expected effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000010998 test method Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to a virtual fully-mechanized coal mining production system deduction method based on multi-agent deep reinforcement learning, which is characterized in that three-dimensional modeling of a coal machine equipment agent body is carried out according to respective structural parameters and freedom characteristics of three machines on a fully-mechanized coal mining working face, and virtual coal layer modeling is carried out according to coal layer detection information and coal cutter cutting information; the established model is led into Unity3D to serve as an initial scene of three machines of the whole fully mechanized coal mining face, historical operation information in the equipment mining process is accessed into the scene, and key operation information of the coal equipment is extracted to serve as a data driving source of initial operation; and performing centralized iterative training on equipment operation data, obtaining decision information of equipment operation by adopting a distributed control mode, and finally realizing three-machine cooperative propulsion of the fully mechanized coal mining face and dynamic coupling with a working space under the control of a decision result, thereby providing a technical platform support for the efficient operation of a fully mechanized coal mining face production system.
Description
Technical Field
The invention relates to the technical field of fully mechanized coal mining face simulation, in particular to a virtual fully mechanized coal mining face production system deduction method based on multi-agent deep reinforcement learning.
Background
With the advance of national intelligent manufacturing, the position of a digital twin technology in the manufacturing process is gradually increased, a more efficient, safer and more transparent production manufacturing mode is pursued, the realization of the above contents cannot leave the mapping effect of a virtual scene, and a virtual scene which is completely equivalent to a physical production manufacturing scene needs to be established. Coal mining belongs to a deep operation space, and the development of coal mining intelligence is slower than that of other industries due to the mining particularity, so that a virtual mining process of complete mapping of a coal mining process needs to be established urgently.
The mining operation of the fully mechanized coal mining face is used as an important ring in the coal mining process, the mining process involves more equipment, the requirement on the cooperation among the equipment in the mining process is high, and the requirement on the process and the straightness in the mining process is also high, so that the challenge on realizing the cooperative propulsion of the full life cycle of the fully mechanized coal mining face is large.
The invention patent of publication number CN111140231A discloses a coal seam top and bottom plate path virtual planning method facing to space-time kinematics of fully mechanized mining equipment, which establishes an inherent coal seam top and bottom plate through Unity3D software; the method comprises the steps of constructing a space-time kinematic relationship between fully mechanized mining equipment and a coal seam top and bottom plate by using a physical engine, dynamically generating a single-cycle coal seam top and bottom plate by using a mesh component, and cooperatively propelling a scraper conveyor and a hydraulic support along with the leading of a coal mining machine. The invention patent of publication No. CN109783962A discloses a virtual reality physical engine-based fully-mechanized coal mining equipment cooperative propulsion simulation method, which is characterized in that virtual fully-mechanized coal mining equipment is subjected to model rigid body repair and then is in virtual contact with a virtual coal seam, so as to simulate equipment underground operation information, virtual real-time coal seam updating is realized by carrying out Mesh grid collision body reconstruction in Unity3D software through recording cutting tracks of front and rear rollers of a coal mining machine in real time, and virtual coal seam data information is updated in real time through controlling existence and display of inherent coal seam information and virtual real-time updating coal seam attributes, so that the equipment self-adaptive propulsion process in an underground coal seam environment is truly reproduced. The invention patent of publication number CN108643884A discloses a jumbolter propelling rotary system and a cooperative adaptive control method thereof, which implement active disturbance rejection control on a single loop of the propelling system and the rotary system to enhance the robustness of a drilling process; and determining the optimal propelling force and the optimal rotating speed of the drilling machine in the current drilling process according to the estimated rock hardness coefficient by adopting a composite control mode, so as to realize the cooperative action of the anchor rod rotation and the propelling system. The invention patent of publication number CN112392485A discloses a transparent digital twin self-adaptive mining system and method for a fully mechanized coal mining face, which constructs a digital twin three-dimensional virtual mining scene based on a unified geodetic coordinate system and utilizes an intelligent sensing technology to acquire equipment end information in real time; by driving the twin model in the virtual scene through information, transparent sensing of the mining environment of the underground coal mining operation site, intelligent monitoring of equipment, visual self-adaptive mining, failure prediction and the like are achieved, coal mining operation personnel are reduced, and the intelligent mining level of a coal mine is improved. The invention patent of publication number CN111208759A discloses a method for performing perception analysis, simulation, iterative optimization and decision control by using a convolutional network deep learning algorithm based on a three-dimensional visual virtual scene through constructing a digital twin model. Based on data twinning and data driving, real-time monitoring, intelligent sensing, accurate positioning and health prediction of the remote physical space mine unmanned fully-mechanized working face are achieved through the virtual space digital twin unmanned fully-mechanized working face. The invention patent of publication number CN111210359A discloses a digital twinning evolution mechanism and method facing to an intelligent mine scene, and realizes data mirror image and information interaction between a digital twinning body and a physical entity by constructing a digital twinning model, and realizes object twinning, process twinning and performance twinning of the physical space physical entity and a virtual space digital twinning body; according to the invention, through a digital twin evolution mechanism and method, the physical space intelligent mine scene is remotely and visually monitored in the virtual space intelligent mine scene. The invention patent with the publication number of CN112945160A provides a virtual-real fused relative pose test platform and a test method between hydraulic supports, which adopts a virtual-real fused method to simulate the working flow and supporting scene of adjacent hydraulic supports under a real coal bed environment, a plurality of support virtual test scenes simulate the action flow, real-time relative pose state and coal bed inclination condition of the adjacent supports under a real well, and the real-time pose picture, relative pose data and real-time pressure data of the actual tested hydraulic supports are displayed through the alternate actions of the adjacent virtual hydraulic supports and the actual tested hydraulic supports; the patent of the invention with the publication number of CN109989751A discloses a cross-platform remote real-time motion tracking method for three fully mechanized coal mining machines, which constructs a driving module of a virtual model of the three fully mechanized coal mining machines, realizes the real-time driving of the virtual model of the three fully mechanized coal mining machines, dynamically displays the real-time running state of equipment of the three fully mechanized coal mining machines, and displays the real-time running state on a client computer. The invention patent of publication number CN113128109A discloses a testing and evaluating method for an intelligent fully-mechanized mining robot production system, an AI robot system constructs a deep reinforcement learning model of virtual equipment based on sensing error analysis and error analysis of execution errors, determines simulation initial data and virtual scene operation data, inputs equipment and geological detection means according to future intelligent development operation parameters, and performs virtual rehearsal and iterative optimization according to the input parameters; and establishing comprehensive evaluation indexes considering cutting track, straightness, working space and dynamic coal seam, simulating the operation of the fully mechanized mining robot in the future, determining the development trend and testing the operation performance of the robot.
The research contents show that the establishment of the virtual model and the healthy operation thereof play an important role in the realization of the digital twin technology, and the construction precision of the virtual model has a great influence on the reliability of the scene operation, but the following problems still exist in the current research: (1) the influence of the environment on the operation of the equipment in the implementation process of the cooperative propulsion of the virtual fully mechanized coal mining face is mainly limited to the additional action of a virtual simulation engine, and the influence of the operation condition of other equipment on the operation of the equipment is not considered comprehensively; (2) the influence of data driving is mainly embodied in the process of constructing the model, the influence of historical mining data is required to run through the whole virtual operation process of the equipment, namely, key information of equipment operation is extracted from the historical data, and the information is required to be combined with the virtual operation of the equipment; (3) in the aspect of application of deep reinforcement learning knowledge to equipment, a global target is limited to engineering problems, only single equipment operation information and historical mining information are utilized during iterative training, and the influence of interaction and joint action among the equipment is not considered.
In summary, for the aspect of virtual debugging, process design and later-stage service in the operation process of the digital twin fully-mechanized coal mining face, which requires real-time updating of a virtual scene, in the prior art, the flexibility of virtual reconstruction in the operation process of the three machines of the fully-mechanized coal mining face in a virtual environment is poor, the operation and maintenance requirements of the current digital twin fully-mechanized coal mining face cannot be met, and the autonomous coordination capability of the three machines of the virtual fully-mechanized coal mining face is deficient.
Therefore, on the basis of establishing the height mapping of the physical fully mechanized mining face in the virtual environment, a data-driven coupling operation mechanism which gives consideration to the mutual influence between the coal bed environment and the equipment during scene operation needs to be established in the equipment operation process.
Disclosure of Invention
The invention aims to provide a virtual fully-mechanized coal mining production system deduction method based on multi-agent deep reinforcement learning, so as to improve the flexibility of virtual scene operation, consider the mutual influence and interaction among equipment, equipment and coal layers, ensure that the equipment of a virtual coal mining machine operates efficiently and stably under a working space jointly constructed by virtual coal layers and the equipment, and provide guidance and service for the production operation of an actual fully-mechanized coal mining working face.
In order to achieve the purpose, the invention adopts the technical scheme that: a virtual fully-mechanized coal mining production system coupling deduction method based on multi-agent learning is characterized in that three-dimensional modeling of a coal machine equipment agent body is carried out according to respective structural parameters and freedom degree characteristics of three machines on a fully-mechanized coal mining working face, and modeling of a virtual coal layer is carried out according to coal layer detection information and coal cutter cutting information; the established model is led into Unity3D to serve as an initial scene of three machines of the whole fully mechanized coal mining face, historical operation information in the equipment mining process is accessed into the scene, and key operation information of the coal equipment is extracted to serve as a data driving source of initial operation;
carrying out centralized iterative training on equipment operation data, obtaining decision information of equipment operation by adopting a distributed control mode, finally realizing speed regulation, roller height regulation and propulsion of a coal mining machine intelligent body under the control of a decision result, adaptively bending and deducing a next circulating mining track of the intelligent body of a scraper conveyor, pushing and sliding, moving a frame, lifting/lowering a column, extending/retracting a mutual aid plate and a correcting frame among the intelligent bodies of a hydraulic support intelligent body, and updating a coal bed according to cutting information of a coal mining machine;
the three machines are a hydraulic support, a scraper conveyor and a coal mining machine, and the established coal mining machine equipment intelligent bodies are a coal mining machine intelligent body, a hydraulic support intelligent body group and a scraper conveyor intelligent body.
Further, under the virtual environment, the motion of the coal machine equipment intelligent body is decided, and the three-machine collaborative propulsion process of the fully mechanized coal mining face is realized, and the method comprises the following steps:
(1) constructing an operating environment by using Ml-Agents plug-ins of Unity3D, and determining information of each agent about the agent and information about other Agents, namely joint information of the agent, azimuth information on a virtual coal seam, key operating action information of the agent, relative azimuth information between the agent and other Agents, position information of the agent and key operating action information of the agent;
(2) the coal equipment intelligent agent selects and executes according to the current coal bed environment and the relative state information between the equipmentThe respective actions further influence the transition and update of the environment state, and the process passes<S,A1,…,An,T,R1,…,Rn>Respectively represent a state set (S) and an action set (A)i) Reward set (R)i) Probability of state transition (T);
(3) using the machine-learned "curiosity options" in Unity3D, the cumulative reward mechanism for a single agent is established as followst，at，st+1Output is
By inputting st，atPredicting the next state
And st+1The greater the difference, atThe greater the curiosity for the unknown state, the greater the reward; by training a filter, some characteristics irrelevant to the behavior of the intelligent agent of the coal equipment are filtered, and the filtered behavior state is input into another network to obtain the behavior state
(4) Recording the expectation of the whole mining process, namely the expectation of realizing the maximum mining rate as Q (s, a) on the premise of ensuring safe mining, and decomposing the expectation into the weighted sum of local Qi (si, ai), wherein all coal equipment intelligent bodies have respective local value functions, and the mining target of three machines on the fully mechanized mining face is decomposed into the operation target of single coal equipment;
(5) integrating the operation decision process of single coal mining equipment, determining the maximum expected value of the three-machine operation of the fully mechanized mining face, combining the decision processes of intelligent bodies of the single coal mining equipment by adopting a hybrid network module QMIX method, and combining the operation actions corresponding to the maximized global Qtot value with the local Qa values;
(6) establishing monotonicity constraint relation between a global Q value of a three-machine multi-agent running on a virtual coal seam of a fully mechanized coal mining face and a local Q value of a single device, solving the uncertain problem in the mining process under the complex coal seam condition by adopting a centralized learning method under the premise of considering the combined action effect among the multi-agents, extracting the running strategy of the single coal machine equipment agent from the uncertain problem, and further realizing distributed control, wherein the constraint relation is shown as the following formula, and n is the number of the equipment agents:
(7) in the training process, adding global information of the mining amount, the mining time, the rock remaining amount and the straightness of the fully-mechanized working face in the mining process of the fully-mechanized working face for auxiliary training, guiding the optimization of a strategy through a combined action Q value, and simultaneously, an individual can extract a local Qi value from the global Q value to complete respective decision, so that the distributed control of multiple intelligent agents is realized; selecting a maximized global Qtot value as an iterative update target, and selecting the action of each agent in each iteration;
(8) to obtain the optimal QtotAnd an optimal strategy, establishes the fully mechanized coal mining industryAnd the operation mechanism is a coupling propulsion operation mechanism of three machines on the working face, and the intelligent bodies of the coal equipment realize the coupling propulsion of the three machines on the fully mechanized coal mining face according to respective optimal operation strategies and the guidance of the integral operation strategy.
Further, in the step (6), the joint action effect means that in the process of achieving the optimal joint action among the coal equipment intelligent agents, the motions among the intelligent agents are modeled mutually, and potential motion information can be obtained from other intelligent agents to make a decision, so that a foundation is established for a cooperation mechanism of the equipment intelligent agents.
Further, the equipment intelligent agent cooperation mechanism is a cooperative operation mechanism between the coal mining machine intelligent agent and the coal seam, between the scraper conveyer intelligent agent, between the hydraulic support intelligent agent and the coal seam, between the scraper conveyer intelligent agent and the virtual coal seam, between the coal mining machine intelligent agent and the hydraulic support intelligent agent group.
Further, the equipment intelligent agent cooperation mechanism is that under the virtual environment created by Unity3D, based on the virtual coal seam mining space, the scraper conveyor intelligent agent is self-adaptively laid on the virtual coal seam floor, the coal mining machine intelligent agent takes the scraper conveyor intelligent agent as a track, the front and rear rollers are self-adaptively heightened to cut coal, and the hydraulic support intelligent agent is timely propelled and supported according to the cutting condition of the coal seam roof floor.
Further, the mutual modeling means that one of the coal equipment agents models the operation strategy of the other agent based on the historical operation information of the other agent in the combined action learning process.
Further, the mutual modeling means that the coal cutter intelligent body performs coal cutting action according to coal bed environment information and posture information of the scraper conveyor intelligent body; the intelligent body of the scraper conveyor interacts with the coal bed environment according to the cutting process information of the intelligent body of the coal mining machine to obtain the bending information of the intelligent body of the scraper conveyor in the next mining cycle; the hydraulic support intelligent body group carries out self-adaptive support according to the cutting top and bottom plate conditions of the coal mining machine intelligent body and pushes according to the acquired bending information of the scraper conveyor intelligent body of the next mining cycle.
Further, the distributed control means that the local value function of each coal equipment intelligent agent only needs to perform local observation under the view angle of the intelligent agent, the action with the largest accumulated expected reward is selected through the local value function to be executed in a distributed mode, and the whole virtual reconstruction system is in a distributed mode during execution.
Further, the hybrid network module QMIX is responsible for merging local value functions of a single coal equipment agent, and in the module, weights of each layer are generated by utilizing a hyper network and absolute value calculation, so that the weights are always in a positive and monotonicity constraint relationship.
Compared with the prior art, the virtual fully-mechanized coal mining face production system deduction method based on multi-agent deep reinforcement learning provided by the invention has the following advantages and prominent innovation points:
1. on the basis of the influence of the existing coal bed conditions on each equipment intelligent body, the mutual influence among the equipment intelligent bodies is considered, a total operation environment based on the coal bed mining space of a single equipment intelligent body and the operation information of other equipment intelligent bodies is formed, the overall information of the mining process of the fully-mechanized coal face is added for auxiliary training, a centralized training and distributed decision control mode is adopted, the unstable problem that the coal bed environment fluctuates indefinitely in the advancing process of the fully-mechanized coal face and the influence of the joint action effect among the intelligent bodies are solved, the optimal action of each equipment intelligent body and the optimal cooperation strategy among the equipment intelligent bodies are ensured, and a cooperative advancing mechanism of mutual influence and mutual guidance among equipment and equipment, equipment and coal beds during operation is realized.
2. On the basis of historical mining data, coal seam prediction information, cutting height information and pose information of an intelligent body of a coal mining machine, pose information of an intelligent body of a scraper conveyor and support information of an intelligent body group of a hydraulic support are fused in real time in Unity3D, a virtual coal seam dynamic updating mechanism is established, key characteristic information is extracted through centralized iterative training, updating of the coal seam along with the advance of three machines of a fully mechanized mining working face is achieved, the updating of coal and rock capacity proportion and boundaries is achieved, whether complex geology is encountered in the future mining process can be checked in a virtual environment, and decision is made on cutting track information of the coal mining machine in the mining process according to the fluctuation condition of the virtual coal seam.
3. The intelligent body of the coal mining machine makes action decisions according to the virtual coal rock boundary, forms an optimal cutting scheme according to the virtual coal bed conditions and feeds the optimal cutting scheme back to the actual coal mining machine to drive the cutting process, so that the maximum mining benefit is ensured; in the process of cutting the coal wall by the intelligent body of the coal mining machine, the effect of group combined action can be considered, and the decision of the intelligent body of other equipment on the optimal behavior of the intelligent body can be guided; under the condition that the inclination angle of the coal seam is large and equipment slipping is easy to occur, the coal mining machine intelligent body calculates the environmental state transition probability of the scraper conveyor intelligent body and an adjacent hydraulic support intelligent body group, determines whether the equipment intelligent body is abnormal or not, and makes a timely decision and stops the abnormal intelligent body state.
4. In the operation decision process of the hydraulic support intelligent body, optimizing the operation of the hydraulic support intelligent body by learning the operation state of the coal bed environment and other equipment intelligent bodies, and in the support process of the hydraulic support intelligent body, establishing optimal decision information between coal bed top plate information and the support posture of the intelligent body so as to analyze the support-surrounding rock coupling relation; when the hydraulic support intelligent body and the scraper conveyor intelligent body operate cooperatively, the behavior decision of the pushing mechanism is influenced by the pose characteristics of the scraper conveyor intelligent body and the fluctuation of the coal bed, and an optimal pushing path is determined; in view of discrete action spaces among the intelligent bodies of the hydraulic support, the interference condition of each action space is analyzed, further, the decision is made whether the actions of frame biting and frame squeezing occur among the intelligent bodies, the cutting action of the intelligent bodies of the coal mining machine is modeled, and the healthy operation of the intelligent body group of the hydraulic support in the subsequent propelling process is ensured.
5. The intelligent body of the scraper conveyor can perform strategy integration according to the running states of the intelligent body of the coal mining machine and the intelligent body group of the hydraulic support and the fluctuation condition of the coal bed, so as to realize self-adaptive bending; the pose information of the intelligent scraper conveyor can guide the modeling of the pushing action of the intelligent hydraulic support, and the straightness control of the intelligent hydraulic support group and the walking track of the intelligent coal mining machine are decided, so that the straightness problem of the whole fully mechanized coal mining face is controlled; according to the width of the cutting bottom plate and the width of the middle groove, the laying posture of the intelligent body of the scraper conveyor can invert the fluctuation of the coal bed, the posture can influence the updating reliability of the virtual coal bed bottom plate, and the laying posture information of the intelligent body of the scraper conveyor and the cutting bottom plate information of the intelligent body of the coal mining machine can be combined to be used as prior information to be provided for the coal bed updating process.
6. The method can be applied to the production, design and service processes of the actual fully mechanized coal mining face, the solution is formulated and evaluated for common engineering problems in the mining process, and the equipment operation information and fault information in the advancing process of the fully mechanized coal mining face are identified and analyzed, so that the purposes of operation rehearsal and monitoring for the whole mining process, active problem discovery and solution strategy providing are achieved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate exemplary embodiments of the invention and, together with the description, serve to explain the invention and not to limit the invention.
FIG. 1 is a block diagram of a virtual fully-mechanized coal mining face production system coupling deduction method based on multi-agent deep reinforcement learning;
FIG. 2 is a schematic view of the fully mechanized coal mining face propulsion;
FIG. 3 is a frame diagram of a virtual reconstruction method of a fully mechanized coal mining face production system based on multi-agent deep reinforcement learning;
FIG. 4 is a virtual reconstruction Q-value network architecture of a fully mechanized coal mining face production system;
FIG. 5 is a schematic view of a virtual reconfiguration decision and control method of a fully mechanized coal mining face production system;
FIG. 6 is a diagram illustrating functions that can be realized after the virtual fully-mechanized coal mining face production system deduction method based on multi-agent deep reinforcement learning is applied;
FIG. 7 is a selection and optimization of equipment for a virtual fully-mechanized coal mining face production system deduction method based on multi-agent deep reinforcement learning;
FIG. 8 is a diagram illustrating functions that can be realized after the virtual fully-mechanized coal mining face production system deduction method based on multi-agent deep reinforcement learning is applied;
FIG. 9 is a model selection and optimization performed by a virtual fully-mechanized coal mining face production system deduction method based on multi-agent deep reinforcement learning;
FIG. 10 is a scraper conveyor straightness adjustment for a virtual fully mechanized coal mining face production system deduction method based on multi-agent deep reinforcement learning;
fig. 11 shows a monitoring study on the up-and-down movement of the scraper conveyor based on the virtual fully mechanized coal mining face production system deduction method of multi-agent deep reinforcement learning.
Detailed Description
The invention adopts multi-agent deep reinforcement learning as the technical support of three-machine cooperation of the fully mechanized coal mining face, and realizes self-adaptive cooperative propulsion of equipment on a virtual coal seam and mutual cooperative propulsion of the equipment in the process of three-machine operation propulsion of the fully mechanized coal mining face. According to the running characteristics of the coal equipment in the advancing process, the coal equipment is regarded as an intelligent agent of the coal equipment, so that the cooperative advancing process of three machines on a virtual coal seam of a fully mechanized mining face can be regarded as that under the driving of historical mining information, under the environment of a virtual mining space, the intelligent agents of the equipment consider the combined action effect with other intelligent agents in the working process, and cooperate with each other to jointly complete the mining process.
Fig. 1 is a block diagram of a method for deducing a virtual fully-mechanized coal mining face production system for multi-agent deep reinforcement learning. The method comprises the steps of carrying out three-dimensional modeling on an intelligent body according to respective structural parameters and freedom degree characteristics of three machines (namely a hydraulic support, a scraper conveyor and a coal mining machine) of the fully mechanized mining face, carrying out virtual coal seam modeling according to coal seam detection information and cutting information of the coal mining machine, leading the established three-dimensional model into Unity3D to serve as an initial scene of the three machines of the fully mechanized mining face, accessing historical operation information in an equipment mining process into the scene, and extracting coal equipment operation key information to serve as a data driving source of initial operation. The method comprises the steps of carrying out centralized iterative training on equipment operation data, obtaining decision information of equipment operation by adopting a distributed control mode, finally realizing speed regulation, roller height adjustment and propulsion of a coal mining machine intelligent body in a virtual environment under the control of a decision result, carrying out self-adaptive bending and deduction of a next circulating mining track by the intelligent body of a scraper conveyor, carrying out pushing and sliding, moving, lifting/lowering columns, stretching/retracting mutual-aid plates and correcting frames among the intelligent bodies of a hydraulic support intelligent body, and updating a coal bed according to cutting information of the coal mining machine.
As shown in fig. 2, which is a schematic view of the fully mechanized mining face, the coal mining machine performs coal cutting operation with the scraper conveyor as a running track, and after the coal mining machine cuts a certain distance, the hydraulic support pushes the scraper conveyor, and supports the top plate in time to avoid collapse. After the coal cutter finishes cutting a cut, the coal cutter cuts coal reversely, and the process is repeated in a circulating way, so that the propulsion of the fully mechanized mining face is realized.
In the process of realizing the three-machine cooperative propulsion of the virtual fully mechanized coal mining face, the three-machine cooperative propulsion is mainly divided into three parts: the problems of cutting operation of a coal cutter, flatness guarantee of a scraper conveyor and support of hydraulic support groups are solved, but the three problems are related to each other. The method specifically comprises the following steps: 1. the coal cutter cuts coal by taking the scraper conveyor as a track, the straightness of a coal wall needs to be ensured in the coal cutting process, the straightness of the scraper conveyor needs to be ensured in the process, and the straightness control needs to be realized by controlling the pushing action of the hydraulic support. 2. The height of the roller needs to be adjusted during the cutting process of the coal mining machine so as to ensure the maximum production of the fully mechanized coal mining face, and the goaf after the cutting is finished needs a hydraulic support to be supported in time, which also determines the supporting posture of the hydraulic support. 3. When the hydraulic support pushes the scraper conveyor, the specific pushing distance required for ensuring the straight line needs to be determined by considering the motion process of each structure of the pushing mechanism, and then the straightness of the scraper conveyor is controlled.
In view of the implementation of the above, the motion of the coal equipment intelligent agent needs to be decided in a virtual environment, so as to realize a three-machine cooperative propulsion process of the fully mechanized coal mining face.
Fig. 3 is a diagram of an implementation process of a virtual fully-mechanized coal mining face production system deduction process of multi-agent deep reinforcement learning, specifically including the following steps:
(1) and constructing a running environment by using Ml-Agents plug-in of Unity3D, and determining information of each agent about the agent and information about other Agents, namely joint information of each agent, azimuth information on the virtual coal seam, key running action information of each agent, relative azimuth information between each agent and other Agents, position information of each agent and key running action information of each agent.
(2) The coal equipment intelligent body selects and executes respective actions according to the current coal bed environment and the relative state information among the equipment, so as to influence the transfer and update of the environment state, and the process passes<S,A1,…,An,T,R1,…,Rn>Respectively represent a state set (S) and an action set (A)i) Reward set (R)i) The state transition probability (T).
(3) Using the machine-learned "curiosity options" in Unity3D, a cumulative reward mechanism for a single agent as shown in FIG. 4 is established as followst，at，st+1Output is
By inputting st，atPredicting the next state
And st+1The greater the difference, atThe greater the curiosity for the unknown state, the greater the reward; by training a filter, filtering some characteristics irrelevant to the behavior of the coal equipment intelligent body, inputting the filtered behavior state into another network to better optimize the Feature extractor Feature Ext to obtain
The key to the above process is the curiosity module shown in fig. 5. Network1 and Network2 in the figure are represented by create _ forward _ model and create _ inverse _ model in the script respectively, using the Curiosity function by adding forward _ loss and inverse _ loss in the script.
(4) As shown in fig. 6, the expected effect of the whole mining process, that is, the achievement of the maximum mining rate is marked as Q (s, a) on the premise of ensuring safe mining, and the maximum mining rate is decomposed into the weighted sum of local Qi (si, ai), and the coal equipment agents all have respective local value functions, so that the mining target of the three machines on the fully mechanized mining face is decomposed into the operation target of a single coal equipment.
(5) Integrating the operation decision processes of single coal mining equipment, determining the maximum expected value of the three-machine operation of the fully mechanized mining face, combining the decision processes of the intelligent bodies of the single coal mining equipment by adopting a QMIX (hybrid network module) method, wherein the operation action corresponding to the maximum global Qtot value is the combination of the local Qa values.
(6) As shown in fig. 7, a monotonicity constraint relationship between a global Q value of a fully mechanized coal mining face "three-machine" multi-agent operating on a virtual coal seam and a local Q value of a single device is established, an uncertain problem in an mining process under a complex coal seam condition is solved by adopting a centralized learning method under the premise of considering a joint action effect among the multi-agents, an operating strategy of a single coal machine equipment agent is extracted from the uncertain problem, and distributed control is further realized, wherein the constraint relationship is shown in the following formula, wherein n is the number of equipment agents:
(7) in the training process, the overall information of the mining amount, the mining time, the rock remaining amount and the straightness of the fully-mechanized working face in the mining process of the fully-mechanized working face is added for auxiliary training, the optimization of the strategy is guided through the combined action Q value, meanwhile, the individual can extract the local Qi value from the overall Q value to complete respective decision, and the distributed control of the multiple intelligent agents is realized. And selecting the maximized global Qtot value as an iterative updating target, and selecting the action of each agent in each iteration.
(8) To obtain the optimal QtotAnd an optimal strategy is established, a three-machine coupling propulsion operation mechanism of the fully mechanized coal mining face is established, and all coal machine equipment intelligent bodies realize the three-machine coupling propulsion of the fully mechanized coal mining face according to the respective optimal operation strategy and the guidance of the integral operation strategy.
As shown in fig. 8, the functions that can be realized after the method for deducing "three machines" of the fully mechanized working face based on multi-agent deep reinforcement learning is applied can be realized, after the highly reliable collaborative operation process of "three machines" of the fully mechanized working face is realized, the dynamic coupling between the "three machines" of the fully mechanized working face and the working space can be realized, the analysis and the preview of the whole process of the operation and the monitoring of the whole production system can be realized, the dynamic characteristics of the equipment and the association relationship between the equipment and the coal seam during the operation of the coal machine are revealed, the health condition of the whole machine of the equipment of the coal machine in the mining process can be recognized and predicted, a visual analysis platform and a health monitoring platform are provided for the mining coal mine, and a technical platform support is provided for the highly efficient operation of the production system of the fully mechanized working face.
After the three-machine cooperative propulsion of the fully mechanized coal mining face is realized, the digitalized equipment is controlled to run in real time in a digitalized underground environment by means of connecting a real control program, manual button operation and the like when a virtual debugging process is realized. The digital underground environment design needs to comprehensively consider complex geology, equipment faults and common engineering problems, carry out problem analysis and determine a reasonable scheme. The real description of the problems possibly occurring in the mining process is realized in the joint debugging process, so that the aim of realizing the actual underground debugging by the virtual debugging in the digital space is fulfilled. Some of the key parameters are set as follows:
(1) the problem of information transmission interruption is set through program interaction in the Unity3D, and related data in the well are analyzed to obtain an operation rule;
(2) in consideration of sensor failure and abnormal problems, the problems are effectively processed by establishing an early warning mechanism in the interaction process of the Unity3D and performing relevant compensation.
(3) The combined debugging under complex working conditions such as real placement of the equipment is carried out, the setting is carried out through related data of similar mining areas, and the equipment performance and the sensing control element are calibrated.
After the highly reliable deduction process of 'three machines' of the fully mechanized coal mining face is realized, corresponding service can be provided for the coal mining process. And transmitting the equipment operation information in the real-time mining process to a virtual scene, identifying the equipment operation state and sensing the space fault in the mining process by using the visual virtual scene, and providing a solution.
The coal mining machine equipment intelligent bodies refer to a coal mining machine intelligent body, a hydraulic support intelligent body group and a scraper conveyor intelligent body.
The coal mining machine intelligent body is capable of achieving self-adaptive height adjustment of front and rear rollers of the coal mining machine according to coal seam fluctuation conditions and self-adaptive walking along a scraper conveyor, the whole virtual coal mining process is more efficient due to the application of the coal mining machine intelligent body, and the height is mapped to the actual coal mining process.
The hydraulic support intelligent agent group is composed of single hydraulic support intelligent agents, the single hydraulic support intelligent agents can realize pushing sliding, moving, lifting, descending and stretching/retracting of the mutual aid plates, the hydraulic support intelligent agents can be adaptively supported and corrected, and abnormal postures among the hydraulic support intelligent agents are avoided.
The intelligent body of the scraper conveyor can be paved on a virtual coal seam bottom plate in a self-adaptive manner under a virtual environment, and can be pushed in a self-adaptive manner in a bending manner along with the movement of a coal mining machine on the basis of the cooperative action of a hydraulic support group.
The joint action effect means that the movement among the intelligent agents of the coal equipment can be modeled mutually when the intelligent agents of the coal equipment achieve the optimal joint action problem, and the potential movement information can be obtained from other intelligent agents to make decisions, so that a foundation is laid for establishing a cooperative mechanism of the intelligent agents of the equipment.
The equipment intelligent body cooperation mechanism is a cooperative operation mechanism between a coal mining machine intelligent body and a coal seam, between a scraper conveyor intelligent body, between a hydraulic support intelligent body and a coal seam, between a scraper conveyor intelligent body and a virtual coal seam, between a coal mining machine intelligent body and a hydraulic support intelligent body group. Specifically, under a virtual environment created by Unity3D, based on a virtual coal seam mining space, a scraper conveyor intelligent body is paved on a virtual coal seam bottom plate in a self-adaptive manner, a coal mining machine intelligent body takes the scraper conveyor intelligent body as a track, front and rear rollers are heightened in a self-adaptive manner to cut coal, and a hydraulic support intelligent body is timely propelled and supported according to the cutting condition of the coal seam top bottom plate.
The mutual modeling means that one of the coal machine equipment intelligent agents can model the operation strategies of other intelligent agents based on historical operation information of other intelligent agents in the combined action learning process. Specifically, the coal cutter intelligent body performs coal cutting action according to coal bed environment information and the posture information of the scraper conveyor intelligent body; the intelligent body of the scraper conveyor interacts with the coal bed environment according to the cutting process information of the intelligent body of the coal mining machine to obtain the bending information of the intelligent body of the scraper conveyor in the next mining cycle; the hydraulic support intelligent body group carries out self-adaptive support based on the cutting top and bottom plate condition of the coal mining machine intelligent body, and pushes according to the acquired bending information of the scraper conveyor intelligent body of the next mining cycle.
The distributed control means that a local value function of each coal machine equipment intelligent body only needs to perform local observation under the view angle of the intelligent body, the action with the largest accumulated expected reward is selected through the local value function to be executed in a distributed mode, and the whole virtual reconstruction system is in a distributed mode during execution.
The hybrid network module is responsible for merging local value functions of intelligent agents of single coal equipment, and in the hybrid network module, weights of all layers are generated by utilizing a hyper network and absolute value calculation, so that the weights are always in a positive and monotonicity constraint relation.
As shown in fig. 9, taking the equipment model selection stage in the early stage of the fully mechanized mining face production as an example, the established three-machine deduction method of the fully mechanized mining face for multi-agent deep reinforcement learning is used for model selection and optimization. In the type selection process of the three-machine equipment product on the fully mechanized coal mining face, because each equipment is subjected to more condition constraints during type selection, a typical multi-target decision phenomenon exists. The method comprises the following steps of selecting the model of a coal mining machine according to the mining height of a coal seam, the coal mining transportation requirement and the reliability requirement of fully-mechanized working face equipment, selecting the model of a scraper conveyor according to the maximum cutting amount of the coal mining machine, the width of a roller of the coal mining machine and the length of a fully-mechanized working face, selecting the model of a hydraulic support according to the mining height and the fluctuation condition of the coal seam and the requirements of the coal mining machine and the scraper conveyor, and thus preliminarily selecting the model of matched equipment: the model of the hydraulic support is ZY11000/18/38D, the model of the coal mining machine is MG400/920-WD, and the model of the scraper conveyor is SGZ800/2x 525. After modeling, the method is applied to a virtual environment and a virtual fully-mechanized coal mining face production system deduction method based on multi-agent deep reinforcement learning is established, the operation condition of the matched equipment is analyzed, and a matched scheme is optimized. The original and optimized protocols obtained are shown in the following table.
Fig. 10 shows a research process for the linearity adjustment of the scraper conveyor in a virtual environment during the operation of the fully mechanized mining face. Performing centralized training by taking historical pose information of an actual hydraulic support group and trajectory information of a scraper conveyor obtained by inversion based on coal mining machine trajectory information in a virtual environment as a data source in a straightening process, obtaining a motion decision result of each degree of freedom of a single hydraulic support intelligent body pushing mechanism, and predicted trajectory information of the scraper conveyor and decision position information of a hydraulic support pushing point under a multi-factor condition, wherein the pushing mechanism is always connected with the scraper conveyor and moves according to the motion decision result of each degree of freedom of the pushing mechanism according to a linkage effect between the hydraulic support intelligent body and the scraper conveyor intelligent body, so as to complete virtual connection between the hydraulic support intelligent body group and the scraper conveyor; when the intelligent body of the scraper conveyor is pushed forward, the intelligent body of the hydraulic support is timely pushed under the action of decision information, and straightening of the scraper conveyor in a virtual environment is realized.
Fig. 11 is a schematic view of virtual monitoring of the run-up and run-down of a scraper conveyor. The method comprises the steps of constructing a multi-agent operation environment based on coal seam detection information and three-machine equipment historical operation information of the fully mechanized coal face, establishing a three-machine cooperative operation method of the virtual fully mechanized coal face by using knowledge in fig. 3, virtually monitoring the position of a scraper conveyor in the advancing process, judging whether upward movement and downward movement occur according to the movement amount of the head of the scraper conveyor, and giving an early warning for whether the upward movement and downward movement phenomenon can occur in the advancing process of the real fully mechanized coal face. From two angles between the coal seam and the equipment, considering the fluctuation condition of the bottom plate of the coal seam, the inclination angle of the coal seam, the friction factor between the equipment and the bottom plate, the acting force of the coal mining machine on the scraper conveyor and the influence of the virtual running resistance of each middle groove on the upward movement and the downward movement of the scraper conveyor, on the basis of realizing virtual monitoring, the influences of the A. coal seam inclination angle, the B. friction factor and the C. rotating speed of a roller of the coal mining machine on the upward movement and the downward movement of the scraper conveyor are analyzed, and the analysis results are shown in the following two tables.
Test protocol and results analysis Table
Analysis of variance table
From the comparison of the F value and the critical value, factor a is a significant factor. The primary and secondary sequence of the influence of each factor on the test results is A, C, B, namely the coal seam inclination angle, the rotation speed of the roller of the coal mining machine and the friction factor.
The scope of the invention is not limited to the above embodiments, and various modifications and changes may be made by those skilled in the art, and any modifications, improvements and equivalents within the spirit and principle of the invention should be included in the scope of the invention.
Claims (9)
1. A virtual fully mechanized mining production system deduction method based on multi-agent deep reinforcement learning is characterized in that:
carrying out three-dimensional modeling on an intelligent body of coal equipment according to respective structural parameters and freedom characteristics of three machines of the fully mechanized coal mining face, and carrying out modeling on a virtual coal bed according to coal bed detection information and cutting information of a coal mining machine; the established model is led into Unity3D to serve as an initial scene of three machines of the whole fully mechanized coal mining face, historical operation information in the equipment mining process is accessed into the scene, and key operation information of the coal equipment is extracted to serve as a data driving source of initial operation;
carrying out centralized iterative training on equipment operation data, obtaining decision information of equipment operation by adopting a distributed control mode, finally realizing speed regulation, roller height regulation and propulsion of a coal mining machine intelligent body under the control of a decision result, adaptively bending and deducing a next circulating mining track of the intelligent body of a scraper conveyor, pushing and sliding, moving a frame, lifting/lowering a column, extending/retracting a mutual aid plate and a correcting frame among the intelligent bodies of a hydraulic support intelligent body, and updating a coal bed according to cutting information of a coal mining machine;
the three machines are a hydraulic support, a scraper conveyor and a coal mining machine, and the established coal mining machine equipment intelligent bodies are a coal mining machine intelligent body, a hydraulic support intelligent body group and a scraper conveyor intelligent body.
2. The virtual fully-mechanized mining production system deduction method based on multi-agent deep reinforcement learning of claim 1, wherein: under the virtual environment, the motion of an intelligent agent of coal equipment is decided, and the three-machine cooperative propulsion process of the fully mechanized coal mining face is realized, and the method comprises the following steps:
(1) constructing an operating environment by using Ml-Agents plug-ins of Unity3D, and determining information of each agent about the agent and information about other Agents, namely joint information of the agent, azimuth information on a virtual coal seam, key operating action information of the agent, relative azimuth information between the agent and other Agents, position information of the agent and key operating action information of the agent;
(2) the coal equipment intelligent body selects and executes respective actions according to the current coal bed environment and the relative state information among the equipment, so as to influence the transfer and update of the environment state, and the process passes<S,A1,…,An,T,R1,…,Rn>Respectively represent a state set (S) and an action set (A)i) Reward set (R)i) Probability of state transition (T);
(3) using the machine-learned "curiosity options" in Unity3D, the cumulative reward mechanism for a single agent is established as followst，at，st+1Output is
By inputting st，atPredicting the next state
And st+1The greater the difference, atThe greater the curiosity for the unknown state, the greater the reward; by training a filter, some characteristics irrelevant to the behavior of the intelligent agent of the coal equipment are filtered, and the filtered behavior state is input into another network to obtain the behavior state
(4) Recording the expectation of the whole mining process, namely the expectation of realizing the maximum mining rate as Q (s, a) on the premise of ensuring safe mining, and decomposing the expectation into the weighted sum of local Qi (si, ai), wherein all coal equipment intelligent bodies have respective local value functions, and the mining target of three machines on the fully mechanized mining face is decomposed into the operation target of single coal equipment;
(5) integrating the operation decision process of single coal mining equipment, determining the maximum expected value of the three-machine operation of the fully mechanized mining face, combining the decision processes of intelligent bodies of the single coal mining equipment by adopting a hybrid network module QMIX method, and combining the operation actions corresponding to the maximized global Qtot value with the local Qa values;
(6) establishing monotonicity constraint relation between a global Q value of a three-machine multi-agent running on a virtual coal seam of a fully mechanized coal mining face and a local Q value of a single device, solving the uncertain problem in the mining process under the complex coal seam condition by adopting a centralized learning method under the premise of considering the combined action effect among the multi-agents, extracting the running strategy of the single coal machine equipment agent from the uncertain problem, and further realizing distributed control, wherein the constraint relation is shown as the following formula, and n is the number of the equipment agents:
(7) in the training process, adding global information of the mining amount, the mining time, the rock remaining amount and the straightness of the fully-mechanized working face in the mining process of the fully-mechanized working face for auxiliary training, guiding the optimization of a strategy through a combined action Q value, and simultaneously, an individual can extract a local Qi value from the global Q value to complete respective decision, so that the distributed control of multiple intelligent agents is realized; selecting a maximized global Qtot value as an iterative update target, and selecting the action of each agent in each iteration;
(8) to obtain the optimal QtotAnd an optimal strategy is established, a coupling propulsion operation mechanism of three machines of the fully mechanized coal mining face is established, and the intelligent bodies of the coal machine equipment realize the coupling propulsion of the three machines of the fully mechanized coal mining face according to the respective optimal operation strategy and the guidance of the integral operation strategy.
3. The virtual fully-mechanized mining production system deduction method based on multi-agent deep reinforcement learning of claim 2, wherein: in the step (6), the joint action effect means that the motions among the intelligent agents of the coal equipment are modeled mutually when the optimal joint action problem is achieved among the intelligent agents, and potential motion information can be obtained from other intelligent agents to make decisions, so that a foundation is established for a cooperation mechanism of the intelligent agents.
4. The multi-agent deep reinforcement learning-based virtual fully-mechanized coal mining production system deduction method according to claim 3, wherein: the equipment intelligent body cooperation mechanism is a cooperative operation mechanism between a coal mining machine intelligent body and a coal seam, between a scraper conveyor intelligent body, between a hydraulic support intelligent body and a coal seam, between a scraper conveyor intelligent body and a virtual coal seam, between a coal mining machine intelligent body and a hydraulic support intelligent body group.
5. The multi-agent deep reinforcement learning-based virtual fully-mechanized coal mining production system deduction method according to claim 4, wherein: the equipment intelligent agent cooperation mechanism is that under a virtual environment created by Unity3D, based on a virtual coal seam mining space, a scraper conveyor intelligent agent is self-adaptively laid on a virtual coal seam bottom plate, a coal cutter intelligent agent takes the scraper conveyor intelligent agent as a track, front and rear rollers are self-adaptively heightened to cut coal, and a hydraulic support intelligent agent is timely propelled and supported according to the cutting condition of a coal seam top bottom plate.
6. The multi-agent deep reinforcement learning-based virtual fully-mechanized coal mining production system deduction method according to claim 3, wherein: the mutual modeling means that one of the coal machine equipment intelligent bodies models the operation strategies of other intelligent bodies based on historical operation information of other intelligent bodies in the combined action learning process.
7. The multi-agent deep reinforcement learning-based virtual fully-mechanized coal mining production system deduction method according to claim 6, wherein: the mutual modeling means that the coal cutter intelligent body performs coal cutting action according to coal bed environment information and the posture information of the scraper conveyor intelligent body; the intelligent body of the scraper conveyor interacts with the coal bed environment according to the cutting process information of the intelligent body of the coal mining machine to obtain the bending information of the intelligent body of the scraper conveyor in the next mining cycle; the hydraulic support intelligent body group carries out self-adaptive support according to the cutting top and bottom plate conditions of the coal mining machine intelligent body and pushes according to the acquired bending information of the scraper conveyor intelligent body of the next mining cycle.
8. The multi-agent deep reinforcement learning-based virtual fully-mechanized coal mining production system deduction method according to claim 2, 3, 5 or 7, wherein: the distributed control means that a local value function of each coal machine equipment intelligent body only needs to perform local observation under the view angle of the intelligent body, the action with the largest accumulated expected reward is selected to be executed through the local value function, distributed execution is performed, and the whole virtual reconstruction system is distributed during execution.
9. The multi-agent deep reinforcement learning-based virtual fully-mechanized coal mining production system deduction method according to claim 8, wherein: the hybrid network module QMIX is responsible for merging local value functions of intelligent bodies of single coal equipment, and in the module, weights of all layers are generated by utilizing a hyper network and absolute value calculation, so that the weights are always in a positive and monotonicity constraint relation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111577141.6A CN114329936B (en) | 2021-12-22 | 2021-12-22 | Virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111577141.6A CN114329936B (en) | 2021-12-22 | 2021-12-22 | Virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114329936A true CN114329936A (en) | 2022-04-12 |
| CN114329936B CN114329936B (en) | 2024-03-29 |
Family
ID=81055291
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111577141.6A Active CN114329936B (en) | 2021-12-22 | 2021-12-22 | Virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114329936B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115017677A (en) * | 2022-04-27 | 2022-09-06 | 中国人民解放军军事科学院战略评估咨询中心 | Deduction simulation-oriented action strategy prejudging method and system |
| CN115437248A (en) * | 2022-08-16 | 2022-12-06 | 煤炭科学研究总院有限公司 | Equipment operation control method, device and equipment based on deep Q learning algorithm |
| CN116151043A (en) * | 2023-04-20 | 2023-05-23 | 西安华创马科智能控制***有限公司 | Pose inversion method and device for scraper conveyor |
| CN116291659A (en) * | 2023-05-24 | 2023-06-23 | 太原理工大学 | Hydraulic support man-machine cooperative control strategy recommendation method |
| CN117348500A (en) * | 2023-12-04 | 2024-01-05 | 济南华科电气设备有限公司 | Automatic control method and system for fully-mechanized coal mining face |
| CN118226790A (en) * | 2024-05-22 | 2024-06-21 | 山东济矿鲁能煤电股份有限公司阳城煤矿 | Intelligent coal mining equipment cooperative control system and method |
| WO2024139015A1 (en) * | 2022-12-30 | 2024-07-04 | 中国电子科技集团公司第三十八研究所 | Multi-user collaborative task guaranteeing method based on proxy and backup technology |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102508995A (en) * | 2011-09-26 | 2012-06-20 | 河南理工大学 | Coal mine accident simulating method and system based on multi-intelligent agent |
| CN104317637A (en) * | 2014-10-16 | 2015-01-28 | 安徽理工大学 | Multi-agent-based virtual miner safety behavior modeling and emergency simulation system |
| WO2020000399A1 (en) * | 2018-06-29 | 2020-01-02 | 东莞理工学院 | Multi-agent deep reinforcement learning proxy method based on intelligent grid |
| CN113128109A (en) * | 2021-04-08 | 2021-07-16 | 太原理工大学 | Test and evaluation method for intelligent fully-mechanized mining robot production system |
| WO2021184614A1 (en) * | 2020-03-14 | 2021-09-23 | 天地科技股份有限公司 | Intelligent decision control method and system for fully-mechanized mining equipment used for working surface under complex condition |
-
2021
- 2021-12-22 CN CN202111577141.6A patent/CN114329936B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102508995A (en) * | 2011-09-26 | 2012-06-20 | 河南理工大学 | Coal mine accident simulating method and system based on multi-intelligent agent |
| CN104317637A (en) * | 2014-10-16 | 2015-01-28 | 安徽理工大学 | Multi-agent-based virtual miner safety behavior modeling and emergency simulation system |
| WO2020000399A1 (en) * | 2018-06-29 | 2020-01-02 | 东莞理工学院 | Multi-agent deep reinforcement learning proxy method based on intelligent grid |
| WO2021184614A1 (en) * | 2020-03-14 | 2021-09-23 | 天地科技股份有限公司 | Intelligent decision control method and system for fully-mechanized mining equipment used for working surface under complex condition |
| CN113128109A (en) * | 2021-04-08 | 2021-07-16 | 太原理工大学 | Test and evaluation method for intelligent fully-mechanized mining robot production system |
Non-Patent Citations (2)
| Title |
|---|
| 孙长银;穆朝絮;: "多智能体深度强化学习的若干关键科学问题", 自动化学报, no. 07, 15 July 2020 (2020-07-15) * |
| 谢嘉成;王学文;郝尚清;李娟莉;葛星;史恒波;: "工业互联网驱动的透明综采工作面运行***及关键技术", 计算机集成制造***, no. 12, 15 December 2019 (2019-12-15) * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115017677A (en) * | 2022-04-27 | 2022-09-06 | 中国人民解放军军事科学院战略评估咨询中心 | Deduction simulation-oriented action strategy prejudging method and system |
| CN115437248A (en) * | 2022-08-16 | 2022-12-06 | 煤炭科学研究总院有限公司 | Equipment operation control method, device and equipment based on deep Q learning algorithm |
| WO2024139015A1 (en) * | 2022-12-30 | 2024-07-04 | 中国电子科技集团公司第三十八研究所 | Multi-user collaborative task guaranteeing method based on proxy and backup technology |
| CN116151043A (en) * | 2023-04-20 | 2023-05-23 | 西安华创马科智能控制***有限公司 | Pose inversion method and device for scraper conveyor |
| CN116291659A (en) * | 2023-05-24 | 2023-06-23 | 太原理工大学 | Hydraulic support man-machine cooperative control strategy recommendation method |
| CN116291659B (en) * | 2023-05-24 | 2023-08-08 | 太原理工大学 | Hydraulic support man-machine cooperative control strategy recommendation method |
| CN117348500A (en) * | 2023-12-04 | 2024-01-05 | 济南华科电气设备有限公司 | Automatic control method and system for fully-mechanized coal mining face |
| CN117348500B (en) * | 2023-12-04 | 2024-02-02 | 济南华科电气设备有限公司 | Automatic control method and system for fully-mechanized coal mining face |
| CN118226790A (en) * | 2024-05-22 | 2024-06-21 | 山东济矿鲁能煤电股份有限公司阳城煤矿 | Intelligent coal mining equipment cooperative control system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114329936B (en) | 2024-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114329936B (en) | Virtual fully-mechanized production system deduction method based on multi-agent deep reinforcement learning | |
| WO2021184614A1 (en) | Intelligent decision control method and system for fully-mechanized mining equipment used for working surface under complex condition | |
| CN110195592B (en) | Intelligent shield tunneling pose prediction method and system based on hybrid deep learning | |
| LU501938B1 (en) | Method and system for intelligent analysis of big data on unmanned mining in mine | |
| CN109214076B (en) | Virtual planning method for supporting geographical environment and equipment of fully mechanized coal mining face | |
| CN113128109B (en) | Test and evaluation method for intelligent fully-mechanized mining robot production system | |
| CN112883559B (en) | Planning cutting method and device based on big data system, storage medium and electronic device | |
| CN112132359B (en) | Digital twin satellite pulsation type final assembly workshop design method and system | |
| Shi et al. | An operation optimization method of a fully mechanized coal mining face based on semi-physical virtual simulation | |
| CN111140231B (en) | Coal seam roof and floor path virtual planning method oriented to space-time kinematics of fully-mechanized mining equipment | |
| Ge et al. | A virtual adjustment method and experimental study of the support attitude of hydraulic support groups in propulsion state | |
| AU2021101934A4 (en) | Intelligent prediction of shield existing tunnels and multi objective optimization control method of construction parameters based on data drive | |
| CN115455727A (en) | Digital twin-driven fully-mechanized mining equipment and coal bed coupling system optimization decision method | |
| CN112879024A (en) | Dynamic prediction method, system and equipment for shield attitude | |
| CN115773127A (en) | Intelligent decision-making method, system, equipment and medium for slurry balance shield | |
| CN109783962B (en) | Fully-mechanized coal mining equipment collaborative propulsion simulation method based on virtual reality physical engine | |
| CN116415479A (en) | Continuous oil pipe fatigue life early warning method based on digital twinning | |
| CN114819256A (en) | Continuous real-time trajectory planning method for backhoe excavator | |
| CN112901195B (en) | Automatic section forming control system and method for cantilever longitudinal shaft type heading machine | |
| Wu et al. | Research Status Quo and Trends of Construction Robotics: A Bibliometric Analysis | |
| CN112926154B (en) | Mining model optimization method, mining model optimization system, mining model optimization equipment and readable storage medium | |
| CN113051756B (en) | Virtual-real fusion coal mining machine memory cutting test system and method | |
| JP7492482B2 (en) | Shield tunneling machine tunneling prediction model | |
| CN118192410A (en) | Coal cutter bottom plate cutting track planning method with equipment continuous propulsion as target | |
| Liu | Navigation system of coal mine rescue robot based on virtual reality technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |