CN111949475A - Method and system for achieving distributed task scheduling based on zookeeper shell - Google Patents
Method and system for achieving distributed task scheduling based on zookeeper shell Download PDFInfo
- Publication number
- CN111949475A CN111949475A CN202010681119.5A CN202010681119A CN111949475A CN 111949475 A CN111949475 A CN 111949475A CN 202010681119 A CN202010681119 A CN 202010681119A CN 111949475 A CN111949475 A CN 111949475A
- Authority
- CN
- China
- Prior art keywords
- task
- zookeeper
- scheduling
- shell
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000013515 script Methods 0.000 claims abstract description 14
- 230000001419 dependent effect Effects 0.000 claims abstract description 7
- 238000012544 monitoring process Methods 0.000 claims abstract description 4
- 238000011161 development Methods 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3017—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is implementing multitasking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
- G06F11/3093—Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a method and a system for realizing distributed task scheduling based on zookeeper shell, wherein the operation process of the scheduling method is as follows: the method comprises the following steps: setting all tasks to be scheduled in a zookeeper, wherein each task is set with a node name corresponding to the task; step two: after a task is started, scanning task nodes in a zookeeper, when a certain task starts to be executed, creating a temporary node corresponding to the task on the zookeeper, marking that the task starts to be operated, and writing corresponding information into the temporary node; step three: when other host scripts are started, if a temporary node of the task is found, the host scripts automatically exit; step four: automatically destroying the temporary node on the zookeeper under the condition that the task is quitted after the task is executed or the host computer is down; step five: after the task is completed, another type of node is generated on the zookeeper, the key information for task execution is written in for dependent task scanning, and the information is provided for task execution monitoring. The problem that the current distributed coordination system cannot perform task scheduling well is solved.
Description
Technical Field
The invention relates to the technical field of distributed task scheduling, in particular to a method and a system for achieving distributed task scheduling by using a shell based on zookeeper.
Background
At present, with the development of internet technology and industry, a system architecture with distributed coordination is gradually popularized, a plurality of service systems often exist in a distributed environment, each service system has a requirement for a timed task, a plurality of shell scripts running in a background exist in a maintenance process, and the following problems cannot be effectively solved by a traditional scheduling script based on crontab:
1. some tasks depend on the completion of the preceding task;
2. some tasks are allowed to be executed only one time;
3. after a certain host is hung, tasks cannot be normally executed, and other hosts need to automatically take over;
……
distributed tasks cannot be reliably scheduled, so that the distributed cooperative system faces many technical risks during operation, and challenges are provided for maintenance and later development.
Disclosure of Invention
Aiming at the problem that the current distributed coordination system can not carry out task scheduling well, the invention provides a method and a system for realizing distributed task scheduling by using a shell based on zookeeper.
The invention discloses a method for realizing distributed task scheduling based on zookeeper shell, which comprises the following operation processes:
the method comprises the following steps: setting all tasks to be scheduled in a zookeeper, wherein each task is set with a node name corresponding to the task;
step two: after a task is started, scanning task nodes in a zookeeper, when a certain task starts to be executed, creating a temporary node corresponding to the task on the zookeeper, marking that the task starts to be operated, and writing corresponding information into the temporary node;
step three: when other host scripts are started, if a temporary node of the task is found, the host scripts automatically exit;
step four: automatically destroying the temporary node on the zookeeper under the condition that the task is quitted after the task is executed or the host computer is down;
step five: after the task is completed, another type of node is generated on the zookeeper, the key information for task execution is written in for dependent task scanning, and the information is provided for task execution monitoring.
Preferably, in the step one, the node name corresponding to the task set in the zookeeper is named in a mode of host + task name.
Preferably, the writing of the corresponding information to the temporary node in the second step includes executing the host and executing time.
Preferably, the task execution key information in the fifth step includes log keywords, task execution time consumption, and context information.
Preferably, the method further comprises the step six:
and if the task is required to be dependent, the task completed by the front task is depended on to scan whether the context information is successfully written, if so, the operation is executed, otherwise, the operation is waited.
A system for realizing distributed task scheduling based on zookeeper shell at least comprises a computer and a scheduling system which are connected with the Internet or a local area network system, wherein the computer at least comprises a memory, a processor and a network card, and the memory stores the method for realizing distributed task scheduling based on zookeeper shell or a program for realizing the method; the processor runs an executable program of the method for realizing distributed task scheduling based on the zookeeper shell; the scheduling system runs the executable program of the method for realizing distributed task scheduling based on the zookeeper shell.
Compared with the prior art, the invention has the beneficial effects that:
after the zookeeper-based method and the zookeeper-based system for realizing distributed task scheduling by the shell are adopted, distributed scheduling pain points of shell maintenance scripts are solved by adopting the zookeeper, so that the distributed system can work reliably and stably, the execution condition of the scripts can be monitored by the zookeeper in the operation process of the distributed system, and the controllable and monitorable operation state of the system is ensured. The operation reliability of the distributed system is improved, and the secondary development cost is reduced.
Drawings
FIG. 1 is a system work flow diagram of a method and system for achieving distributed task scheduling with a shell based on zookeeper.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
The invention is described in further detail below with reference to the attached drawing figures:
referring to fig. 1, a method for implementing distributed task scheduling by using a shell based on zookeeper includes the following operation processes:
the method comprises the following steps: setting all tasks to be scheduled in a zookeeper, wherein each task is set with a node name corresponding to the task;
step two: after a task is started, scanning task nodes in a zookeeper, when a certain task starts to be executed, creating a temporary node corresponding to the task on the zookeeper, marking that the task starts to be operated, and writing corresponding information into the temporary node;
step three: when other host scripts are started, if a temporary node of the task is found, the host scripts automatically exit;
step four: automatically destroying the temporary node on the zookeeper under the condition that the task is quitted after the task is executed or the host computer is down;
step five: after the task is completed, another type of node is generated on the zookeeper, the key information for task execution is written in for dependent task scanning, and the information is provided for task execution monitoring.
In specific implementation, the node name corresponding to the task set in the zookeeper in the step one is named in a mode of host + task name.
In a specific implementation, the writing of the corresponding information to the temporary node in the second step includes executing the host and executing time.
In specific implementation, the task execution key information in the fifth step includes log keywords, task execution time consumption, and context information.
In specific implementation, the method further comprises the following steps:
and if the task is required to be dependent, the task completed by the front task is depended on to scan whether the context information is successfully written, if so, the operation is executed, otherwise, the operation is waited.
A system for realizing distributed task scheduling based on zookeeper shell at least comprises a computer and a scheduling system which are connected with the Internet or a local area network system, wherein the computer at least comprises a memory, a processor and a network card, and the memory stores the method for realizing distributed task scheduling based on zookeeper shell or a program for realizing the method; the processor runs an executable program of the method for realizing distributed task scheduling based on the zookeeper shell; the scheduling system runs the executable program of the method for realizing distributed task scheduling based on the zookeeper shell.
According to the method and the system for realizing distributed task scheduling by using the shell based on the zookeeper, disclosed by the invention, distributed scheduling pain points of a shell maintenance script are solved by adopting the zookeeper, so that the distributed system can work reliably and stably, the script execution condition can be monitored by the zookeeper in the operation process of the distributed system, and the controllable and monitorable operation state of the system is ensured. The operation reliability of the distributed system is improved, and the secondary development cost is reduced.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes will occur to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. A method for realizing distributed task scheduling by using a shell based on zookeeper is characterized in that the operation process of the scheduling method is as follows:
the method comprises the following steps: setting all tasks to be scheduled in a zookeeper, wherein each task is set with a node name corresponding to the task;
step two: after a task is started, scanning task nodes in a zookeeper, when a certain task starts to be executed, creating a temporary node corresponding to the task on the zookeeper, marking that the task starts to be operated, and writing corresponding information into the temporary node;
step three: when other host scripts are started, if a temporary node of the task is found, the host scripts automatically exit;
step four: automatically destroying the temporary node on the zookeeper under the condition that the task is quitted after the task is executed or the host computer is down;
step five: after the task is completed, another type of node is generated on the zookeeper, the key information for task execution is written in for dependent task scanning, and the information is provided for task execution monitoring.
2. The method for realizing distributed task scheduling by shell based on zookeeper of claim 1, wherein: in the first step, the node names corresponding to the tasks set in the zookeeper are named in a mode of host computer + task name.
3. The method for realizing distributed task scheduling by shell based on zookeeper of claim 1, wherein: and writing corresponding information into the temporary node in the second step comprises an execution host and an execution time.
4. The method for realizing distributed task scheduling by shell based on zookeeper of claim 1, wherein: and in the fifth step, the task execution key information comprises log key words, task execution time consumption and context information.
5. The method for realizing distributed task scheduling by shell based on zookeeper of claim 1, wherein: the method further comprises the steps of six:
and if the task is required to be dependent, the task completed by the front task is depended on to scan whether the context information is successfully written, if so, the operation is executed, otherwise, the operation is waited.
6. A zookeeper-based system for realizing distributed task scheduling by shell at least comprises a computer connected with the Internet or a local area network system and a scheduling system, and is characterized in that: the computer at least comprises a memory, a processor and a network card, wherein the memory stores the method for realizing distributed task scheduling based on zookeeper shell or the program for realizing the method in the claims 1-5; the processor runs an executable program of the zookeeper-based shell-based distributed task scheduling method of claims 1-5; the scheduling system runs the executable program of the zookeeper-based method for implementing distributed task scheduling with shell as claimed in claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010681119.5A CN111949475A (en) | 2020-07-15 | 2020-07-15 | Method and system for achieving distributed task scheduling based on zookeeper shell |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010681119.5A CN111949475A (en) | 2020-07-15 | 2020-07-15 | Method and system for achieving distributed task scheduling based on zookeeper shell |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111949475A true CN111949475A (en) | 2020-11-17 |
Family
ID=73341096
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010681119.5A Pending CN111949475A (en) | 2020-07-15 | 2020-07-15 | Method and system for achieving distributed task scheduling based on zookeeper shell |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111949475A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656159A (en) * | 2021-08-13 | 2021-11-16 | 山东浪潮科学研究院有限公司 | Decentralized quantum computing distributed task scheduling method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103916481A (en) * | 2014-04-17 | 2014-07-09 | 北京京东尚科信息技术有限公司 | Method and device for processing data |
| CN106371918A (en) * | 2016-08-23 | 2017-02-01 | 北京云纵信息技术有限公司 | Task cluster scheduling management method and apparatus |
| CN108304255A (en) * | 2017-12-29 | 2018-07-20 | 北京城市网邻信息技术有限公司 | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing |
-
2020
- 2020-07-15 CN CN202010681119.5A patent/CN111949475A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103916481A (en) * | 2014-04-17 | 2014-07-09 | 北京京东尚科信息技术有限公司 | Method and device for processing data |
| CN106371918A (en) * | 2016-08-23 | 2017-02-01 | 北京云纵信息技术有限公司 | Task cluster scheduling management method and apparatus |
| CN108304255A (en) * | 2017-12-29 | 2018-07-20 | 北京城市网邻信息技术有限公司 | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656159A (en) * | 2021-08-13 | 2021-11-16 | 山东浪潮科学研究院有限公司 | Decentralized quantum computing distributed task scheduling method |
| CN113656159B (en) * | 2021-08-13 | 2023-05-16 | 山东浪潮科学研究院有限公司 | Decentralized quantum computing distributed task scheduling method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110457556B (en) | Distributed crawler system architecture, method for crawling data and computer equipment | |
| CN105068865A (en) | Task scheduling method and apparatus | |
| CN108089915B (en) | Method and system for business control processing based on message queue | |
| WO2018188380A1 (en) | Method and device for adding control identification | |
| CN110502399B (en) | Fault detection method and device | |
| CN112153126A (en) | Deployment and node management method and system for K8S cluster | |
| CN111865978A (en) | Method, device, equipment and medium for updating request identifier of micro-service | |
| CN111949475A (en) | Method and system for achieving distributed task scheduling based on zookeeper shell | |
| CN115858049B (en) | RPA flow componentization arrangement method, device, equipment and medium | |
| CN113010280A (en) | Distributed task processing method, system, device, equipment and medium | |
| CN111147541A (en) | Node processing method, device and equipment based on parameter server and storage medium | |
| CN105787359A (en) | Course guarding method and device | |
| CN112860391B (en) | Dynamic cluster rendering resource management system and method | |
| CN107015787B (en) | Method and device for designing interactive application framework | |
| CN111708568B (en) | Modularized development decoupling method and terminal | |
| CN108196982A (en) | A kind of LINUX system restarts the configuration method of service automatically | |
| US8359602B2 (en) | Method and system for task switching with inline execution | |
| CN111258592B (en) | Installation-free serial execution method and system applied to cluster server | |
| CN105487847B (en) | A kind of function dissemination method and equipment | |
| CN112632559A (en) | Vulnerability automatic verification method, device, equipment and storage medium | |
| CN109375966B (en) | Method, device and equipment for initializing node and storage medium | |
| CN111552586A (en) | Application crash processing method and device | |
| CN110955539A (en) | Process quitting method and device, electronic equipment and machine-readable storage medium | |
| CN110119325B (en) | Server fault processing method, device, equipment and computer readable storage medium | |
| US11894981B1 (en) | Systems and methods for generating soar playbooks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |