CN107766138A - A kind of hadoop Mission Monitors method and system - Google Patents
A kind of hadoop Mission Monitors method and system Download PDFInfo
- Publication number
- CN107766138A CN107766138A CN201710984747.9A CN201710984747A CN107766138A CN 107766138 A CN107766138 A CN 107766138A CN 201710984747 A CN201710984747 A CN 201710984747A CN 107766138 A CN107766138 A CN 107766138A
- Authority
- CN
- China
- Prior art keywords
- task
- hadoop
- white list
- judge
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/554—Detecting local intrusion or implementing counter-measures involving event detection and direct action
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Debugging And Monitoring (AREA)
Abstract
A kind of hadoop Mission Monitors method, including：(1) start, perform the operation of initialization；(2) user white list white users are obtained：The name on account list having permission is obtained by inquiring about database, is easy to task user name validity checking below；(3) task white list white tasks are obtained：Task white list list is obtained by inquiring about database, is easy to task validity checking below, the task in white list skips over；(4) the task task set being currently running are obtained：Set of tasks is obtained by hadoop api；(5) judge whether task set are empty, be then to perform step (6), otherwise perform step (8)；(6) a task task is removed from task set：A task is taken out from task set；(7) single task legitimacy is checked：The indices of inspection task, judge whether task is illegal, jump to step (5)；(8) terminate.Also system.
Description
Technical field
The present invention relates to the technical field of big data processing, more particularly to a kind of hadoop Mission Monitors method, and
Hadoop Mission Monitor systems.
Background technology
The task look facility realized on web page, user can only check web page at present, if task is seriously dragged
Tired cluster, can only manually find, kill kill manually.So user experience is very poor, wastes a large amount of labours.
The content of the invention
The defects of to overcome prior art, the technical problem to be solved in the present invention is to provide a kind of hadoop Mission Monitors
Method, the occupation condition of each task of its automatic real-time monitoring hadoop cluster, the task that finds take resource and excessively will
Task can be closed automatically.
The technical scheme is that：This hadoop Mission Monitors method, this method comprise the following steps：
(1) start, perform the operation of initialization；
(2) user white list white users are obtained：The name on account list having permission is obtained by inquiring about database,
It is easy to task user name validity checking below；
(3) task white list white tasks are obtained：Task white list list is obtained by inquiring about database, after being easy to
The task validity checking in face, the task in white list skip over；
(4) the task task set being currently running are obtained：Set of tasks is obtained by hadoop api；
(5) judge whether task set are empty, be then to perform step (6), otherwise perform step (8)；
(6) a task task is removed from task set：A task is taken out from task set；
(7) single task legitimacy is checked：The indices of inspection task, judge whether task is illegal, jump to step
Suddenly (5)；
(8) terminate.
The present invention is quasi real time protected the safety of hadoop systems, supervised in real time automatically by the automatic Predicated execution of whole process
The occupation condition of each task of hadoop cluster is controlled, discovery task, which takes resource, will excessively close task automatically.
A kind of hadoop Mission Monitors system is additionally provided, the system includes：
Initialization module, it is configured to perform the operation of initialization；
User's white list module is obtained, it is configured to obtain the name on account list having permission by inquiring about database, just
Task user name validity checking later；
Acquisition task white list module, its configure come by inquire about database obtain task white list list, be easy to behind
Task validity checking, the task in white list skips over；
The task module being currently running is obtained, it, which is configured, to obtain set of tasks by hadoop api；
Judge module, it is configured to judge whether task set are empty, is then to perform to remove module, is otherwise terminated；
Module is removed, it, which is configured, to remove a task task from task set：One is taken out from task set to appoint
Business；
Module is checked, it is configured to check single task legitimacy：Whether the indices of inspection task, judge task
Illegally, judge module is jumped to.
Brief description of the drawings
Fig. 1 show the flow chart of the hadoop Mission Monitor methods according to the present invention.
Fig. 2 show the flow chart of the step of hadoop Mission Monitor methods according to the present invention (7).
Embodiment
As shown in figure 1, this hadoop Mission Monitors method, this method comprise the following steps：
(1) start, perform the operation of initialization；
(2) user white list white users are obtained：The name on account list having permission is obtained by inquiring about database,
It is easy to task user name validity checking below；
(3) task white list white tasks are obtained：Task white list list is obtained by inquiring about database, after being easy to
The task validity checking in face, the task in white list skip over；
(4) the task task set being currently running are obtained：Set of tasks is obtained by hadoop api；
(5) judge whether task set are empty, be then to perform step (6), otherwise perform step (8)；
(6) a task task is removed from task set：A task is taken out from task set；
(7) single task legitimacy is checked：The indices of inspection task, judge whether task is illegal, jump to step
Suddenly (5)；
(8) terminate.
The present invention is quasi real time protected the safety of hadoop systems, supervised in real time automatically by the automatic Predicated execution of whole process
The occupation condition of each task of hadoop cluster is controlled, discovery task, which takes resource, will excessively close task automatically.
In addition, as shown in Fig. 2 the step (7) include it is following step by step：
(7.1) initialization task information；
(7.2) user of submission task is judged whether in user's white list, is then to perform step (7.3), is otherwise performed
Step (7.8)；
(7.3) judge whether the task is white list task, if it is abandons checking, perform step (7.9), otherwise hold
Row step (7.4)；
(7.4) judge that the map numbers of inspection task whether beyond regulation, if it is perform step (7.8), otherwise perform step
Suddenly (7.5)；
(7.5) judge that the reduce numbers of inspection task whether beyond regulation, if it is perform step (7.8), otherwise held
Row step (7.6)；
(7.6) judge that the output quantity of inspection task whether beyond regulation, if it is performs step (7.8), otherwise performed
Step (7.7)；
(7.7) judge whether the run time of inspection task exceeds defined length, if it is perform step (7.8),
Otherwise step (7.9) is performed；
(7.8) task is killed, mail is sent to user：Judge that the task is illegal, kill the task and give submission the task
User send circular mail；
(7.9) individual task inspection terminates：Individual task inspection terminates, the partial information data-in storehouse of task.
In addition, in the step (7.4), map numbers are 20000.
In addition, in the step (7.5), reduce numbers are 1000.
In addition, in the step (7.6), the output quantity of task is 5T.
In addition, in the step (7.7), it is specified that length be 10 hours.
Can be with it will appreciated by the skilled person that realizing that all or part of step in above-described embodiment method is
The hardware of correlation is instructed to complete by program, described program can be stored in a computer read/write memory medium,
The program upon execution, includes each step of above-described embodiment method, and described storage medium can be：ROM/RAM, magnetic
Dish, CD, storage card etc..Therefore, corresponding with the method for the present invention, the present invention also supervises including a kind of hadoop tasks simultaneously
Control system, the system are generally represented in the form of the functional module corresponding with each step of method.Use the system bag of this method
Include：
Initialization module, it is configured to perform the operation of initialization；
User's white list module is obtained, it is configured to obtain the name on account list having permission by inquiring about database, just
Task user name validity checking later；
Acquisition task white list module, its configure come by inquire about database obtain task white list list, be easy to behind
Task validity checking, the task in white list skips over；
The task module being currently running is obtained, it, which is configured, to obtain set of tasks by hadoop api；
Judge module, it is configured to judge whether task set are empty, is then to perform to remove module, is otherwise terminated；
Module is removed, it, which is configured, to remove a task task from task set：One is taken out from task set to appoint
Business；
Module is checked, it is configured to check single task legitimacy：Whether the indices of inspection task, judge task
Illegally, judge module is jumped to.
The detailed process of implementation of the present invention is as follows：
1st, basic framework (Framework)
Hadoop、mysql
2nd, each modular structure (Modules)
Nothing
3rd, agreement and specification
● pass through the task run information for calling hadoop api to obtain.
● the failure of task run information is obtained in running, expression task has been not present, and abandons to the task
Check
● using the output information of data-base recording task
4th, format parameter (Params)
● input (in)：
Hadoop task list information
● output (out)：
Send the mail of kill task, task run information
5th, coding realizes (Programs)
●Coding：Program is checked according to the form and protocol realization task of agreement
6th, structure deployment (Build)
●Ubuntu12.04.4
●Mysql5.5
●Shell
It is described above, be only presently preferred embodiments of the present invention, any formal limitation not made to the present invention, it is every according to
Any simple modification, equivalent change and modification made according to the technical spirit of the present invention to above example, still belong to the present invention
The protection domain of technical scheme.
Claims (7)
- A kind of 1. hadoop Mission Monitors method, it is characterised in that：This method comprises the following steps：(1) start, perform the operation of initialization；(2) user white list white users are obtained：The name on account list having permission is obtained by inquiring about database, is easy to Task user name validity checking below；(3) task white list white tasks are obtained：Task white list list is obtained by inquiring about database, is easy to below Task validity checking, the task in white list skip over；(4) the task task set being currently running are obtained：Set of tasks is obtained by hadoop api；(5) judge whether task set are empty, be then to perform step (6), otherwise perform step (8)；(6) a task task is removed from task set：A task is taken out from task set；(7) single task legitimacy is checked：The indices of inspection task, judge whether task is illegal, jumps to step (5)；(8) terminate.
- 2. hadoop Mission Monitors method according to claim 1, it is characterised in that：The step(7) include it is following step by step：(7.1) initialization task information；(7.2) user of submission task is judged whether in user's white list, is then to perform step (7.3), is otherwise performed step (7.8)；(7.3) judge whether the task is white list task, if it is abandons checking, perform step (7.9), otherwise perform step Suddenly (7.4)；(7.4) judge that the map numbers of inspection task whether beyond regulation, if it is perform step (7.8), otherwise perform step (7.5)；(7.5) judge that the reduce numbers of inspection task whether beyond regulation, if it is perform step (7.8), otherwise perform step Suddenly (7.6)；(7.6) judge that the output quantity of inspection task whether beyond regulation, if it is performs step (7.8), otherwise performs step (7.7)；(7.7) judge that the run time of inspection task whether beyond defined length, if it is performs step (7.8), otherwise Perform step (7.9)；(7.8) task is killed, mail is sent to user：Judge that the task is illegal, kill the task and to the use for submitting the task Family sends circular mail；(7.9) individual task inspection terminates：Individual task inspection terminates, the partial information data-in storehouse of task.
- 3. hadoop Mission Monitors method according to claim 2, it is characterised in that：In the step (7.4), map numbers For 20000.
- 4. hadoop Mission Monitors method according to claim 3, it is characterised in that：In the step (7.5), reduce Number is 1000.
- 5. hadoop Mission Monitors method according to claim 4, it is characterised in that：In the step (7.6), task Output quantity is 5T.
- 6. hadoop Mission Monitors method according to claim 5, it is characterised in that：In the step (7.7), it is specified that Length is 10 hours.
- A kind of 7. hadoop Mission Monitors system, it is characterised in that：The system includes：Initialization module, it is configured to perform the operation of initialization；User's white list module is obtained, it is configured to obtain the name on account list having permission by inquiring about database, after being easy to The task user name validity checking in face；Acquisition task white list module, it is configured to obtain task white list list by inquiring about database, is easy to times below Business validity checking, the task in white list skip over；The task module being currently running is obtained, it, which is configured, to obtain set of tasks by hadoop api；Judge module, it is configured to judge whether task set are empty, is then to perform to remove module, is otherwise terminated；Module is removed, it, which is configured, to remove a task task from task set：A task is taken out from task set；Module is checked, it is configured to check single task legitimacy：The indices of inspection task, judge whether task is non- Method, jump to judge module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710984747.9A CN107766138A (en) | 2017-10-20 | 2017-10-20 | A kind of hadoop Mission Monitors method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710984747.9A CN107766138A (en) | 2017-10-20 | 2017-10-20 | A kind of hadoop Mission Monitors method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107766138A true CN107766138A (en) | 2018-03-06 |
Family
ID=61269766
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710984747.9A Pending CN107766138A (en) | 2017-10-20 | 2017-10-20 | A kind of hadoop Mission Monitors method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107766138A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521101A (en) * | 2011-12-08 | 2012-06-27 | 曙光信息产业(北京)有限公司 | Illegal job monitor method based on process scanning |
| CN102567086A (en) * | 2010-12-30 | 2012-07-11 | ***通信集团公司 | Task scheduling method, equipment and system |
| CN103761146A (en) * | 2014-01-06 | 2014-04-30 | 浪潮电子信息产业股份有限公司 | Method for dynamically setting quantities of slots for MapReduce |
| CN106201681A (en) * | 2016-06-30 | 2016-12-07 | 湖南大学 | Task scheduling algorithm based on pre-release the Resources list under Hadoop platform |
| CN106326137A (en) * | 2015-07-02 | 2017-01-11 | 腾讯科技（深圳）有限公司 | Memory optimization method and apparatus, and user equipment |
-
2017
- 2017-10-20 CN CN201710984747.9A patent/CN107766138A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102567086A (en) * | 2010-12-30 | 2012-07-11 | ***通信集团公司 | Task scheduling method, equipment and system |
| CN102521101A (en) * | 2011-12-08 | 2012-06-27 | 曙光信息产业(北京)有限公司 | Illegal job monitor method based on process scanning |
| CN103761146A (en) * | 2014-01-06 | 2014-04-30 | 浪潮电子信息产业股份有限公司 | Method for dynamically setting quantities of slots for MapReduce |
| CN106326137A (en) * | 2015-07-02 | 2017-01-11 | 腾讯科技（深圳）有限公司 | Memory optimization method and apparatus, and user equipment |
| CN106201681A (en) * | 2016-06-30 | 2016-12-07 | 湖南大学 | Task scheduling algorithm based on pre-release the Resources list under Hadoop platform |
Non-Patent Citations (1)
| Title |
|---|
| 绝艳纵天: "hadoop中的map和reduce的数量设置问题", 《HTTPS://MY.OSCHINA.NET/CHANTHON/BLOG/150500》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108536461A (en) | Resource regeneration method, device, terminal and storage medium | |
| CN104427104B (en) | A kind of information processing method and electronic equipment | |
| CN109684370A (en) | Daily record data processing method, system, equipment and storage medium | |
| CN104821954B (en) | A kind of cross-platform remote procedure calling (PRC) method | |
| EP2994828B1 (en) | Apps store with integrated test support | |
| CN104158826B (en) | A kind of dispatching method and its system of the migration of Virtual machine | |
| CN105871587A (en) | Log uploading method and device | |
| CN107609150A (en) | A kind of interactive network reptile creation method chosen based on page elements and system | |
| CN113269547B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN106685894B (en) | Risk identification method, device and system | |
| CN106557470A (en) | data extraction method and device | |
| CN106169042A (en) | The method and device of administration authority | |
| CN106484520A (en) | A kind of intelligent dispatching method based on data blood relationship and system | |
| CN106383764A (en) | Data acquisition method and device | |
| CN107295082A (en) | Running software processing method, apparatus and system | |
| CN109493224A (en) | A kind of block chain picture social intercourse system | |
| CN113961245A (en) | Security protection system, method and medium based on micro-service application | |
| CN117009483A (en) | Method, device and equipment for generating question-answering service and readable storage medium | |
| CN107147711A (en) | Application service update method and device | |
| CN109388503A (en) | A kind of event-handling method and device | |
| US9762430B2 (en) | Method and apparatus for clearing configuration command in communication equipment | |
| US20160371133A1 (en) | System and method for automatic error classification in integration systems | |
| CN110532773A (en) | Malicious access Activity recognition method, data processing method, device and equipment | |
| CN107766138A (en) | A kind of hadoop Mission Monitors method and system | |
| CN112463266A (en) | Execution policy generation method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180306 |