CN111125444A

CN111125444A - Big data task scheduling management method, device, equipment and storage medium

Info

Publication number: CN111125444A
Application number: CN201911261561.6A
Authority: CN
Inventors: 卢显锋
Original assignee: Ping An Property and Casualty Insurance Company of China Ltd
Current assignee: Ping An Property and Casualty Insurance Company of China Ltd
Priority date: 2019-12-10
Filing date: 2019-12-10
Publication date: 2020-05-08

Abstract

The invention relates to the technical field of big data, and discloses a big data task scheduling management method, which comprises the following steps: reading a task scheduling configuration file; extracting running information of all tasks from the task scheduling configuration file; determining a dependency relationship among tasks based on the running information; integrating the tasks into a task chain with a front-back incidence relation based on the dependency relation; based on the task chain, drawing and displaying a tree-shaped flow chart corresponding to the task; and performing visual scheduling management on each task based on the tree-shaped flow chart. The invention also discloses a big data task scheduling management device, equipment and a computer readable storage medium. The invention realizes unified scheduling management of a plurality of big data tasks and improves the processing timeliness of large-batch data.

Description

Big data task scheduling management method, device, equipment and storage medium

Technical Field

The present invention relates to the field of big data technologies, and in particular, to a big data task scheduling management method, apparatus, device, and computer-readable storage medium.

Background

In the prior art, a big data system usually uses a professional tool (e.g., oozie) to perform multitask scheduling, and both a test environment and a production environment can only be remotely checked and operated through a bastion machine, and only a single task can be managed each time, for example, checking the specific configuration of the task, the scheduling period, the running time of a script under each application and the running condition of a pre-dependency, all needs to be manually searched in a corresponding task popup by operation and maintenance personnel, which is time-consuming and labor-consuming.

Generally, thousands of tasks run in a big data system at the same time, different tasks usually have a certain incidence relation, and because only a single task can be managed at each time, the direct incidence relation of each task cannot be checked, only one task can be inquired, and automatic association cannot be performed to check the overall interdependency condition, so that batch tasks cannot be uniformly scheduled, production environment management is not facilitated, and task management efficiency is low.

Disclosure of Invention

The invention mainly aims to provide a big data task scheduling management method, a device, equipment and a computer readable storage medium, and aims to solve the technical problem that the batch tasks cannot be uniformly scheduled and managed in the existing big data task scheduling management technology, so that the processing timeliness of large-batch data is low.

In order to achieve the above object, the present invention provides a big data task scheduling management method, which includes the following steps:

reading a task scheduling configuration file;

extracting running information of all tasks from the task scheduling configuration file;

determining a dependency relationship among tasks based on the operation information;

integrating the tasks into a task chain with a front-back incidence relation based on the dependency relation;

based on the task chain, drawing and displaying a tree-shaped flow chart corresponding to the task;

and carrying out visual scheduling management on each task based on the tree-shaped flow chart.

Optionally, the operation information includes: task name and task association.

Optionally, the determining a dependency relationship between tasks based on the running information includes:

extracting task names and task operation association relations of all tasks from the operation information;

searching the task operation association relation based on the task name to obtain a search result;

if the search result is empty, determining that the current task does not have a dependency relationship with other tasks;

and if the search result is not empty, determining that the current task has a dependency relationship with other tasks, and generating a task pair set with the dependency relationship.

Optionally, the integrating the tasks into a task chain having a context relationship based on the dependency relationship includes:

initializing a task chain, and selecting a task without a dependency relationship as a chain head task of the task chain;

sequentially traversing the task pair set by taking the task at the head of the chain as a current node based on the dependency relationship among the tasks, and taking other tasks with the dependency relationship with the task corresponding to the current node as a next new node corresponding to the current node in the task chain;

and traversing the task pair set in sequence by taking the new node in the task chain as the current node based on the dependency relationship, and taking other tasks with the dependency relationship with the task corresponding to the current node as the next new node corresponding to the current node in the task chain until other tasks with the dependency relationship with the task corresponding to the current node do not exist in the task pair set.

Optionally, the drawing and displaying a tree-shaped flow chart corresponding to task running based on the task chain includes:

acquiring application information corresponding to each task from a front-end database, and setting a task configuration interface corresponding to each task in the task chain;

generating a DIV partition in a preset scheduling management page, and calling a preset drawing script to draw a tree-shaped flow chart corresponding to the running of tasks in the DIV partition based on the task chain;

associating application information and a task configuration interface corresponding to each task in the task chain with corresponding nodes in the tree-shaped flow chart respectively and displaying the application information and the task configuration interface on the scheduling management page;

the task configuration information displayed by the task configuration interface comprises: the method comprises the steps of maximum running time of an application, task starting time, running database information, host resources for task running, periodic frequency for task running and task running script parameters.

Optionally, before the step of performing visual scheduling management on each task based on the tree-shaped flowchart, the method further includes:

collecting log information generated during task operation through a preset buried point;

based on a preset log alarm rule, carrying out alarm judgment on the log information;

if the log information meets the log alarm rule, generating alarm information and storing the alarm information in a preset database;

the visual scheduling management of each task based on the tree-shaped flow chart comprises the following steps:

reading alarm information from the database in real time;

determining and storing the running state of each task in the tree-shaped flow chart according to the alarm information;

dynamically displaying the running state information of each node at each node position in the tree-shaped flow chart, wherein the running state information comprises: start state, run state, error reporting state, delay state, completed state.

Optionally, the performing, based on the tree-shaped flowchart, visual scheduling management on each task includes:

acquiring a task name to be positioned;

determining running information of a corresponding task based on the task name and displaying the running information on the scheduling management page;

receiving a task operation instruction input by a user based on the running information of the task displayed on the scheduling management page;

based on the task operation instruction, controlling a corresponding task to execute corresponding operation, wherein the task operation instruction comprises: a task re-running instruction, a task run-back instruction and a task start-up instruction.

Further, the present invention also provides a big data task scheduling management device, which is applied to big data task scheduling management, and the big data task scheduling management device includes:

the reading module is used for reading the task scheduling configuration file;

the extraction module is used for extracting the running information of all tasks from the task scheduling configuration file;

the determining module is used for determining the dependency relationship among the tasks based on the running information;

the integration module is used for integrating each task into a task chain with a front-back incidence relation based on the dependency relation;

the drawing module is used for drawing and displaying a tree-shaped flow chart corresponding to task operation based on the task chain;

and the management module is used for carrying out visual scheduling management on each task based on the tree-shaped flow chart.

Optionally, the determining module includes:

the extraction unit is used for extracting task names and task association relations of all tasks from the running information;

the searching unit is used for searching the task incidence relation based on the task name to obtain a searching result;

the determining unit is used for determining that the current task and other tasks do not have a dependency relationship if the searching result is empty; and if the search result is not empty, determining that the current task has a dependency relationship with other tasks, and generating a task pair set with the dependency relationship.

Optionally, the integration module comprises:

the initial unit is used for initializing a task chain and selecting a task without dependency relationship as a chain head task of the task chain;

the traversal unit is used for sequentially traversing the task pair set by taking the chain head task as a current node based on the dependency relationship among the tasks, and taking other tasks with the dependency relationship with the task corresponding to the current node as a next new node corresponding to the current node in the task chain; and traversing the task pair set in sequence by taking the new node in the task chain as the current node based on the dependency relationship, and taking other tasks with the dependency relationship with the task corresponding to the current node as the next new node corresponding to the current node in the task chain until other tasks with the dependency relationship with the task corresponding to the current node do not exist in the task pair set.

Optionally, the rendering module includes:

the setting unit is used for setting a task configuration interface corresponding to each task in the task chain based on the application information corresponding to the task;

the scheduling unit is used for generating a task scheduling (DIV) partition in a preset scheduling management page and calling a preset drawing script to draw a tree-shaped flow chart corresponding to the running of a task in the DIV partition based on the task chain;

and the display unit is used for respectively associating the application information and the task configuration interface corresponding to each task in the task chain with the corresponding node in the tree-shaped flow chart and displaying the application information and the task configuration interface on the scheduling management page.

Optionally, the management module includes:

the acquisition unit is used for acquiring the name of the task to be positioned;

the determining unit is used for determining the running information of the corresponding task based on the task name and displaying the running information on the scheduling management page;

the receiving unit is used for receiving a task operation instruction input by a user based on the running information of the task displayed on the scheduling management page;

the control unit is used for controlling the corresponding task to execute corresponding operation based on the task operation instruction, wherein the task operation instruction comprises: a task re-running instruction, a task run-back instruction and a task start-up instruction.

Further, to achieve the above object, the present invention further provides a big data task scheduling management device, where the big data task scheduling management device includes a memory, a processor, and a big data task scheduling management program stored in the memory and executable on the processor, and the big data task scheduling management program, when executed by the processor, implements the steps of the big data task scheduling management method according to any one of the above mentioned methods.

Further, to achieve the above object, the present invention further provides a computer readable storage medium, where a visualized unified management big data task scheduling running program is stored on the computer readable storage medium, and when executed by a processor, the visualized unified management scheduling task running program implements the steps of the big data task scheduling method according to any one of the above items.

The invention uniformly schedules and manages the batch tasks according to the incidence relation among the tasks, checks the running information of the tasks, monitors the running state of the batch tasks in real time, gives early warning to the delayed tasks, reports errors in time to the tasks which fail to run, provides a basis for code optimization and maintenance, and completes the starting, the rerun or the run supplement of the batch tasks by one-key operation. Meanwhile, the running parameters of the tasks can be configured in a user-defined mode, the running states of the tasks are monitored in real time, the processing timeliness of mass data is improved, the operation difficulty of operation and maintenance personnel is reduced, and the operation and maintenance management efficiency is improved.

Drawings

FIG. 1 is a schematic structural diagram of a hardware operating environment of a device according to an embodiment of a big data task scheduling management device of the present invention;

FIG. 2 is a flowchart illustrating a big data task scheduling management method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram illustrating a detailed flow of step S30 in FIG. 2;

FIG. 4 is a schematic diagram illustrating a detailed flow of step S40 in FIG. 2;

FIG. 5 is a schematic view of a detailed flow chart of an embodiment of step S50 in FIG. 2;

FIG. 6 is a schematic structural diagram illustrating an embodiment of a task chain in the big data task scheduling management method according to the present invention;

FIG. 7 is a schematic diagram illustrating a detailed flow of step S60 in FIG. 2;

fig. 8 is a functional block diagram of an embodiment of a big data task scheduling management apparatus according to the present invention.

The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

The invention provides a big data task scheduling management device.

Referring to fig. 1, fig. 1 is a schematic structural diagram of a device hardware operating environment according to an embodiment of the big data task scheduling management device of the present invention.

As shown in fig. 1, the big data task scheduling management device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a memory device separate from the processor 1001 described above.

Those skilled in the art will appreciate that the hardware architecture of the big data task schedule management apparatus shown in fig. 1 does not constitute a limitation of the big data task schedule management apparatus, and may include more or less components than those shown, or combine some components, or a different arrangement of components.

As shown in fig. 1, a memory 1005, which is a kind of computer-readable storage medium, may include therein an operating system, a network communication module, a user interface module, and a big data task scheduling manager. The operating system is a program for managing and controlling the big data task scheduling management equipment and software resources, and supports the running of a network communication module, a user interface module, a big data task scheduling management program and other programs or software; the network communication module is used to manage and control the network interface 1004; the user interface module is used to manage and control the user interface 1003.

In the hardware structure of the big data task scheduling management device shown in fig. 1, the network interface 1004 is mainly used for connecting to a system background and performing data communication with the system background; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; the big data task scheduling management apparatus calls the big data task scheduling management program stored in the memory 1005 through the processor 1001 and performs the operations of the following embodiments of the big data task scheduling management method.

Based on the hardware structure of the big data task scheduling management equipment, the embodiment of the big data task scheduling management method is provided.

Referring to fig. 2, fig. 2 is a flowchart illustrating a big data task scheduling management method according to an embodiment of the present invention. In this embodiment, the big data task scheduling management method includes the following steps:

step S10, reading a task scheduling configuration file;

in this embodiment, the task scheduling configuration file specifically refers to an operation program encapsulating all task operation information, and is configured in advance by a developer according to a certain configuration parameter and stored in the bottom database. And when the system starts to carry out multi-task scheduling, reading the running information of all tasks of the configuration file.

Step S20, extracting the running information of all tasks from the task scheduling configuration file;

the setting of the running information of the task is not limited in this embodiment, and the setting is specifically performed according to actual needs. Optionally, in an embodiment, the task running information includes a task name, a task association relationship, a maximum time length for running the task, a task state, and the like.

(1) Task name

The task names are names of the tasks, different tasks use different task names, and therefore the tasks are distinguished from other tasks, the setting mode of the task names is not limited, and the task names are specifically set according to actual needs. All current running information of the corresponding task can be searched and displayed based on the task name, the running state of the task is judged according to the current running information of the task, and then a user operation instruction is received to complete the next operation. For example, if the current task fails to run, the user may input a "rerun" instruction until the task runs successfully, and then perform subsequent task scheduling and management.

(2) Maximum duration of task operation

The maximum time length of the task operation is the maximum time length required by the task completion, and if the task is not completed in the time length, the task operation is called as a task operation failure, namely the task operation is in error. For example, the maximum running time of the task X is 1 hour, and if the task X is not completed after exceeding 1 hour, it is determined that the task X is running incorrectly, and the task is running unsuccessfully.

(3) Task state

The task state refers to the current running state of the task, and comprises the following steps: not started, in operation, completed. Wherein "completed" is divided into two cases: timely completion and overdue completion. If the task is overdue and not completed, the task is called as failure, namely the task operation is wrong. For example, if the superior task is successfully executed, the current task is triggered to be executed, and the task is judged to be overtime, namely the task is delayed to be executed, from the time when the current task is issued to the time when the current task is executed to the task execution node until the time when the current task is successfully executed exceeds a preset first time length but does not exceed a preset second time length; and if the execution result of the current task is not received after the preset second time length is exceeded, judging that the task operation fails, namely the task operation is in error.

(4) Task associative relations

The task association relationship means that a binding relationship or a mapping relationship exists between tasks, and the setting of the association relationship between tasks is not limited in this embodiment. For example, a task appears in another task in the form of a member variable, or a task appears in an attribute of another task, and an association relationship between the two tasks is said to exist.

In this embodiment, the running information of the task is stored in the underlying database, and is read and displayed to the scheduling management page through the web application.

In the embodiment, if the task running information is stored in the database, the task running information is read and displayed through the web application; and if the task running information is not in the database, the task running information needs to be stored in the database, and then the task running information is read and displayed to a scheduling management page through a web application. For example, when the oozie is used, at an interface of the oozie, a data buried point statistical method is used for recording log information of buried points, and the log information is synchronously recorded to a database. And for the normally running tasks, the running information of the tasks can be directly read and displayed by the web application, and the tasks which fail to run or are not successfully submitted are recorded by the preset buried points and then displayed to the user, so that the user is waited to send out a new operation instruction, and further, the subsequent scheduling and management of the tasks are completed.

Step S30, determining the dependency relationship among tasks based on the operation information;

in this embodiment, the dependency relationship may also be referred to as a "logical relationship". In the two tasks of X, Y, the change in the state of task X may affect task Y, or if the object of task X is to complete some operation, it must use some operation of the object of task Y to help it. For example, class a uses the object of class B as a parameter of an operation or class a uses the object of class B as its data member, but in any case, class B exists in the form of a local variable in class a, class a has a local variable of type B, so to say class a depends on class B, and class a and class B have a dependency relationship.

In this embodiment, the dependency relationship between tasks may be determined according to the obtained running information of the tasks, where the running information for determining the dependency relationship between the tasks is not limited, and for example, the running information may be a pre-defined parameter, where the parameter describes a plurality of tasks having a dependency relationship, and then the parameter may determine that the dependency relationship exists between corresponding tasks.

In an embodiment, the dependency relationship between tasks is preferably determined by task name information and task association relationship information in the task running information.

For example, the dependency relationship between tasks may include four cases:

FS (completion-start) relationship, task X must be completed before task Y can start, say: task X 'input', task Y 'print';

SS (start-start) relationship, if task X is not started, then task Y cannot start either, say: in the restaurant, a task1 is used for serving, and a task2 is used for eating the dish, so that the dish can be served continuously;

FF (completion-completion) relation, the completion time of task X determines the completion time of task Y, for example, task1 "lay wire", task2 "check line". before task1 is not completed, task2 cannot be completed;

SF (start-finish) relationship, the start time of task X determines the finish time of task Y, for example, task X "opens the screen", task Y "meeting place arrangement".

It should be noted that, in this embodiment, the dependency relationship between the tasks is unidirectional, a plurality of tasks have a sequence when running, and the latter must depend on the former. For example, the upper task1, the current task2, the lower task 3; if and only if the upper task1 successfully executes, the current task2 can be triggered and executed; similarly, the subordinate task3 may be triggered and executed if and only if the current task2 is successfully executed. This sequence must be: the upper level tasks 1- -current task2- -lower level tasks 3.

Step S40, integrating each task into a task chain with a front-back incidence relation based on the dependency relation;

in this embodiment, the task chain refers to a set of tasks composed of directed tasks having a dependency relationship, and each task has a certain context relationship.

In this embodiment, a starting point task and a subordinate task of the starting point task are determined from the tasks based on the dependency relationship between the tasks, execution of the starting point task is started, after the starting point task is executed, the subordinate task of the starting point task is defined as a current task, the current task is loaded, then a superior task and a subordinate task of the current task are determined according to the task dependency relationship, and according to the method, the tasks are sequentially arranged and arranged into a task chain with a directed relationship. For example, the number of the tasks is N, the starting point task is found, the task1 is executed, the tasks 2, 3 and 4 which have a dependency relationship with the task1 are started to be executed, the tasks 2, 3 and 4 are used as the current tasks and executed, the tasks which have a dependency relationship with the current tasks are sequentially found according to the method and are sorted until all the tasks are sorted into the task chain and are finished.

In this embodiment, dependency relationship information between tasks can be clearly obtained through a task chain.

Step S50, based on the task chain, drawing and displaying a tree-shaped flow chart corresponding to the task;

in this embodiment, the tree-shaped flowchart refers to a task processing flowchart similar to a tree structure.

And simultaneously setting a task configuration interface corresponding to each task in the task chain through application information corresponding to each task acquired from a front-end database, and calling a preset drawing script to draw a tree-shaped flow chart based on the task chain on a scheduling management page.

In this embodiment, at the corresponding node of the tree-shaped flowchart, the task having the dependency relationship in the corresponding task chain is corresponding, and the application information corresponding to each task can be directly viewed through the tree-shaped flowchart. Meanwhile, the running information such as the running state, the cycle and the like of each task can be checked through one-key operation; the running time of a plurality of tasks is monitored in real time, delayed tasks or tasks with running failure are alarmed or reported by mistake in time, and further the processing timeliness of mass data can be improved.

And step S60, performing visual scheduling management on each task based on the tree-shaped flow chart.

In this embodiment, performing visual scheduling management on each task means displaying each task in the tree-shaped flowchart and dependency relationship information and operation information between the tasks in a visual manner through a scheduling management page, so that a user can determine an execution result of each task, and according to the execution result, managing task scheduling.

In the embodiment, on the scheduling management page, the tasks are scheduled and managed through the tree-shaped flow chart, specific operation information of each task, including the operation state, the operation time and the like, can be checked through one-key operation, the operation condition of each task is monitored in real time, the delayed tasks and the tasks with operation failure can be timely alarmed and error reported, meanwhile, the tasks can be uniformly scheduled and managed, and the processing timeliness of mass data is obviously improved.

Referring to fig. 3, fig. 3 is a detailed flowchart of step S30 in fig. 2. In this embodiment, the step S30 includes:

step S301, extracting task names and task operation association relations of all tasks from the operation information;

step S302, based on the task name, searching the task operation association relation to obtain a search result;

in this embodiment, the task association relationship is another expression manner of the dependency relationship between tasks. The task association relation is searched through the task name, and two search results can be obtained according to whether the current task running state is influenced by the running results of other tasks: 1. other tasks having the association relation with the current task do not exist, namely the search result is null; 2. there are other tasks that have an association with the current task, i.e. the search result is not empty.

Step S303, if the search result is empty, determining that the current task and other tasks do not have a dependency relationship;

and step S304, if the search result is not empty, determining that the current task has a dependency relationship with other tasks, and generating a task pair set with the dependency relationship.

In this embodiment, the finding result being null means that there is no other task having a relationship with the current task, and the running state of the current task is not affected by the running results of the other tasks in the task chain. And searching for the task having a dependency relationship with the current task based on the task name of the current task, wherein if the running result of the task is not influenced by the running state of any other task, the current task does not have any dependency relationship with other tasks, namely the searching result is empty. For example, in a task chain, the subordinate tasks of the current task a are task B and task C, the subordinate task of task B is task F, and the subordinate task of task C is task E. At this time, the running state of task K is affected by the running result of task E, the running state of task F is affected by the running result of task B, the running states of task B and task C are affected by the running result of task a, and the running state of current task a is not affected by any other task in the task chain. So the lookup result is empty. Further, if the search result is null, it indicates that the current task is the head task of the task chain.

In this embodiment, as mentioned above, the dependency relationship between the tasks refers to that the running result of the current task affects the running state of other tasks, and the two tasks are said to have a dependency relationship therebetween. For example, the following steps: a current task1 and a subordinate task 2; whether the current task1 successfully runs or not can affect the running of the task2 of the lower level, so that the task1 and the task2 have a dependency relationship, the association relationship between the tasks is searched based on the task2, the task pair sets task1 and task2 with the dependency relationship are obtained, and the search result is non-empty.

In this embodiment, a task pair refers to a combination of two tasks having a dependency relationship, and this combination is called a task pair. For example, task a, task B, task C, task D, and task E are five tasks, and if task a is the current task, its subordinate tasks are task B and task C; taking the task B as a current task, and taking the lower-level tasks of the task B as a task D and a task E, wherein the task A and the task B are a task pair; similarly, task a and task C, task B and task E, and task B and task D are all task pairs. In short, two tasks with dependencies are called task pairs.

In this embodiment, the task pair set refers to a set of all task pairs having a dependency relationship. For example, there are N tasks in a total task set, and it is assumed that task1 is an initial task, and subordinate tasks of task1 include task2, task3, and task 4; taking the task2 as the current task, wherein the lower-level task of the task2 comprises a task 5 and a task 6; taking the task3 as the current task, taking the lower-level tasks of the task3 as tasks 7, 8, and sorting the N tasks according to the rule until the task which has no dependency relationship with the current task in the set is finished, wherein the task pair comprises: task1 and task2, task1 and task3, task1 and task 4, task2 and task 5, task2 and task 6, task3 and task 7, and all these task pairs, form a set, called task pair set.

Referring to fig. 4, fig. 4 is a detailed flowchart of step S40 in fig. 2. In this embodiment, the step S40 includes:

step S401, initializing a task chain, and selecting a task without dependency relationship as a chain head task of the task chain;

in this embodiment, the chain head task of the task chain refers to a starting point task that has no dependency relationship with other tasks.

In this embodiment, the execution result of the head-of-chain task does not depend on any other task in the task chain, and the execution result of the head-of-chain task (current node) determines whether the lower-level task (next new node) can start executing. For example, the lower level task is triggered to execute only if the chain head task is executed successfully. However, whether the lower task can be successfully executed is not related to the chain head task. For example, the following steps: the current task2 can be triggered to execute only by the upper task1 in a task chain, but whether the execution result is successful or not is irrelevant to task 1.

Step S402, the task pair set is traversed in sequence by taking the task at the head of the chain as the current node based on the dependency relationship among the tasks, and other tasks with the dependency relationship with the task corresponding to the current node are taken as the next new node corresponding to the current node in the task chain;

and S403, sequentially traversing the task pair set by taking the new node in the task chain as the current node based on the dependency relationship, and taking other tasks with dependency relationships with the task corresponding to the current node as the next new node corresponding to the current node in the task chain until other tasks with dependency relationships with the task corresponding to the current node do not exist in the task pair set.

In this embodiment, Traversal (Traversal) refers to using a rule to sequentially access each node in the tree structure diagram once and only once, and the operation performed by accessing the node depends on a specific application problem.

In this embodiment, the task information corresponding to the current node is determined by sequentially traversing the task pair sets, and the next new node is determined based on the dependency relationship between tasks.

In this embodiment, if a task needs to be completed in several stages or steps, a transfer point at the beginning of a stage or step is called a node.

In this embodiment, the nodes are connected one by one in sequence, and all tasks are arranged into a directed task chain having a front-back association relationship. For example, a task set X has N tasks in total, and first finds out a chain head task (starting point task) task1 therein, and takes the task1 as a current node; finding out all task pairs according to the dependency relationship among the tasks, namely finding out a task2, a task3 and a task 4 which have the dependency relationship with the task1, and respectively taking the task2, the task3 and the task 4 as a next new node; and taking the new node as the current node, finding out tasks with the dependency relationship with the current node based on the dependency relationship, sorting the task pairs with the dependency relationship into a task pair set until other tasks with the dependency relationship with the current node do not exist in the set X, and connecting all the nodes to form a chain, namely a task chain with a front-back incidence relationship.

Referring to fig. 5, fig. 5 is a detailed flowchart of step S50 in fig. 2. In this embodiment, the step S50 includes:

step S501, acquiring application information corresponding to each task from a front-end database, and setting a task configuration interface corresponding to each task in the task chain;

in this embodiment, the application information refers to all running information of the task. And setting a task configuration interface corresponding to each task in the task chain according to the running information of the task. The most useful and most common part in the application configuration file is actually a custom configuration part in the configuration interface, and user-defined paragraphs exist in the configuration file, and configuration data can be arranged in the custom paragraphs.

In this embodiment, the user may arrange configuration data in the customized paragraph, so that the user can schedule a management task according to the requirement. For example, if a user wants to record task execution results in different time periods in a differentiated manner, the operation parameters of the task may be configured in the task configuration interface, specifically, the name of the task, the maximum time length of application operation, the start time of the task, the information of the operation database, the host resources for task operation, and the like may be configured. Because the same task is executed again after every other operation period, furthermore, an operation time can be created according to the operation period of the task and marked as a task label. According to the task name and the task label, the distinguished record of the task execution results in different time periods is realized, and the scheduling management page is convenient to use by operation and maintenance.

Step S502, generating a DIV partition in a preset scheduling management page, and calling a preset drawing script to draw a tree-shaped flow chart corresponding to the task operation in the DIV partition based on the task chain;

in this embodiment, the DIV is a positioning technique in the cascading style sheet, and is called divsion in full, that is, the partition; sometimes it can be called layer, DIV is also called integer division in programming, i.e. integer of quotient; DIV elements are elements used to provide structure and context for the content of large blocks (block-level) within an HTML (an application under the standard generalized markup language) document, and may define partitions or sections within the document.

Step S503, associating the application information and the task configuration interface corresponding to each task in the task chain with the corresponding node in the tree-shaped flow chart respectively and displaying the application information and the task configuration interface on the scheduling management page;

in this embodiment, the application information refers to program information for smoothly starting an execution program corresponding to the current task. Determining the upper and lower level tasks in the task chain according to the dependency relationship among the tasks, corresponding the chain head task in the task chain to the first node of the tree-shaped flow chart, finding out the task with the dependency relationship with the chain head task, then corresponding the current task as a new node to the second node in the tree-shaped flow chart, and sequentially corresponding all the tasks in the task chain to the nodes in the tree-shaped flow chart, wherein the running information of all the tasks is displayed on the tree-shaped flow chart as shown in fig. 6.

Optionally, the task configuration information displayed by the task configuration interface includes: the method comprises the steps of maximum running time of an application, task starting time, running database information, host resources for task running, periodic frequency for task running and task running script parameters.

Optionally, the running database information includes: connection address, user, password.

Optionally, the task start time is: and starting the task in the task running period by each task.

Optionally, the running cycle of the task includes: hour, day, week, month.

Optionally, the running database information refers to application data information required by task running.

Referring to fig. 7, fig. 7 is a detailed flowchart of step S60 in fig. 2. In this embodiment, the step S60 includes:

step S601, acquiring a task name to be positioned;

step S602, determining running information of a corresponding task based on the task name and displaying the running information on the scheduling management page;

in this embodiment, the task to be positioned refers to a task for which task operation information is currently to be acquired. Determining a current task from a scheduling management page, monitoring the running state of the task in real time based on the running information of the task, and judging the execution result of the task according to the running state. For example, if the current task is unchanged, the state information of the current task is calibrated to be 'not started'; if the current task is being executed and a result is not obtained, the state information 'running' of the current task is calibrated; and if the current task is successfully executed, updating the state information of the current task in the database to be 'finished' according to the execution result, and the like.

Step S603, receiving a task operation instruction input by a user based on the running information of the task displayed on the scheduling management page;

step S604, based on the task operation instruction, controlling a corresponding task to execute a corresponding operation, where the task operation instruction includes: a task re-running instruction, a task run-back instruction and a task start-up instruction.

(1) Task re-run instruction

Task rerun refers to that when some data in a task is wrong or missing, the task fails to run, and when a correct running result is obtained, the task needs to be rerun, in other words, a rerun task is a task rerun. For example, when a data collection of buried points is wrong or a data collection of buried points is wrong, or when historical data is artificially modified in a database, tasks using the data source, data models depending on the tasks, and partitions from time when the tasks are in problem need to be rerun. The instruction sent by the user to rerun the task is called a task rerun instruction.

(2) Task run-back instruction

When a timing task is suddenly stopped after being executed for a period of time and cannot be continuously operated, the task is required to be operated again to obtain the operation result of the task, and the task is called to run again. Generally, there are several situations when the running of the timed task is suddenly stopped, for example, when the configuration of the timed task is in problem or the database connection is over time, for example, when the thread is dead locked or blocked. When the task stops running, in order to enable the task to continue running, the operation sent by the user is called a task run-back instruction.

(3) Task initiating instruction

Task starting refers to task starting, and an instruction sent for starting the task to run is called a task starting instruction.

In this embodiment, the running state of the current task is determined based on the task running state of the scheduling management page, and further, an operation instruction input by a user is received, and a next action is executed based on the operation instruction.

The running state and the cycle of the batch tasks are checked clearly, a plurality of tasks are managed through batch scheduling, and the processing timeliness of the batch data is obviously improved.

In this embodiment, when the user performs the custom configuration on the task through the scheduling management page, the task running time may be further configured. Determining a current task and a lower task according to the dependency relationship between the tasks, and sending the current task to a task execution node for execution if the upper task is successfully executed based on the execution result of the upper task; if the waiting time exceeds a first preset time, judging that the current task is delayed, and sending out delay alarm information; if the time length exceeds the second preset time length, the current task is still not executed successfully, the current task is judged to be executed wrongly, and error reporting is carried out, so that error reporting and task delay alarming are effectively carried out.

For example, when the waiting time exceeds the preset time, the current task1 issues a delay alarm, and the alarm information may include a task name, an alarm type, and the like. The user timely recovers the fault according to the alarm information, and the scheduling management server obtains the execution result of the superior task again according to the preset rule after receiving the instruction sent by the user, so as to execute the subsequent task scheduling and management.

For another example, the current task is task2, and when the time for executing the task issuing execution node exceeds the preset time, the task fails to run incorrectly, and the error reporting alarm information sent may include a task name, an alarm type, and the like. And the user timely recovers the fault according to the alarm information, and the scheduling management server sends the current task to the task execution node again for execution after receiving the instruction sent by the user, so as to perform subsequent task scheduling and management.

In the embodiment, both the delayed alarm information and the error report belong to alarms.

By effectively reporting errors and delaying task early warning for operation failure or operation delay tasks, the processing timeliness of mass data is obviously improved, the operation difficulty of operation and maintenance personnel is reduced, and the operation and maintenance management efficiency is improved.

Referring to fig. 8, fig. 8 is a functional module diagram of an embodiment of a big data task scheduling management device according to the present invention. In this embodiment, the big data task scheduling management apparatus includes:

a reading module 10, configured to read a task scheduling configuration file;

an extracting module 20, configured to extract operation information of all tasks from the task scheduling configuration file;

a determining module 30, configured to determine a dependency relationship between tasks based on the operation information;

an integration module 40, configured to integrate each task into a task chain having a context relationship based on the dependency relationship;

the drawing module 50 is used for drawing and displaying a tree-shaped flow chart corresponding to the task operation based on the task chain;

and a management module 60, configured to perform visual scheduling management on each task based on the tree-shaped flowchart.

In this embodiment, a task scheduling configuration file is read, and operation information of all tasks is extracted from the task scheduling configuration file, where the task operation information is a task name and a task association relationship; determining the dependency relationship among the tasks based on the task names and the task association relationship, then integrating the tasks into a task chain with a front-back association relationship based on the dependency relationship, drawing and displaying a tree-shaped flow chart corresponding to task operation by adopting the task chain, and carrying out visual scheduling management on the tasks based on the tree-shaped flow chart.

In the embodiment, a conventional task scheduling mode that multitask scheduling testing is performed based on an oozie tool and a production environment can only be remotely checked and operated through a bastion machine and only a single task can be managed each time is not adopted, but batch scheduling management is performed on batch tasks based on big data tasks, so that the processing timeliness of large-batch data is improved by uniformly scheduling the batch tasks.

According to the method for scheduling and managing the large-data batch tasks, the running information of the batch tasks can be checked at a glance according to the incidence relation among the tasks, the running states of the batch tasks are monitored in real time, the batch tasks are scheduled and managed in a unified mode, the task running parameter function is configured in a user-defined mode, meanwhile, the operation difficulty of operation and maintenance personnel is reduced, and the operation and maintenance management efficiency is improved greatly.

Based on the same description content as the embodiment of the big data task scheduling management method of the present invention, the embodiment of the big data task scheduling management apparatus will not be described in detail in this embodiment.

The invention also provides a computer readable storage medium.

In this embodiment, the computer-readable storage medium stores a big data task scheduling management program, and the big data task scheduling management program, when executed by a processor, implements the steps of the big data task scheduling management method according to any of the embodiments described above. The method for implementing the big data task scheduling management program when executed by the processor may refer to each embodiment of the big data task scheduling management method of the present invention, and therefore, redundant description is not repeated.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM), and includes instructions for causing a terminal (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.

The present invention is described in connection with the accompanying drawings, but the present invention is not limited to the above embodiments, which are only illustrative and not restrictive, and those skilled in the art can make various changes without departing from the spirit and scope of the invention as defined by the appended claims, and all changes that come within the meaning and range of equivalency of the specification and drawings that are obvious from the description and the attached claims are intended to be embraced therein.

Claims

1. A big data task scheduling management method is characterized by comprising the following steps:

reading a task scheduling configuration file;

2. The big data task scheduling management method of claim 1, wherein the running information comprises: task name and task association.

3. The big data task scheduling management method according to claim 2, wherein the determining the dependency relationship between tasks based on the running information comprises:

extracting task names and task association relations of all tasks from the running information;

based on the task name, searching the task association relation to obtain a search result;

4. The big data task scheduling management method according to claim 3, wherein the integrating tasks into a task chain having a context relationship based on the dependency relationship comprises:

5. The big data task scheduling management method according to claim 4, wherein the drawing and displaying a tree-shaped flow chart corresponding to task running based on the task chain comprises:

6. The big data task scheduling management method according to claim 1, wherein before the step of performing visual scheduling management on each task based on the tree-shaped flowchart, the method further comprises:

and if the log information meets the log alarm rule, generating alarm information and storing the alarm information in a preset database.

7. The big data task scheduling management method according to claim 1, wherein the performing visual scheduling management on each task based on the tree-shaped flowchart comprises:

acquiring a task name to be positioned;

8. A big data task scheduling management apparatus, wherein the big data task scheduling apparatus comprises:

the reading module is used for reading the task scheduling configuration file;

9. A big data task scheduling management device, characterized in that the big data task scheduling management device comprises a memory, a processor and a big data task scheduling management program stored on the memory and executable on the processor, and when executed by the processor, the big data task scheduling management program implements the steps of the big data task scheduling management method according to any one of claims 1 to 7.

10. A computer readable storage medium, having stored thereon a big data task scheduling manager, which when executed by a processor implements the steps of the big data task scheduling management method according to any of claims 1-7.