CN109814991A - A kind of data administer in task management method and device - Google Patents
A kind of data administer in task management method and device Download PDFInfo
- Publication number
- CN109814991A CN109814991A CN201811594698.9A CN201811594698A CN109814991A CN 109814991 A CN109814991 A CN 109814991A CN 201811594698 A CN201811594698 A CN 201811594698A CN 109814991 A CN109814991 A CN 109814991A
- Authority
- CN
- China
- Prior art keywords
- task
- data
- operator
- management
- nexus
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application discloses the task management methods and device in a kind of improvement of data；Above-mentioned task management method, comprising: the initial data that building is administered using data is the task nexus figure of starting point, wherein task nexus figure is directed acyclic graph；Task based access control relational graph provides the task management interface for supporting visualized operation；In response to the operation detected in task management interface, data management task is managed.
Description
Technical field
This application involves but be not limited to field of computer technology, espespecially a kind of data administer in task management method and dress
It sets.
Background technique
With the development of big data technology and application, promoting traditional industries (such as public security, finance, industry) to need magnanimity
Business datum is migrated to big data platform, and in this process, the process converted, cleaned to data, referred to as data are administered.
However, due to lacking unified standard, the problems such as initial data is lack of standardization, cause data to be administered difficult.Currently, data
Improvement is primarily present following problems: data providing different style, and multi-source data usually requires effectively use by fusion；
Most of initial data it is second-rate, there is a large amount of wrong row, wrongly written character, ambiguity, need the problems such as converting；For quality compared with
High initial data needs to carry out some extra process, for example, multiple and different values of same field, it usually needs deposit respectively
Storage just can guarantee subsequent use needs；The data of many data sources are all sustainable growths, need a reliable, stable stream
Formula processing or cyclical process process.
Summary of the invention
The embodiment of the present application provide a kind of data administer in task management method and device, realize that flexible management data are controlled
Reason task administers demand so as to cope with complicated data, promotes data governance efficiency.
On the one hand, the embodiment of the present application provide a kind of data administer in task management method, comprising: building is controlled with data
The initial data of reason is the task nexus figure of starting point；Task based access control relational graph provides the task management for supporting visualized operation
Interface；In response to the operation detected in task management interface, data management task is managed；Wherein, task nexus figure is to have
To acyclic figure.
On the other hand, the embodiment of the present application provide a kind of data administer in task management device, comprising: task nexus figure
Module is constructed, suitable for constructing the initial data administered using data as the task nexus figure of starting point；Interface provides module, is suitable for base
In task nexus figure, the task management interface for supporting visualized operation is provided；Task management module is adapted in task pipe
The operation detected on reason interface, manages data management task；Wherein, task nexus figure is directed acyclic graph.
On the other hand, the embodiment of the present application provides a kind of terminal device, comprising: display unit, input unit, memory and
Processor；The display unit connects the processor, is suitable for display task management interface；The input unit connects the place
Manage device, the operation being adapted to detect in the task management interface；The memory is suitable for the task management in storing data improvement
The step of program, the task management program realizes above-mentioned task management method when being executed by the processor.
On the other hand, the embodiment of the present application provides a kind of computer-readable medium, the task pipe being stored in data improvement
The step of reason program, the task management program realizes above-mentioned task management method when being executed by processor.
In the embodiment of the present application, task nexus figure is constructed by starting point of initial data, and provide support visualized operation
Task management interface, realize flexible management data management task, administer demand so as to cope with complicated data, promote number
According to governance efficiency.
Other features and advantage will illustrate in the following description, also, partly become from specification
It obtains it is clear that being understood and implementing the application.The purpose of the application and other advantages can be by specifications, right
Specifically noted structure is achieved and obtained in claim and attached drawing.
Detailed description of the invention
Attached drawing is used to provide to further understand technical scheme, and constitutes part of specification, with this
The embodiment of application is used to explain the technical solution of the application together, does not constitute the limitation to technical scheme.
Fig. 1 is the flow chart of the task management method during data provided by the embodiments of the present application are administered；
The exemplary diagram of Fig. 2 dependence between the operator in the embodiment of the present application；
Fig. 3 is the exemplary diagram of the global DAG in the embodiment of the present application；
Fig. 4 is the global DAG in the embodiment of the present application and the exemplary relationship figure between task queue；
Fig. 5 is the schematic diagram of the task management device during data provided by the embodiments of the present application are administered；
Fig. 6 is the schematic diagram of terminal device provided by the embodiments of the present application.
Specific embodiment
Embodiments herein is described in detail below in conjunction with attached drawing.It should be noted that in the feelings not conflicted
Under condition, the features in the embodiments and the embodiments of the present application can mutual any combination.
Step shown in the flowchart of the accompanying drawings can be in a computer system such as a set of computer executable instructions
It executes.Also, although logical order is shown in flow charts, and it in some cases, can be to be different from herein suitable
Sequence executes shown or described step.
The embodiment of the present application propose a kind of data administer in task management method and device, can provide based on original
The task management mode of the support visualized operation of data can cope with complexity so as to promote the flexibility of task management
Data administer demand, promoted data governance efficiency.In the embodiment of the present application, for data administer in there is it is a large amount of answer
The data management task miscellaneous, repeatability is relatively high can execute orderly and automatically they, to maximumlly reduce
Implementation cost.
Fig. 1 is the flow chart of the task management method during data provided by the embodiments of the present application are administered.The embodiment of the present application
The task management method of offer can be administered by data in task management device (for example, the task tune in data governing system
Spend device) it executes.However, the application does not limit this.
As shown in Figure 1, task management method provided in this embodiment may comprise steps of:
The initial data that step 101, building are administered using data is the task nexus figure of starting point, wherein task nexus figure
For directed acyclic graph (DAG, Directed Acyclic Graph)；
Step 102, task based access control relational graph provide the task management interface for supporting visualized operation；
Step 103, the operation in response to detecting in task management interface manage data management task.
Wherein, DAG refers to that any a line has direction, and the figure of loop is not present.
In one exemplary embodiment, step 101 may include: to establish original table and object table, wherein original table is to use
In the table for the initial data that storage data are administered, object table is the table for target data needed for storing business；It will be from original
The data improvement of data to target data is in turn divided into pretreatment stage, increment synchronization stage, incremental computations stage and melts
The conjunction stage；According to original table, object table, pretreatment stage, increment synchronization stage, incremental computations stage and fusing stage, wound
Build the operator of following four type: preconditioning operator, increment synchronization operator, incremental computations operator, fusion operator；Based on above-mentioned four
The dependence between dependence and original table and preconditioning operator between the operator of seed type constructs task nexus
Figure；Wherein, task nexus figure can be using original table as the directed acyclic graph of starting point, alternatively, with the corresponding data of original table
Source is the directed acyclic graph of starting point.
Wherein, by establishing original table, the initial data that data source (for example, database or file) can be provided is stored
Into the storage medium of data governing system local, administers and use so as to follow-up data.What object table can be provided according to user
Business demand is created.
Wherein, original data processing can be divided into following four rank at the process (i.e. data improvement) of target data
Section: pretreatment stage, increment synchronization stage, incremental computations stage and fusing stage.Wherein, pretreatment stage can be for original
Beginning data carry out the stage of preliminary treatment, for example, carrying out the processing such as data volume statistics；The increment synchronization stage can be by data source
Initial data be synchronized to data governing system local original table stage；The incremental computations stage can be to extract from original table
Stage of the data to object table；Fusing stage can in the future from the incremental datas of multiple data sources and historical data press major key or
Joint major key carries out the process of multivalue processing, duplicate removal and sequence.The input data and output data in above-mentioned each stage are arranged
There is fixed data format, and above-mentioned each stage can carry out corresponding data processing according to predefined logic.It is controlled in data
In the case that the initial data of reason is given, a corresponding operator can be created to any instance in above-mentioned each stage.
In one exemplary embodiment, it can be one-to-one relationship between preconditioning operator and increment synchronization operator, increase
Measuring can be one-to-many relationship between synchronous operator and incremental computations operator, increment synchronization operator with merge between operator can be with
For the relationship of multi-to-multi；It can be one-to-many relationship between original table and preconditioning operator.
In one exemplary embodiment, the task management interface that step 102 provides can show that complete task is closed in advance
System's figure, alternatively, in response to the clicking operation detected in task management interface, from initial data (for example, data source or original
Table) start successively display task nexus figure.However, the application does not limit this.
In one exemplary embodiment, step 103 may include: the operator in response to detecting in task management interface
Selection operation determines operator set to be processed；Operation is executed in response to the management detected in task management interface, is handled
The operator set.
Wherein, operator selection operation may include continuous page clicking operation, however, the application does not limit this.
For example, in other embodiments, operator selection operation may include page frame selection operation.
Wherein, it may include: the clicking operation to the specific command shown in task management interface that management, which executes operation,.So
And the application does not limit this.For example, in other embodiments, management execute operation may include page right click operation with
And after right click the order of display interface selection operation clicking operation.
In one exemplary embodiment, operation is executed in response to the management detected in task management interface, processing is calculated
Subclass may include at least one of:
Corresponding task addition order is operated when management executes, each operator in operator set is generated into corresponding data and is controlled
Reason task, and task queue is added in the data management task of generation, wait scheduling to execute；
Corresponding task delete command is operated when management executes, the corresponding data of each operator in operator set are administered and are appointed
Business is deleted from task queue；
Order is re-run when management executes the corresponding task of operation, in task queue, resets each calculation in operator set
The state of the corresponding data management task of son, so that data management task is re-scheduled for execution.
Wherein, the executable management of data management task may include at least one of following: task addition, task are deleted
It removes, task is re-run.Moreover, the embodiment of the present application support manages batch tasks accordingly.
In one exemplary embodiment, above-mentioned task management method can also include: the mission failure of configuration task queue
Processing strategie；Wherein, mission failure processing strategie includes at least one of the following: that any data management task in task queue is held
After row failure, suspend the data management task in task dispatching queue, until the data management task of the failure is merely re-executed
Success；After any data management task in task queue executes failure, continue the data in task dispatching queue with the failure
Management task does not have the follow-up data management task of dependence, until not schedulable data management task.Wherein it is possible to
Before or after step 103, mission failure processing strategie is configured.However, the application does not limit this.
In one exemplary embodiment, in response to the operator selection operation detected in task management interface, determine to
The operator set of processing may include: display data sources or original table in task management interface, in response in task management field
The clicking operation to data source or original table detected on face, expansion display is dependent on clicked data source or original table
The list of subsequent stage or multistage operator；Operator to be processed is selected from the list of expansion.
Task management method provided by the embodiments of the present application constructs task nexus figure, and base by starting point of initial data
In task nexus figure, the task management interface for supporting visualized operation is provided, to realize the flexible management of data management task.
For example, the operator set that execute operation can be chosen, and a key is realized in the task management interface for supporting visualized operation
The batch of the corresponding data management task of operator set is added, deletes or re-run operation.In this way, provided in this embodiment
Business management method can cope with complicated data and administer demand, to promote data governance efficiency.
Task management method provided by the embodiments of the present application is illustrated below by an exemplary embodiment.Its
In, it is illustrated so that the task dispatcher in data governing system executes task management method as an example.The present exemplary embodiment mentions
The task management method of confession may comprise steps of:
Step 1: importing initial data.Wherein, by the relevant information of disposition data source, data can be made to administer system
System sends data access request to source data server according to these configuration informations.Each different data source can be used as data
Source instance is recorded in data governing system.Several library tables is corresponded to below each data source, these library tables can be believed
Breath is synchronized in data governing system, as the original table (Schema) of entire data governing system, wherein may include table
The information such as Chinese name, English name, list of fields, field data types.It, can be by data source subsequently through the information of original table
Raw data base of the initial data storage into local storage medium, as follow-up data improvement.
Step 2: building object table.In this step, it can be constructed on the basis of initial data according to business demand
The model (referred to as object table) for the target data for needing to administer out.Entire data improvement can be regarded as substantially will be at initial data
Manage into the process of target data.Wherein, business demand can be provided according to the actual situation by user.
Step 3: creation operator.
In the present exemplary embodiment, the whole flow process that data are administered can be divided into the following four stage: pretreatment rank
Section, increment synchronization stage, incremental computations stage and fusing stage.Each stage can correspond to a type of operator, every type
The operator of type has outputting and inputting for oneself fixed data format, and can handle data according to logic predetermined.Its
In, in the case where given input value, corresponding operator can be created to the example in each stage.
Wherein, pretreatment stage, which refers to, carries out some preliminary pretreatments, such as statistics, connection processing etc. to initial data；
The increment synchronization stage refers to the stage that data are synchronized to the local original table of data governing system from data source；The incremental computations stage
Refer to the stage that data are drawn into object table from original table, wherein may include that data are converted, cleaned etc. with processing；Fusion
Stage refer to by incremental data and historical data from multiple data sources by major key or combine major key carry out multivalue processing, duplicate removal and
The process of sequence.Data after fusing stage (can be administered by data as the source data that upper layer analysis is applied
Obtained target data).
In the present exemplary embodiment, the operator of following four type can be created: preconditioning operator, increment synchronization operator,
Incremental computations operator and fusion operator.
Step 4: building DAG.
It, can be based between operator since there is specific dependences between the operator of above-mentioned four seed type
Dependence constructs DAG.The exemplary diagram of Fig. 2 dependence between the operator in the embodiment of the present application.Wherein, in Fig. 2
Arrow direction indicates the direction of dependence between operator.As shown in Fig. 2, being one between preconditioning operator and increment synchronization operator
One-to-one correspondence；It is one-to-many relationship between increment synchronization operator and incremental computations operator, this is because same original table
Data may be drawn into multiple object tables；Increment synchronization with merge be between operator multi-to-multi relationship, this be by
Caused by the difference of the target data difference and versions of data that are merged.
Based on above-mentioned analysis, the operator created in entire data being administered, which constitutes one, has multiple start nodes
DAG.In the present exemplary embodiment, global DAG can be obtained using data source and original table as the preposition node of preconditioning operator
(GlobalDAG).Fig. 3 is the exemplary diagram of the global DAG in the embodiment of the present application.Wherein, the arrow direction in Fig. 3 indicates operator
Between dependence direction.Global DAG as shown in Figure 3 is using data source as starting point.As shown in figure 3, data source with it is original
It is one-to-one relationship between table, is one-to-one relationship between original table and preconditioning operator.However, the application to this not
It limits.In other embodiments, can be one-to-many relationship between data source and original table, original table and preconditioning operator it
Between can be one-to-many relationship.
Step 5: providing the task management interface for supporting visualized operation.
In the present exemplary embodiment, based on the task management interface that GlobalDAG is provided, can with display data sources, according to
The point selection operation of user, then successively expansion shows relevant original table and operator.For example, user can be from some data of display
Source is set out, and sequentially finds and choose the operator (for example, increment synchronization operator, fusion operator etc.) dependent on the data source, and under
Send out the administration order to selected operator.
For example, in task management interface, after user clicks certain data source S, unfolded display in task management interface
The list T of original table below data source S；User continues to click certain original table t in list T, meeting in task management interface
Be unfolded all incremental data version Vi in original table t (i=1,2 ... n) (if original table t is non-increment list, will be only
One versions of data)；Wherein, due to being one-to-one relationship between preconditioning operator and increment synchronization operator, Yong Hudian
When hitting original table t, preconditioning operator list L1 and increment synchronization operator list L2 can also be unfolded in task management interface simultaneously.With
Family can choose the increment synchronization operator in preconditioning operator and list L2 in list L1, and selected operator will be added into
It has chosen in operator set.It similarly, will expansion in task management interface after user clicks the increment synchronization operator in list L2
Corresponding incremental computations operator list L3.User can choose the addition of the incremental computations operator in list L3 and choose operator set
In.Similarly, after user clicks the incremental computations operator in list L3, corresponding fusion operator will be unfolded in task management interface
List L4, user can choose the addition of the fusion operator in list L4 and choose in operator set.Above-mentioned user is in task management
Operator selection operation on interface is only a kind of example.In other embodiments, can be directly displayed in task management interface with
Data source is the task nexus figure (i.e. overall situation DAG) of starting point, and user directly can pass through clicking operation in task relational graph
Corresponding operator addition is chosen to have chosen in operator set.However, the application does not limit this.
In the present exemplary embodiment, after corresponding operator is chosen and is added to and has chosen operator set, Yong Huke
To trigger the processing to operator set has been chosen by clicking the administration order shown in task management interface.However, the application
This is not limited.In other embodiments, user can select corresponding administration order by clicking menu bar, to trigger pair
The processing of operator set is chosen.
For example, user can click addition order, the corresponding data management task of the operator in operator set will have been chosen
It is added to task queue, scheduling is waited to execute；User can click delete command, corresponding by the operator in operator set has been chosen
Data management task removed from task queue；User can click and re-run order, so that in task queue, resetting is
The state for choosing the corresponding data management task of operator in operator set enables corresponding data management task by again
Scheduling executes.
Wherein, after the task dispatcher of data governing system receives the addition order that user issues, addition can be executed
Operation, at this point, task queue can initialize the relevant parameter (including service parameter and operating parameter) in the operator chosen, and
Environmental variance, check logic etc. are added, the data management task that can be dispatched by task dispatcher is generated.
Wherein, task queue is the important component of the task dispatcher of data governing system, is responsible for that data is allowed to administer
Task is carried out in the correct order.Fig. 4 is the global DAG in the embodiment of the present application and the relationship between task queue is shown
It is intended to.All unfinished data management tasks in task queue (including do not start, fail, waiting data are administered and are appointed
Business), all from having chosen operator set.The operator in operator set has been chosen from global DAG.As shown in figure 4, appointing
The DAG that operator set corresponding to all unfinished data management tasks is constituted in business queue is the SubDAG (son of global DAG
DAG).It should be noted that the Open circle in Fig. 4 indicates operator, filling the round of grid indicates data management task.
Step 6: configuring and starting task queue.
Wherein, after task queue is generated or updated, it can star task queue to dispatch the execution of data management task.?
In one exemplary embodiment, two kinds of mission failure processing strategies are can be set in task dispatcher, to handle in task queue
Data management task executes the case where failure.Wherein, mission failure processing strategie may include: mode one, in task queue
Any one data management task execute failure after, pause dispatch the task queue in data management task (come failure
Data management task after task will not be scheduled), until failure data management task re-execute success；Mode two,
After any one data management task in task queue executes failure, continue the data management task of scheduling and the failure without according to
The subsequent executable data management task for the relationship of relying in other words, will not until not schedulable data management task
Suspend the scheduling to data management task.Two kinds of mission failure processing strategies are supported by setting task dispatcher, it can basis
Dispatching requirement is flexibly switched, and administers demand to adapt to different data.
In an illustrative embodiments, the data management task that task dispatcher can while concurrently execute can also be set
Quantity, so as to support not dependence data management task concurrently execution.
The present exemplary embodiment constructs the task nexus figure of complete set using data source as starting point, and is based on this
Business relational graph, provides the task management interface for supporting visualized operation, appoints so that user quickly finds specific data and administers
Business, or flexible management is carried out to data management task, for example, batch tasks are added, delete or re-run by a key.And
And it can also flexibly switch mission failure processing strategie according to dispatching requirement.
Fig. 5 is the schematic diagram of the task management device during data provided by the embodiments of the present application are administered.As shown in figure 5, this
The task management device that embodiment provides may include: task nexus figure building module 501, interface offer module 502 and appoint
Business management module 503；Wherein, task nexus figure constructs module 501, suitable for constructing the initial data administered using data as starting point
Task nexus figure, wherein task nexus figure is directed acyclic graph；Interface provides module 502, is suitable for task based access control relational graph,
The task management interface for supporting visualized operation is provided；Task management module 503 is adapted for examining in task management interface
The operation measured manages data management task.
In one exemplary embodiment, task nexus figure building module 501 may be adapted to be constructed in the following manner with number
Initial data according to improvement is the task nexus figure of starting point: establishing original table and object table, wherein original table is for storing
The table for the initial data that data are administered, object table are the table for target data needed for storing business；Will from initial data to
The data improvement of target data is in turn divided into pretreatment stage, increment synchronization stage, incremental computations stage and fusing stage；
According to pretreatment stage, increment synchronization stage, incremental computations stage and the fusion rank in original table, object table, data improvement
Section creates the operator of following four type: preconditioning operator, increment synchronization operator, incremental computations operator, fusion operator；It is based on
The dependence between dependence and original table and preconditioning operator between the operator of above-mentioned four seed type, building are appointed
Business relational graph；Wherein, task nexus figure is the directed acyclic graph using original table as starting point, alternatively, with the corresponding number of original table
It is the directed acyclic graph of starting point according to source.
In one exemplary embodiment, task management module 503 may be adapted in the following manner in response in task pipe
The operation detected on reason interface, manages data management task: choosing in response to the operator detected in task management interface
Operation, determines operator set to be processed；Operation is executed in response to the management detected in task management interface, handles operator
Set.
Related description about task management device provided in this embodiment is referred to retouching for above-mentioned task management method
It states, therefore is repeated no more in this.
Fig. 6 is the schematic diagram of terminal device provided by the embodiments of the present application.As shown in fig. 6, terminal provided in this embodiment
Equipment (for example, the fixed terminals such as the mobile terminals such as smart phone, tablet computer or desktop computer), comprising: processor 601,
Memory 602, display unit 603 and input unit 604；Wherein, display unit 603 connects processor 601, is suitable for display and appoints
Business administration interface；Input unit 604 connects processor 601, the operation being adapted to detect in task management interface；Memory 602 is suitable
Task management program in storing data improvement, the task management program realize above-described embodiment when being executed by processor 601
The step of task management method of offer, such as the step of task management method shown in FIG. 1.
It should be noted that the structure of terminal device shown in Fig. 6 does not constitute the restriction to terminal device, can wrap
It includes than illustrating more or fewer components, perhaps combine certain components or different component layouts is provided.
Wherein, processor 601 can include but is not limited to microprocessor (MCU, Microcontroller Unit) or can
The processing unit of programmed logic device (FPGA, Field Programmable Gate Array) etc..Memory 602 can be used for
The software program and module for storing application software, such as the corresponding program instruction of task management method or mould in the present embodiment
Block, the software program and module that processor 601 is stored in memory 602 by operation, thereby executing various function application
And data processing, for example realize task management method provided in this embodiment.Memory 602 may include high speed random storage
Device may also include nonvolatile memory, such as one or more magnetic storage device, flash memory or other are non-volatile solid
State memory.In some instances, memory 602 may include the memory remotely located relative to processor 601, these are long-range
Memory can pass through network connection to terminal device.The example of above-mentioned network include but is not limited to internet, intranet,
Local area network, mobile radio communication and combinations thereof.
Wherein, input unit 604 may be adapted to the information for receiving input.Illustratively, input unit 604 may include touching
Control panel (or being touch screen) and other input equipments (for example, mouse, keyboard, operating stick etc.).Display unit 603 can be with
Suitable for showing the information of user's input or being supplied to the information of user.Display unit 603 may include display panel, for example, liquid
Crystal display, Organic Light Emitting Diode etc..Illustratively, touch panel can cover on a display panel, when touch panel is examined
After measuring touch operation on it or nearby, processor 601 is transferred to determine the type of touch event, is followed by subsequent processing device
601 provide corresponding visual output according to the type of touch event on a display panel.Illustratively, touch panel and display surface
What plate can be used as two independent components to realize terminal device outputs and inputs function, alternatively, touch panel and display surface
Plate, which can integrate, outputs and inputs function together to realize.
In addition, the embodiment of the present application also provides a kind of computer-readable medium, the task management being stored in data improvement
The step of program, which realizes above-mentioned task management method when being executed by processor, such as step shown in FIG. 1
Suddenly.
It will appreciated by the skilled person that whole or certain steps, system, dress in method disclosed hereinabove
Functional module/unit in setting may be implemented as software, firmware, hardware and its combination appropriate.In hardware embodiment,
Division between the functional module/unit referred in the above description not necessarily corresponds to the division of physical assemblies；For example, one
Physical assemblies can have multiple functions or a function or step and can be executed by several physical assemblies cooperations.Certain groups
Part or all components may be implemented as by processor, such as the software that digital signal processor or microprocessor execute, or by
It is embodied as hardware, or is implemented as integrated circuit, such as specific integrated circuit.Such software can be distributed in computer-readable
On medium, computer-readable medium may include computer storage medium (or non-transitory medium) and communication media (or temporarily
Property medium).As known to a person of ordinary skill in the art, term computer storage medium is included in for storing information (such as
Computer readable instructions, data structure, program module or other data) any method or technique in the volatibility implemented and non-
Volatibility, removable and nonremovable medium.Computer storage medium include but is not limited to RAM, ROM, EEPROM, flash memory or its
His memory technology, CD-ROM, digital versatile disc (DVD) or other optical disc storages, magnetic holder, tape, disk storage or other
Magnetic memory apparatus or any other medium that can be used for storing desired information and can be accessed by a computer.This
Outside, known to a person of ordinary skill in the art to be, communication media generally comprises computer readable instructions, data structure, program mould
Other data in the modulated data signal of block or such as carrier wave or other transmission mechanisms etc, and may include any information
Delivery media.
Claims (10)
1. the task management method in a kind of data improvement characterized by comprising
The initial data administered using data is constructed as the task nexus figure of starting point, wherein the task nexus figure is oriented nothing
Ring figure；
Based on the task nexus figure, the task management interface for supporting visualized operation is provided；
In response to the operation detected in the task management interface, data management task is managed.
2. the method according to claim 1, wherein the initial data administered using data that constructs is starting point
Task nexus figure, comprising:
Establish original table and object table；Wherein, the original table is the table for storing the initial data of data improvement, the mesh
Mark table is the table for target data needed for storing business；
Pretreatment stage, increment synchronization rank will be in turn divided into from the initial data to the improvement of the data of the target data
Section, incremental computations stage and fusing stage；
According to the pretreatment stage in the original table, the object table, data improvement, increment synchronization stage, increment meter
Calculation stage and fusing stage, create the operator of following four type: preconditioning operator, increment synchronization operator, incremental computations are calculated
Son, fusion operator；
Between dependence and the original table and the preconditioning operator between operator based on above-mentioned four seed type
Dependence constructs task nexus figure；Wherein, the task nexus figure is using the original table as the directed acyclic of starting point
Figure, alternatively, using the corresponding data source of the original table as the directed acyclic graph of starting point.
3. according to the method described in claim 2, it is characterized in that, between the preconditioning operator and the increment synchronization operator
It is one-to-many relationship, the increment between the increment synchronization operator and the incremental computations operator for one-to-one relationship
Synchronize operator and the relationship merged between operator for multi-to-multi；It is a pair of between the original table and the preconditioning operator
More relationships.
4. according to the method described in claim 2, it is characterized in that, described in response to being detected in the task management interface
Operation, manage data management task, comprising:
In response to the operator selection operation detected in the task management interface, operator set to be processed is determined；
Operation is executed in response to the management detected in the task management interface, handles the operator set.
5. according to the method described in claim 4, it is characterized in that, described in response to being detected in the task management interface
Management execute operation, handle the operator set, including at least one of:
Corresponding task addition order is operated when the management executes, each operator in the operator set is generated into corresponding number
Task queue is added according to management task, and by the data management task of generation, scheduling is waited to execute；
Corresponding task delete command is operated when the management executes, the corresponding data of each operator in the operator set are controlled
Reason task is deleted from the task queue；
Order is re-run when the management executes the corresponding task of operation, in the task queue, is reset in the operator set
The corresponding data management task of each operator state, so that the data management task is re-scheduled for execution.
6. according to the method described in claim 5, it is characterized in that, the method also includes: configure the task queue appoint
Business failure handling strategy；Wherein, the mission failure processing strategie includes at least one of the following:
After any data management task in the task queue executes failure, the data that pause is dispatched in the task queue are controlled
Reason task, until the data management task of the failure re-executes success；
After any data management task in the task queue executes failure, continue to dispatch in the task queue with the mistake
The data management task lost does not have the follow-up data management task of dependence, until not schedulable data management task.
7. according to the method described in claim 4, it is characterized in that, described in response to being detected in the task management interface
Operator selection operation, determine operator set to be processed, comprising:
The data source or original table are shown in the task management interface, in response to detecting in the task management interface
The clicking operation to the data source or original table arrived, expansion display is dependent on the subsequent of clicked data source or original table
The list of one or more levels operator；Operator to be processed is selected from the list of expansion.
8. the task management device in a kind of data improvement characterized by comprising
Task nexus figure constructs module, suitable for constructing the initial data administered using data as the task nexus figure of starting point, wherein
The task nexus figure is directed acyclic graph；
Interface provides module, is suitable for being based on the task nexus figure, provides the task management interface for supporting visualized operation；
Task management module is adapted for the operation detected in the task management interface, manages data management task.
9. a kind of terminal device characterized by comprising display unit, input unit, memory and processor；The display
Unit connects the processor, is suitable for display task management interface；The input unit connects the processor, is adapted to detect for institute
State the operation in task management interface；The memory is suitable for the task management program in storing data improvement, the task pipe
The step of reason program realizes the task management method as described in any one of claims 1 to 7 when being executed by the processor.
10. a kind of computer-readable medium, which is characterized in that the task management program being stored in data improvement, the task
The step of task management method as described in any one of claims 1 to 7 is realized when management program is executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811594698.9A CN109814991A (en) | 2018-12-25 | 2018-12-25 | A kind of data administer in task management method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811594698.9A CN109814991A (en) | 2018-12-25 | 2018-12-25 | A kind of data administer in task management method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109814991A true CN109814991A (en) | 2019-05-28 |
Family
ID=66602412
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811594698.9A Pending CN109814991A (en) | 2018-12-25 | 2018-12-25 | A kind of data administer in task management method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109814991A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110362362A (en) * | 2019-07-24 | 2019-10-22 | 北京明略软件***有限公司 | Method for scheduling task and device, storage medium, electronic device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090276449A1 (en) * | 2008-04-30 | 2009-11-05 | Business Objects, S.A. | Apparatus and method to transform an extract transform and load (etl) task into a delta load task |
| CN105976158A (en) * | 2016-04-26 | 2016-09-28 | 中国电子科技网络信息安全有限公司 | Visual ETL flow management and scheduling monitoring method |
| CN106897411A (en) * | 2017-02-20 | 2017-06-27 | 广东奡风科技股份有限公司 | ETL system and its method based on Spark technologies |
| CN110232085A (en) * | 2019-04-30 | 2019-09-13 | 中国科学院计算机网络信息中心 | A kind of method of combination and system of big data ETL task |
-
2018
- 2018-12-25 CN CN201811594698.9A patent/CN109814991A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090276449A1 (en) * | 2008-04-30 | 2009-11-05 | Business Objects, S.A. | Apparatus and method to transform an extract transform and load (etl) task into a delta load task |
| CN105976158A (en) * | 2016-04-26 | 2016-09-28 | 中国电子科技网络信息安全有限公司 | Visual ETL flow management and scheduling monitoring method |
| CN106897411A (en) * | 2017-02-20 | 2017-06-27 | 广东奡风科技股份有限公司 | ETL system and its method based on Spark technologies |
| CN110232085A (en) * | 2019-04-30 | 2019-09-13 | 中国科学院计算机网络信息中心 | A kind of method of combination and system of big data ETL task |
Non-Patent Citations (1)
| Title |
|---|
| 赵双瑞: "通用ETL工具的研究与开发", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110362362A (en) * | 2019-07-24 | 2019-10-22 | 北京明略软件***有限公司 | Method for scheduling task and device, storage medium, electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210318851A1 (en) | Systems and Methods for Dataset Merging using Flow Structures | |
| US9659012B2 (en) | Debugging framework for distributed ETL process with multi-language support | |
| US10073867B2 (en) | System and method for code generation from a directed acyclic graph using knowledge modules | |
| US10216814B2 (en) | Supporting combination of flow based ETL and entity relationship based ETL | |
| CN109800226A (en) | A kind of data administer in task management method and device | |
| CN107506442B (en) | Modeling method and device of model | |
| US9519663B2 (en) | Upgrading and migrating a database by a migration tool | |
| US20190138532A1 (en) | Load plan generation | |
| US9507838B2 (en) | Use of projector and selector component types for ETL map design | |
| CN105389251A (en) | Structured query language debugger | |
| CN105528418B (en) | A kind of design documentation generation method and device | |
| US10466981B1 (en) | System and method for generative programming in an integrated development environment (IDE) | |
| CN111324610A (en) | Data synchronization method and device | |
| CN108388623B (en) | ER relationship generation method and device, computer equipment and storage medium | |
| CN105956087A (en) | Data and code version management system and method | |
| CN108427709B (en) | Multi-source mass data processing system and method | |
| CN110908793A (en) | Long-time task execution method, device, equipment and readable storage medium | |
| CN109800069A (en) | A kind of method and device realizing data and administering | |
| CN107463391A (en) | Task processing method, device and equipment | |
| CN109814991A (en) | A kind of data administer in task management method and device | |
| CN113741931B (en) | Software upgrading method and device, electronic equipment and readable storage medium | |
| Pastrana-Pardo et al. | Approach to the Best Practices in Software Development Based on DevOps and SCRUM Used in Very Small Entities | |
| CN112597121A (en) | Logic script processing method and device, electronic equipment and storage medium | |
| CN112367205A (en) | Processing method and scheduling system for HTTP scheduling request | |
| CN115460077B (en) | Configuration information updating method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190528 |
|
| RJ01 | Rejection of invention patent application after publication |