CN108509440A - A kind of data processing method and device - Google Patents
A kind of data processing method and device Download PDFInfo
- Publication number
- CN108509440A CN108509440A CN201710103390.9A CN201710103390A CN108509440A CN 108509440 A CN108509440 A CN 108509440A CN 201710103390 A CN201710103390 A CN 201710103390A CN 108509440 A CN108509440 A CN 108509440A
- Authority
- CN
- China
- Prior art keywords
- node
- data source
- calculating
- dependence
- destination node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a kind of data processing methods, for intuitively showing the relationship between each calculating operation so that user can rapidly find out abnormal calculating operation.Present invention method includes：Obtain data source and the corresponding output result of the data source；Determine that the dependence between data source and the output result, the dependence include from each data source in the data source to the calculating job relations the output result；Dependence graph is generated according to the dependence, the dependence graph includes data source nodes, calculating jobs node and the node connector corresponding to each data source, the node connector is used to be directed toward second node from first node, the first node is data source nodes or calculating jobs node, the second node is to calculate jobs node, and corresponding the calculatings job dependence of the second node is in the corresponding data source of the first node or calculating job result.
Description
Technical field
The present invention relates to computer application field more particularly to a kind of data processing method and devices.
Background technology
Big data refers to the number that can not be captured, managed and be handled with conventional software tool within certain time
It is to need new tupe that could have stronger decision edge, see clearly magnanimity, the height for finding power and process optimization ability according to set
Growth rate and diversified information assets.With the development of all types of enterprises, the requirement of social big data off-line calculation is also more next
It is higher.
Computer equipment needs to use more points of mass datas, passes through multiple calculating when carrying out explication de texte to big data
Operation can just obtain final analysis result.The prior art is generally multiple by what is be applied in analytic process by text mode
Operation is calculated to be shown.
But the analytic process of offline big data is comparatively laborious, and calculating operation chain is typically very large, once meter
It calculates some node in chain and exception occurs, want the abnormal calculating operation of positioning according to prior art user, can only just pass through text
Operation is calculated one by one and goes investigation problem, this needs to expend user's a large amount of time.
Invention content
An embodiment of the present invention provides a kind of data processing method and device, for intuitively show each calculating operation it
Between relationship so that user can rapidly find out abnormal calculating operation.
In view of this, first aspect of the embodiment of the present invention provides a kind of data processing method, including：
Obtain data source and the corresponding output result of the data source；
Determine that the dependence between data source and the output result, the dependence include from the data source
Each data source to it is described output result between calculating job relations；
Dependence graph is generated according to the dependence, the dependence graph includes corresponding to each data source
Data source nodes calculate jobs node and node connector, and the node connector is used to be directed toward second node from first node,
The first node is data source nodes or calculates jobs node, and the second node is to calculate jobs node, second section
The corresponding calculating job dependence of point is in the corresponding data source of the first node or calculates job result.
Second aspect of the embodiment of the present invention provides a kind of data processing equipment, including：
Acquisition module, for obtaining data source and the corresponding output result of the data source；
First determining module, for determining the dependence between data source and the output result, the dependence
Including from each data source in the data source to the calculating job relations the output result；
Generation module, for generating dependence graph according to the dependence, the dependence graph includes described each
Data source nodes, calculating jobs node and node connector, the node connector corresponding to data source are used for from first segment
Point is directed toward second node, and the first node is data source nodes or calculates jobs node, and the second node is to calculate operation
Node, the corresponding calculating job dependence of the second node is in the corresponding data source of the first node or calculates job result.
As can be seen from the above technical solutions, the embodiment of the present invention has the following advantages：
The embodiment of the present invention can be by data source and corresponding output result identification therebetween dependence, using identifying
The dependence come generates dependence graph, which includes data source nodes, calculates jobs node and node connection
Symbol, wherein node connector is used to be directed toward second node from first node, and first node is data source nodes or calculating operation section
Point, second node are to calculate jobs node, the corresponding calculating job dependence of second node in the corresponding data source of first node or
Calculate job result.That is the embodiment of the present invention is shown by figure calculates operation and each pass calculated between operation
System can intuitively show the relationship between each calculating operation, and to when some node occurs abnormal, user is according to should be according to
Bad relational graph can rapidly find out the exception and calculate operation.
Description of the drawings
In order to illustrate the technical solution of the embodiments of the present invention more clearly, required use in being described below to embodiment
Attached drawing be briefly described, it should be apparent that, drawings in the following description are only some embodiments of the invention.
Fig. 1 is one embodiment schematic diagram of data processing system in the embodiment of the present invention；
Fig. 2 is one embodiment flow chart of data processing method in the embodiment of the present invention；
Fig. 3 is one embodiment flow chart of data processing method in the embodiment of the present invention；
Fig. 4 is one embodiment flow chart of data processing method in the embodiment of the present invention；
Fig. 5 is one embodiment schematic diagram of dependence graph in data processing method in the embodiment of the present invention；
Fig. 6 is one embodiment schematic diagram of data processing method interior joint information in the embodiment of the present invention；
Fig. 7 is one embodiment schematic diagram of data processing equipment in the embodiment of the present invention；
Fig. 8 is another embodiment schematic diagram of data processing equipment in the embodiment of the present invention；
Fig. 9 is another embodiment schematic diagram of data processing equipment in the embodiment of the present invention；
Figure 10 is another embodiment schematic diagram of data processing equipment in the embodiment of the present invention.
Specific implementation mode
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.
Term " first ", " second ", " third " " in description and claims of this specification and above-mentioned attached drawing
The (if present)s such as four " are for distinguishing similar object, without being used to describe specific sequence or precedence.It should manage
The data that solution uses in this way can be interchanged in the appropriate case, so that the embodiment of the present invention described herein for example can be to remove
Sequence other than those of illustrating or describe herein is implemented.In addition, term " comprising " and " having " and theirs is any
Deformation, it is intended that cover it is non-exclusive include, for example, containing the process of series of steps or unit, method, system, production
Product or equipment those of are not necessarily limited to clearly to list step or unit, but may include not listing clearly or for this
The intrinsic other steps of processes, method, product or equipment or unit a bit.
An embodiment of the present invention provides a kind of data processing method and device, for intuitively show each calculating operation it
Between relationship so that user can rapidly find out abnormal calculating operation.
Embodiment to facilitate the understanding of the present invention below simply introduces the scene that the embodiment of the present invention is applicable in,
Referring to Fig. 1, it illustrates a kind of system compositions that a kind of data processing method provided in an embodiment of the present invention and device are applicable in
Structural schematic diagram.
As shown in Figure 1, the system may include by service system that at least a server 101 forms and more ends
End 102.Wherein, can be stored in the server 101 in service system for data source and the data source it is corresponding output as a result,
And the dependence graph of generation is transferred to terminal 102.Terminal 102 can be used for uploading for generating dependence to server
Data and the output of figure are as a result, show the dependence graph of server return.It should be understood that terminal 102 is not limited to shown in FIG. 1
People's computer (PC, Personal Computer), can also be mobile phone, tablet computer etc., other can obtain and show dependence
The equipment of relational graph.
Such as, target data source is inputted on user terminal, and is directed to target data source setup algorithm task, and terminal is according to setting
Calculating task generate target data source it is corresponding output as a result, terminal by target data source and output result be uploaded to service
Device, the data processing method during server is according to embodiments of the present invention generates dependence graph, and returns to the dependence to the terminal
Relational graph, terminal show the dependence graph to user again.
It should be understood that the data processing method in the embodiment of the present invention can be applicable to it in addition to being suitable for above-mentioned scene
His scene, this is not limited here.Embodiment to facilitate the understanding of the present invention, below to some passes in the embodiment of the present invention
Keyword is introduced：
Data source：Data source refers to the untreated initial data stored in database in the embodiment of the present invention.
Export result：Output result in the embodiment of the present invention refer to to big data carry out explication de texte when, to original
After data carry out multiple calculating operations, the result of output.
Calculate operation：The embodiment of the present invention, which is fallen into a trap, can be regarded as industry and refers to when carrying out explication de texte to big data, from original
Data to obtain output result during, used algorithm, analysis method, the processing steps such as computational methods.
Dependence：In big data explication de texte, it is typically extremely complex to calculate step, needs to use more parts of magnanimity
Data can just obtain final analysis result by multiple calculating operations, these data form the present invention in fact with operation is calculated
The dependence in example is applied, only could carry out next calculating operation after the data on dependency chain upper layer or operation are completed.
It is lined up and takes：For calculating the dependency chain at any one of operation place, to startup, the calculating is made since data source
The industry required time is that the queuing of the calculating operation in the dependency chain takes.
It executes and takes：Execute the time used in the calculating operation.
Start the time：Start the time of the calculating operation.
Historical operation number：System executes the number of the calculating operation before current time.
The data processing method in the embodiment of the present invention is first introduced below, referring to Fig. 2, in the embodiment of the present invention at data
One embodiment of reason method includes：
201, data source and the corresponding output result of data source are obtained；
Data processing equipment obtains data source and the corresponding output of the data source as a result, specifically, which can be
Data processing equipment is in running game, Financial Management, is generated during the application programs such as data analysis, can be at data
Reason device is obtained from other devices, can also be that data processing equipment is obtained and stored in the database by other means
, this is not limited here.
202, it determines data source and exports the dependence between result；
After obtaining data source and the corresponding output result of data source, data processing equipment is according to each data source and corresponding defeated
Go out result and automatically identify dependence therebetween, which includes from each data source to corresponding output result
Calculate job relations.
203, dependence graph is generated according to dependence.
After determining dependence, dependence graph is generated according to the dependence, specifically, which includes each
The corresponding data source nodes of data source calculate jobs node and node connector, and the node connector from first node for referring to
To second node, first node is data source nodes or calculates jobs node, and second node is to calculate jobs node, second node
Corresponding calculating job dependence is in the corresponding data source of first node or calculates job result.
The embodiment of the present invention can be by data source and corresponding output result identification therebetween dependence, using identifying
The dependence come generates dependence graph, which includes data source nodes, calculates jobs node and node connection
Symbol, wherein node connector is used to be directed toward second node from first node, and first node is data source nodes or calculating operation section
Point, second node are to calculate jobs node, the corresponding calculating job dependence of second node in the corresponding data source of first node or
Calculate job result.That is the embodiment of the present invention is shown by figure calculates operation and each pass calculated between operation
System can intuitively show the relationship between each calculating operation, and to when some node occurs abnormal, user is according to should be according to
Bad relational graph can rapidly find out the exception and calculate operation.
It should be understood that being based on the corresponding embodiments of above-mentioned Fig. 2, after generating dependence graph, can be shown to user should
Dependence graph, and the task of response is executed depending on the user's operation, specifically, wherein several modes of operation are carried out below detailed
It describes in detail bright.
One, inquiry operation.
Referring to Fig. 3, another embodiment of data processing method includes in the embodiment of the present invention：
301, data source and the corresponding output result of data source are obtained；
Data processing equipment obtains data source and the corresponding output of the data source as a result, specifically, which can be
Data processing equipment is in running game, Financial Management, is generated during the application programs such as data analysis, can be at data
Reason device is obtained from other devices, can also be that data processing equipment is obtained and stored in the database by other means
, this is not limited here.
Specifically, data processing equipment can obtain data source and corresponding output result in the following way：It determines and uses
The target of family selection calculates operation, obtains the data source for having incidence relation with target calculating operation and output result.It needs
Illustrate, the data source with incidence relation refers to the process packet from the data source to the corresponding output result of the data source
The data source that operation is calculated containing the target, the output result with incidence relation are referred to from the corresponding data of output result
The process of source to the output result includes the output result that the target calculates operation.
It should be understood that data processing equipment can also obtain automatically is separated by all of preset time with current time in database
Data source and the corresponding output of these data sources are as a result, data processing equipment can also obtain preset quantity in database automatically
Data source and the corresponding output of these data sources are as a result, data processing equipment can also obtain data source and defeated by other means
Go out as a result, this is not limited here.
302, it determines data source and exports the dependence between result；
After obtaining data source and the corresponding output result of data source, data processing equipment is according to each data source and corresponding defeated
Go out result and automatically identify dependence therebetween, which includes from each data source to corresponding output result
Calculate job relations.
303, dependence graph is generated according to dependence；
After determining dependence, dependence graph is generated according to the dependence, specifically, which includes each
The corresponding data source nodes of data source calculate jobs node and node connector, and the node connector from first node for referring to
To second node, first node is data source nodes or calculates jobs node, and second node is to calculate jobs node, second node
Corresponding calculating job dependence is in the corresponding data source of first node or calculates job result.
It should be understood that the dependence graph can also include status indicator, the status indicator is for identifying in dependence graph
The corresponding state of calculating jobs node or the corresponding state of data source nodes, these states can operate successfully, operation is prolonged
Late, operation failure, operation it is not actuated or other, this is not limited here.
It, specifically can will be from same dress it should also be understood that the data source got can be grouped by data processing equipment
The data source for setting acquisition is divided into one group, or the data source that the user with identical permission is uploaded is divided into one group, or passes through it
Data source is grouped mode by he, and this is not limited here.Correspondingly, dependence graph can also include Data Identification,
The Data Identification is used to identify the data source for belonging to same group.
304, inquiry instruction of the user to destination node in the dependence graph is received；
After data processing equipment generates dependence graph, show that the dependence graph, user select the dependence to user
Destination node in relational graph, and inquiry instruction is sent for the destination node, data processing equipment receives looking into for user's transmission
Ask instruction.
It should be understood that destination node can include one or more nodes, the one or more node can be data source section
Point can also be calculating jobs node, and this is not limited here.
305, the corresponding nodal information of destination node is determined；
After receiving inquiry instruction, data processing equipment determines that the corresponding nodal information of the destination node specifically works as mesh
It is when calculating jobs node to mark node, and nodal information may include：Destination node corresponds to the title for calculating operation, destination node pair
The creation time for the calculating operation answered, the corresponding historical operation number for calculating operation of destination node, the corresponding meter of destination node
The queuing that can be regarded as industry takes, and the corresponding execution for calculating operation of destination node takes, the destination node corresponding startup time, target
The corresponding history for calculating operation of node, which is lined up, to be taken, and the corresponding history for calculating operation of destination node, which executes, to be taken, target section
The corresponding history for calculating operation of point starts time or other information；When destination node is data source nodes, nodal information can
To include：The corresponding data source of destination node, the creation time or other information of the corresponding data source of destination node.
It should be understood that it can be specifically recent execution calculating operation that history, which is lined up time-consuming, in the embodiment of the present invention
Be lined up take, can be execute several times recently the calculating operation average queuing take, can execute the meter every time in history
The average queuing that can be regarded as industry takes, and can also be that other corresponding history of the calculating operation are lined up and takes, does not limit herein specifically
It is fixed.History starts the time that the time can be specifically the last startup calculating operation in the embodiment of the present invention, can be most
The nearly time for starting the calculating operation several times, it can also be that other start the time of the calculating operation in history, specifically herein not
It is construed as limiting.History in the embodiment of the present invention execute take can be specifically it is the last execute spent by the calculating operation when
Between, it can be the average time executed several times recently spent by the calculating operation, can execute calculating work every time in history
Average time spent by industry can also be that other execute the time spent by the calculating operation in history, not make herein specifically
It limits.
Specifically, data processing equipment can be determined as follows the corresponding nodal information of destination node：At data
Reason device determines the corresponding nodal information of destination node according to the type of inquiry instruction, such as, the inquiry instruction that user sends
For title inquiry instruction, then the nodal information that data processing equipment determines is the title for calculating operation, and the inquiry that user sends refers to
It enables to start time inquiring instruction, then the nodal information that data processing equipment determines is the startup time for calculating operation, herein not
It enumerates again.
Data processing equipment can also be determined as follows the corresponding nodal information of destination node：Data processing equipment
The quantity for being included according to ownership goal node determines the corresponding nodal information of destination node, such as, destination node includes one
When a calculating jobs node, the nodal information that data processing equipment determines is the corresponding title for calculating operation of the destination node,
Creation time and the historical operation number for calculating operation；When the destination node of user's selection includes multiple calculating jobs nodes, number
It is that the corresponding queuing for calculating operation of the destination node takes according to the nodal information that processing unit determines, executes and take, history row
Team takes and history is executed and taken.
Data processing equipment can also determine the corresponding nodal information of destination node by other means, not make herein specifically
It limits.
306, the nodal information is returned to user.
After data processing equipment determines the corresponding nodal information of destination node, the nodal information is returned to user, specifically
The nodal information can be shown by way of text, and the nodal information can also be shown by way of chart, can also be led to
It crosses other modes and shows the nodal information, this is not limited here.
The embodiment of the present invention can be by data source and corresponding output result identification therebetween dependence, using identifying
The dependence come generates dependence graph, which includes data source nodes, calculates jobs node and node connection
Symbol, wherein node connector is used to be directed toward second node from first node, and first node is data source nodes or calculating operation section
Point, second node are to calculate jobs node, the corresponding calculating job dependence of second node in the corresponding data source of first node or
Calculate job result.That is the embodiment of the present invention is shown by figure calculates operation and each pass calculated between operation
System can intuitively show the relationship between each calculating operation, and to when some node occurs abnormal, user is according to should be according to
Bad relational graph can rapidly find out the exception and calculate operation.
Secondly, dependence graph in the embodiment of the present invention can also include status information, then user directly by this according to
Bad relational graph can understand each state for calculating operation, i.e. data processing equipment can automatically identify abnormal calculating and make
Industry does not need user and is analyzed for each calculating operation, saves the operating time of user, further promote recognition speed.
Again, the embodiment of the present invention can return to the nodal information of destination node according to the inquiry instruction of user to user,
Each calculating operation and data source are managed convenient for user.
Two, operation is re-run.
Referring to Fig. 4, another embodiment of data processing method includes in the embodiment of the present invention：
401, data source and the corresponding output result of data source are obtained；
Data processing equipment obtains data source and the corresponding output of the data source as a result, specifically, which can be
Data processing equipment is in running game, Financial Management, is generated during the application programs such as data analysis, can be at data
Reason device is obtained from other devices, can also be that data processing equipment is obtained and stored in the database by other means
, this is not limited here.
Specifically, data processing equipment can obtain data source and corresponding output result in the following way：It determines and uses
The target of family selection calculates operation, obtains the data source for having incidence relation with target calculating operation and output result.It needs
Illustrate, the data source with incidence relation refers to the process packet from the data source to the corresponding output result of the data source
The data source that operation is calculated containing the target, the output result with incidence relation are referred to from the corresponding data of output result
The process of source to the output result includes the output result that the target calculates operation.
It should be understood that data processing equipment can also obtain automatically is separated by all of preset time with current time in database
Data source and the corresponding output of these data sources are as a result, data processing equipment can also obtain preset quantity in database automatically
Data source and the corresponding output of these data sources are as a result, data processing equipment can also obtain data source and defeated by other means
Go out as a result, this is not limited here.
402, it determines data source and exports the dependence between result；
After obtaining data source and the corresponding output result of data source, data processing equipment is according to each data source and corresponding defeated
Go out result and automatically identify dependence therebetween, which includes from each data source to corresponding output result
Calculate job relations.
403, dependence graph is generated according to dependence；
After determining dependence, dependence graph is generated according to the dependence, specifically, which includes each
The corresponding data source nodes of data source calculate jobs node and node connector, and the node connector from first node for referring to
To second node, first node is data source nodes or calculates jobs node, and second node is to calculate jobs node, second node
Corresponding calculating job dependence is in the corresponding data source of first node or calculates job result.
It should be understood that the dependence graph can also include status indicator, the status indicator is for identifying in dependence graph
Calculating jobs node or the corresponding state of data source nodes, the state can operate successfully, operating delay, operation failure,
Operate it is not actuated or other, this is not limited here.
It, specifically can will be from same dress it should also be understood that the data source got can be grouped by data processing equipment
The data source for setting acquisition is divided into one group, or the data source that the user with identical permission is uploaded is divided into one group, or passes through it
Data source is grouped mode by he, and this is not limited here.Correspondingly, dependence graph can also include Data Identification,
The Data Identification is used to identify the data source for belonging to same group.
404, it receives user jobs node is calculated to target in the dependence graph and re-run instruction；
After data processing equipment generates dependence graph, show that the dependence graph, user select the dependence to user
Target in relational graph calculates jobs node, and re-runs instruction for destination node transmission, and data processing equipment, which receives, to be used
What family was sent re-runs instruction.
It should be understood that it can include one or more calculating jobs nodes that target, which calculates jobs node, do not limit herein specifically
It is fixed.
Specifically, for include status indicator dependence graph, user can select state for operation failure, operate
Postpone or operate the corresponding calculating jobs node of not actuated calculating operation, the calculating operation section of other states can also be selected
Point, this is not limited here.For not including the dependence graph of status indicator, user can select to need to optimize
Or the corresponding calculating jobs node of calculating operation of analysis, other can also be selected to calculate jobs node, do not limited herein specifically
It is fixed.
405, instruction is re-run according to this and executes the corresponding calculating operation of target calculating jobs node again；
Data processing equipment receives re-run instruction after, according to this re-run instruction execute again the target calculate operation
The corresponding calculating operation of node.Specifically, when data processing equipment executes again, the meter can be executed based on newest data source
It can be regarded as industry, can also be based on the calculating operation be executed for generating the data source of the dependence graph, this is not limited here.
406, the result executed again is returned to user.
Data processing equipment will execute the target and calculate the corresponding obtained result of calculating operation of jobs node to user
It returns, specifically, which can be a state instruction, that is, indicate that user's operation is successful, operating delay, operation failure,
Operate not actuated or other states.The result can also be a result of calculation, that is, execute the data obtained after the calculating operation.
The result can also be other information, and this is not limited here.
The embodiment of the present invention can be by data source and corresponding output result identification therebetween dependence, using identifying
The dependence come generates dependence graph, which includes data source nodes, calculates jobs node and node connection
Symbol, wherein node connector is used to be directed toward second node from first node, and first node is data source nodes or calculating operation section
Point, second node are to calculate jobs node, the corresponding calculating job dependence of second node in the corresponding data source of first node or
Calculate job result.That is the embodiment of the present invention is shown by figure calculates operation and each pass calculated between operation
System can intuitively show the relationship between each calculating operation, and to when some node occurs abnormal, user is according to should be according to
Bad relational graph can rapidly find out the exception and calculate operation.
Secondly, the embodiment of the present invention can execute calculating operation again according to the instruction that re-runs of user so that Yong Huke
With by re-running as a result, for further analysis to abnormal calculating operation, convenient for user to each optimization for calculating operation.
In order to make it easy to understand, the data processing method in the embodiment of the present invention is introduced with an application scenarios below：
During user U1 is uploaded onto the server data B, C and D by computer P1, B, C and D are stored in database by server
In.During user U2 is uploaded onto the server data E by computer P2, server stores E in the database.Server is according to pre-
The calculating task first set completes following calculate：Data B obtains output result J by calculating operation A；Data C is made by calculating
After industry F, then by calculating operation A obtain output result K；Data D, which is first passed through, calculates operation G, then is finally led to by calculating operation I
It crosses calculating operation A and obtains output result L；Data E, which is first passed through, calculates operation H, then by calculating operation I, makees finally by calculating
Industry A obtains output result M.User U1 calculates operation A (target calculating operation) by computer P1 login service devices, selection, concurrently
The instruction for generating dependence graph is sent, server is determined according to the instruction calculates operation A, and obtains and calculate operation tool with the target
Relevant data source B, C, D and E (data source) and output result J, K, L and M, according to the data source of acquisition and defeated
Go out the dependence during result identifies, specifically：It needs to complete to calculate operation A from data B to output result J；From data C
It needs first to complete to calculate operation F to output result K, then completes to calculate operation A；It needs first to complete to count from data D to output result L
It can be regarded as industry G, then complete to calculate operation I, then complete to calculate operation A；From data E to output result M, need first to complete to calculate operation
H, then complete to calculate operation I, then complete to calculate operation A.According to above-mentioned dependence, dependence graph as shown in Figure 5 is generated,
Wherein, B, C, D and E are data source nodes, and A, F, G, H and I are to calculate jobs node, calculate operation F and calculate operation G operations at
Work(, calculate operation A, F, H and I has delay in operation.Directed line segment in figure is node connector, for should be according to from B to A
Rely relational graph in a complete dependency chain, from C to F again to A be the dependence graph in a complete dependency chain, from D to F
It is again a complete dependency chain in the dependence graph to A to I, it is complete for one in the dependence graph to A again to I from E to H
Whole dependency chain.
After server generates dependence graph as described in Figure 5, which is shown to user U1 by computer P1,
User U1 selects the calculating jobs node A in the dependence graph and calculating operation F, and click function button " analysis ", computer
P1 servers send the analysis instruction for being directed to and calculating jobs node A and calculating jobs node F (destination node), and server receives
The analysis instruction determines and calculates jobs node A startup times corresponding with jobs node F is calculated, the last startup
Time is lined up and takes, and history, which is averagely lined up, to be taken, and execution takes and history is averagely lined up and takes (the corresponding node of destination node
Information), server generates line chart as shown in Figure 6 according to these information, and returns to the line chart to user.
The data processing method in the embodiment of the present invention is described above, is described below at the data in the embodiment of the present invention
Device is managed, referring to Fig. 7, one embodiment of data processing equipment includes in the embodiment of the present invention：
Acquisition module 701, for obtaining data source and the corresponding output result of data source；
First determining module 702, for determine data source and export result between dependence, dependence include from
In data source each data source to output result between calculating job relations；
Generation module 703, for generating dependence graph according to dependence, dependence graph includes that each data source institute is right
The data source nodes answered calculate jobs node and node connector, and node connector is used to be directed toward second node from first node,
First node is data source nodes or calculates jobs node, and second node is to calculate jobs node, the corresponding calculating of second node
Job dependence is in the corresponding data source of first node or calculates job result.
The embodiment of the present invention can be by data source and corresponding output result identification therebetween dependence, using identifying
The dependence come generates dependence graph, which includes data source nodes, calculates jobs node and node connection
Symbol, wherein node connector is used to be directed toward second node from first node, and first node is data source nodes or calculating operation section
Point, second node are to calculate jobs node, the corresponding calculating job dependence of second node in the corresponding data source of first node or
Calculate job result.That is the embodiment of the present invention is shown by figure calculates operation and each pass calculated between operation
System can intuitively show the relationship between each calculating operation, and to when some node occurs abnormal, user is according to should be according to
Bad relational graph can rapidly find out the exception and calculate operation.
Based on the corresponding embodiments of above-mentioned Fig. 7, referring to Fig. 8, in data processing equipment provided in an embodiment of the present invention
In another embodiment, which further includes：
First receiving module 704, the inquiry instruction for receiving user to destination node in dependence graph；
Second determining module 705, for determining the corresponding nodal information of destination node；
First sending module 706 is used for user's return node information.
It should be understood that in the embodiment of the present invention, when destination node is to calculate jobs node, nodal information may include：Mesh
It marks node and corresponds to the title for calculating operation, the corresponding creation time for calculating operation of destination node, the corresponding calculating of destination node
The historical operation number of operation, the corresponding queuing for calculating operation of destination node take, the corresponding calculating operation of destination node
It executes and takes, the destination node corresponding startup time, the corresponding history for calculating operation of destination node, which is lined up, to be taken, destination node
The corresponding history for calculating operation, which executes, to be taken, and the corresponding history for calculating operation of destination node starts time or other information；
When destination node is data source nodes, nodal information may include：The corresponding data source of destination node, destination node are corresponding
The creation time or other information of data source.
Dependence graph in the embodiment of the present invention can also include status information, then user directly passes through the dependence
Figure can understand each state for calculating operation, i.e. data processing equipment can automatically identify abnormal calculating operation, be not required to
It wants user to be analyzed for each calculating operation, saves the operating time of user, further promote recognition speed.
The embodiment of the present invention can return to the nodal information of destination node according to the inquiry instruction of user to user, convenient for using
Family is managed each calculating operation and data source.
Based on the corresponding embodiments of above-mentioned Fig. 7 or Fig. 8, referring to Fig. 9, being filled in data processing provided in an embodiment of the present invention
In another embodiment set, which further includes：
Second receiving module 707 calculates jobs node to target in dependence graph for receiving user and re-runs finger
It enables；
Execution module 708, for performance objective to calculate the corresponding calculating operation of jobs node again according to instruction is re-run；
Second sending module 709, for returning to the result executed again to user.
The embodiment of the present invention can execute calculating operation again according to the instruction that re-runs of user so that user can pass through
It is re-running as a result, for further analysis to abnormal calculating operation, convenient for user to each optimization for calculating operation.
Based on any one embodiment in above-mentioned Fig. 7 to Fig. 9 corresponding embodiments, in data provided in an embodiment of the present invention
In another embodiment of processing unit, acquisition module may include：
Determination unit, for determining that the target of user's selection calculates operation；
Acquiring unit calculates data source and output result of the operation with incidence relation for obtaining with target.
Data source is obtained an embodiment of the present invention provides a kind of acquisition module and exports the concrete mode of result, the side of improving
The realizability of case.
It should be understood that based on any one embodiment in above-mentioned Fig. 7 to Fig. 9 corresponding embodiments, carried in the embodiment of the present invention
In other embodiments of the data processing equipment of confession, dependence graph can include status indicator, and the status indicator is for marking
Know the calculating jobs node or the corresponding state of data source nodes in dependence graph, which can operate successfully, operation
Delay, operation failure, operation it is not actuated or other, this is not limited here.
It should be understood that based on any one embodiment in above-mentioned Fig. 7 to Fig. 9 corresponding embodiments, carried in the embodiment of the present invention
In other embodiments of the data processing equipment of confession, the data source got can be grouped by data processing equipment, tool
The data source obtained from same device can be divided into one group by body, or the data source that the user with identical permission is uploaded point
It is one group, or is by other means grouped data source, this is not limited here.Correspondingly, dependence graph may be used also
With comprising Data Identification, which is used to identify the data source for belonging to same group.
The data processing equipment in the embodiment of the present invention is described from the angle of function module above, below from functional hardware
Angle introduce the data processing equipment in the embodiment of the present invention, referring to Fig. 10, Figure 10 is in the embodiment of the present invention at data
Manage the structural schematic diagram of device 80.Data processing equipment 80 may include input equipment 810, output equipment 820,830 and of processor
Memory 840.Output equipment in the embodiment of the present invention can be display equipment.
Memory 840 may include read-only memory and random access memory, and provide instruction sum number to processor 830
According to.The a part of of memory 840 can also include nonvolatile RAM (Non-Volatile Random
Access Memory, NVRAM).
Memory 840 stores following element, executable modules or data structures either their subset or
Their superset：
Operational order：Including various operational orders, for realizing various operations.
Operating system：Including various system programs, for realizing various basic businesses and the hardware based task of processing.
Processor 830 is used in the embodiment of the present invention：
Obtain data source and the corresponding output result of the data source；
Determine that the dependence between data source and the output result, the dependence include from the data source
Each data source to it is described output result between calculating job relations；
Dependence graph is generated according to the dependence, the dependence graph includes corresponding to each data source
Data source nodes calculate jobs node and node connector, and the node connector is used to be directed toward second node from first node,
The first node is data source nodes or calculates jobs node, and the second node is to calculate jobs node, second section
The corresponding calculating job dependence of point is in the corresponding data source of the first node or calculates job result.
Processor 830 controls the operation of data processing equipment 80, and processor 830 can also be known as central processing unit
(Central Processing Unit, CPU).Memory 840 may include read-only memory and random access memory, and
Instruction and data is provided to processor 830.The a part of of memory 840 can also include NVRAM.In specific application, data
The various components of processing unit 80 are coupled by bus system 850, wherein bus system 850 except include data/address bus it
Outside, it can also include power bus, controlling bus and status signal bus in addition etc..It, will in figure but for the sake of clear explanation
Various buses are all designated as bus system 850.
The method that the embodiments of the present invention disclose can be applied in processor 830, or be realized by processor 830.
Processor 830 may be a kind of IC chip, the processing capacity with signal.During realization, the above method it is each
Step can be completed by the integrated logic circuit of the hardware in processor 830 or the instruction of software form.Above-mentioned processing
Device 830 can be general processor, digital signal processor (Digital Signal Processing, DSP), special integrated
Circuit (Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components.It may be implemented or execute disclosed each method, step and the logic diagram in the embodiment of the present invention.It is general
Processor can be microprocessor or the processor can also be any conventional processor etc..In conjunction with institute of the embodiment of the present invention
The step of disclosed method, can be embodied directly in hardware decoding processor and execute completion, or with the hardware in decoding processor
And software module combination executes completion.Software module can be located at random access memory, and flash memory, read-only memory may be programmed read-only
In the storage medium of this fields such as memory or electrically erasable programmable memory, register maturation.The storage medium is located at
The step of memory 840, processor 830 reads the information in memory 840, the above method is completed in conjunction with its hardware.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In several embodiments provided herein, it should be understood that disclosed system, device and method can be with
It realizes by another way.For example, the apparatus embodiments described above are merely exemplary, for example, the unit
It divides, only a kind of division of logic function, formula that in actual implementation, there may be another division manner, such as multiple units or component
It can be combined or can be integrated into another system, or some features can be ignored or not executed.Another point, it is shown or
The mutual coupling, direct-coupling or communication connection discussed can be the indirect coupling by some interfaces, device or unit
It closes or communicates to connect, can be electrical, machinery or other forms.
The unit illustrated as separating component may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, you can be located at a place, or may be distributed over multiple
In network element.Some or all of unit therein can be selected according to the actual needs to realize the mesh of this embodiment scheme
's.
In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, it can also
It is that each unit physically exists alone, it can also be during two or more units be integrated in one unit.Above-mentioned integrated list
The form that hardware had both may be used in member is realized, can also be realized in the form of SFU software functional unit.
If the integrated unit is realized in the form of SFU software functional unit and sells or use as independent product
When, it can be stored in a computer read/write memory medium.Based on this understanding, technical scheme of the present invention is substantially
The all or part of the part that contributes to existing technology or the technical solution can be in the form of software products in other words
It embodies, which is stored in a storage medium, including some instructions are used so that a computer
Equipment (can be personal computer, server or the network equipment etc.) executes the complete of each embodiment the method for the present invention
Portion or part steps.And storage medium above-mentioned includes：USB flash disk, mobile hard disk, read-only memory (full name in English：Read-Only
Memory, english abbreviation：ROM), random access memory (full name in English：Random Access Memory, english abbreviation：
RAM), the various media that can store program code such as magnetic disc or CD.
The above, the above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although with reference to before
Stating embodiment, invention is explained in detail, it will be understood by those of ordinary skill in the art that：It still can be to preceding
The technical solution recorded in each embodiment is stated to modify or equivalent replacement of some of the technical features；And these
Modification or replacement, the spirit and scope for various embodiments of the present invention technical solution that it does not separate the essence of the corresponding technical solution.
Claims (14)
1. a kind of data processing method, which is characterized in that including：
Obtain data source and the corresponding output result of the data source；
Determine that the dependence between data source and the output result, the dependence include each number from the data source
According to source to the calculating job relations between the output result；
Dependence graph is generated according to the dependence, the dependence graph includes the data corresponding to each data source
Source node calculates jobs node and node connector, and the node connector is used to be directed toward second node from first node, described
First node is data source nodes or calculates jobs node, and the second node is to calculate jobs node, the second node pair
The calculating job dependence answered is in the corresponding data source of the first node or calculates job result.
2. according to the method described in claim 1, its spy is, after the generation dependence graph according to the dependence
Including：
Receive inquiry instruction of the user to destination node in the dependence graph；
Determine the corresponding nodal information of the destination node；
The nodal information is returned to the user.
3. according to the method described in claim 2, it is characterized in that, when the destination node be calculate jobs node when, it is described
Nodal information includes：The corresponding title for calculating operation of the destination node, the corresponding wound for calculating operation of the destination node
It builds the time, the corresponding historical operation number for calculating operation of the destination node, the destination node is corresponding to calculate operation
Start the time, the corresponding queuing for calculating operation of the destination node takes, and the destination node is corresponding to calculate holding for operation
Row takes, and the corresponding history for calculating operation of the destination node starts the time, and the destination node is corresponding to calculate operation
History is lined up the corresponding history execution for calculating operation of the time-consuming or described destination node and takes.
4. according to the method described in claim 2, it is characterized in that, when the destination node be data source nodes when, the section
Putting information includes：The corresponding data source of the destination node, the creation time of the corresponding data source of the destination node.
5. according to the method described in claim 1, it is characterized in that, it is described according to the dependence generate dependence graph it
After include：
It receives user jobs node is calculated to target in the dependence graph and re-run instruction；
According to it is described re-run instruction and execute the target again calculate the corresponding calculating operation of jobs node；
To the result executed again described in user return.
6. the method according to any one of claims 1 to 5, it is characterized in that, the acquisition data source and the data
The corresponding output result in source includes：
Determine that the target of user's selection calculates operation；
It obtains and calculates data source and output result of the operation with incidence relation with the target.
7. the method according to any one of claims 1 to 5, it is characterized in that, the dependence graph also includes state
Mark, the status indicator are used to identify the calculating jobs node or the corresponding shape of data source nodes in the dependence graph
State, the state include：It operates successfully, operating delay, operation failure or operation are not actuated.
8. a kind of data processing equipment, which is characterized in that including：
Acquisition module, for obtaining data source and the corresponding output result of the data source；
First determining module, for determining the dependence between data source and the output result, the dependence includes
From each data source in the data source to the calculating job relations the output result；
Generation module, for generating dependence graph according to the dependence, the dependence graph includes each data
Data source nodes, calculating jobs node corresponding to source and node connector, the node connector from first node for referring to
To second node, the first node is data source nodes or calculates jobs node, and the second node is to calculate jobs node,
The corresponding calculating job dependence of the second node is in the corresponding data source of the first node or calculates job result.
9. device according to claim 8, spy are, described device further includes：
First receiving module, the inquiry instruction for receiving user to destination node in the dependence graph；
Second determining module, for determining the corresponding nodal information of the destination node；
First sending module, for returning to the nodal information to the user.
10. device according to claim 9, which is characterized in that described when the destination node is to calculate jobs node
Nodal information includes：The corresponding title for calculating operation of the destination node, the corresponding wound for calculating operation of the destination node
It builds the time, the corresponding historical operation number for calculating operation of the destination node, the destination node is corresponding to calculate operation
Start the time, the corresponding queuing for calculating operation of the destination node takes, and the destination node is corresponding to calculate holding for operation
Row takes, and the corresponding history for calculating operation of the destination node starts the time, and the destination node is corresponding to calculate operation
History is lined up the corresponding history execution for calculating operation of the time-consuming or described destination node and takes.
11. device according to claim 9, which is characterized in that when the destination node is data source nodes, the section
Putting information includes：The corresponding data source of the destination node, the creation time of the corresponding data source of the destination node.
12. device according to claim 8, which is characterized in that described device further includes：
Second receiving module calculates jobs node to target in the dependence graph for receiving user and re-runs instruction；
Execution module executes the corresponding calculating work of the target calculating jobs node again for re-running instruction according to
Industry；
Second sending module, for the result that executes again described in being returned to the user.
13. the device according to any one of claim 8 to 12, which is characterized in that the acquisition module includes：
Determination unit, for determining that the target of user's selection calculates operation；
Acquiring unit calculates data source and output result of the operation with incidence relation for obtaining with the target.
14. the device according to any one of claim 8 to 12, which is characterized in that the dependence graph also includes shape
State identifies, and the status indicator is used to identify the calculating jobs node or the corresponding shape of data source nodes in the dependence graph
State, the state include：It operates successfully, operating delay, operation failure or operation are not actuated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710103390.9A CN108509440A (en) | 2017-02-24 | 2017-02-24 | A kind of data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710103390.9A CN108509440A (en) | 2017-02-24 | 2017-02-24 | A kind of data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108509440A true CN108509440A (en) | 2018-09-07 |
Family
ID=63372949
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710103390.9A Pending CN108509440A (en) | 2017-02-24 | 2017-02-24 | A kind of data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108509440A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109598407A (en) * | 2018-10-26 | 2019-04-09 | 阿里巴巴集团控股有限公司 | A kind of execution method and device of operation flow |
| CN109800278A (en) * | 2018-12-29 | 2019-05-24 | 亚信科技（南京）有限公司 | Data assets map application method, device, computer equipment and storage medium |
| CN110032554A (en) * | 2019-04-10 | 2019-07-19 | 北京字节跳动网络技术有限公司 | Management method, device, storage medium and the electronic equipment of data warehouse table |
| CN113569184A (en) * | 2021-07-16 | 2021-10-29 | 众安在线财产保险股份有限公司 | Configurable data calculation method, device, equipment and computer readable medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101147187A (en) * | 2005-04-19 | 2008-03-19 | 国际商业机器公司 | System and method for managing complex relationships over distributed heterogeneous data sources |
| CN102239458A (en) * | 2008-12-02 | 2011-11-09 | 起元技术有限责任公司 | Visualizing relationships between data elements |
| CN106293928A (en) * | 2015-06-05 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of overall situation task node dependence method for visualizing, device and system |
-
2017
- 2017-02-24 CN CN201710103390.9A patent/CN108509440A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101147187A (en) * | 2005-04-19 | 2008-03-19 | 国际商业机器公司 | System and method for managing complex relationships over distributed heterogeneous data sources |
| CN102239458A (en) * | 2008-12-02 | 2011-11-09 | 起元技术有限责任公司 | Visualizing relationships between data elements |
| CN106293928A (en) * | 2015-06-05 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of overall situation task node dependence method for visualizing, device and system |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109598407A (en) * | 2018-10-26 | 2019-04-09 | 阿里巴巴集团控股有限公司 | A kind of execution method and device of operation flow |
| CN109598407B (en) * | 2018-10-26 | 2024-04-05 | 创新先进技术有限公司 | Method and device for executing business process |
| CN109800278A (en) * | 2018-12-29 | 2019-05-24 | 亚信科技（南京）有限公司 | Data assets map application method, device, computer equipment and storage medium |
| CN110032554A (en) * | 2019-04-10 | 2019-07-19 | 北京字节跳动网络技术有限公司 | Management method, device, storage medium and the electronic equipment of data warehouse table |
| CN113569184A (en) * | 2021-07-16 | 2021-10-29 | 众安在线财产保险股份有限公司 | Configurable data calculation method, device, equipment and computer readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108595157B (en) | Block chain data processing method, device, equipment and storage medium | |
| CN108509440A (en) | A kind of data processing method and device | |
| US9471470B2 (en) | Automatically recommending test suite from historical data based on randomized evolutionary techniques | |
| EP3468095A1 (en) | Transaction selection device for selecting blockchain transactions | |
| CN109544293A (en) | Trading limit processing method and processing device | |
| CN108198076A (en) | A kind of financial investment method, apparatus, terminal device and storage medium | |
| US8683027B2 (en) | Utilization of uncertainty dependency relationships between items in a data stream | |
| EP3018581B1 (en) | Data staging management system | |
| CN106648839A (en) | Method and device for processing data | |
| CN106095511A (en) | A kind of server updating method and apparatus | |
| CN109657240A (en) | Determine the method, apparatus, equipment and medium of fault type | |
| US8539496B1 (en) | Method and apparatus for configuring network systems implementing diverse platforms to perform business tasks | |
| CN115357897A (en) | Open source software identification method and device | |
| CN112181964A (en) | Business notification duplicate removal method, device, server and storage medium | |
| CN108876339B (en) | E-pet transaction method and device based on certificate | |
| CN116860344A (en) | Flow management method, system, equipment and medium | |
| CN111861100A (en) | Work order processing method and device based on process scoring | |
| CN116521945A (en) | Method for constructing fund association diagram for block chain encrypted currency transaction traceability and control system | |
| CN113139102B (en) | Data processing method, device, nonvolatile storage medium and processor | |
| CN114860608A (en) | Scene construction based system automation testing method, device, equipment and medium | |
| CN111130921B (en) | Method and device for processing performance index of core network element | |
| US20220046339A1 (en) | Data collection device, data collection method, and program | |
| CN109871331A (en) | Software function trial and error method of decision analysis and device | |
| CN110493058A (en) | The construction method and device of network topology structure, storage medium, terminal | |
| CN109246125A (en) | A kind of Host Security condition evaluation system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180907 |
|
| RJ01 | Rejection of invention patent application after publication |