CN113495805B - Data processing method, node determining method, device, equipment and medium - Google Patents
Data processing method, node determining method, device, equipment and medium Download PDFInfo
- Publication number
- CN113495805B CN113495805B CN202010197455.2A CN202010197455A CN113495805B CN 113495805 B CN113495805 B CN 113495805B CN 202010197455 A CN202010197455 A CN 202010197455A CN 113495805 B CN113495805 B CN 113495805B
- Authority
- CN
- China
- Prior art keywords
- node
- data
- target
- brushing
- data table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 238000003672 processing method Methods 0.000 title abstract description 16
- 230000001680 brushing effect Effects 0.000 claims abstract description 155
- 238000011010 flushing procedure Methods 0.000 claims abstract description 132
- 238000005192 partition Methods 0.000 claims description 42
- 238000012545 processing Methods 0.000 claims description 29
- 238000011144 upstream manufacturing Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 7
- 238000012216 screening Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 18
- 238000010586 diagram Methods 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 9
- 238000001514 detection method Methods 0.000 description 5
- 238000007689 inspection Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000012827 research and development Methods 0.000 description 3
- 230000002411 adverse Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 238000009960 carding Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1438—Restarting or rejuvenating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1471—Saving, restoring, recovering or retrying involving logging of persistent data for recovery
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, a node determining method, a device, equipment and a medium. Firstly, acquiring a brushing node set related to data updating, and configuration information of brushing nodes in the brushing node set; then, determining a target data table with a dependency relationship with the back-flushing node according to the configuration information; and then, when the target data table is detected to meet the first preset condition, displaying the target node corresponding to the target data table, so that a user adds the target node to the back-flushing node set. In this way, the efficiency of data swiping is improved.
Description
Technical Field
The present invention relates to the field of computers, and in particular, to a data processing method, a node determining device, a device, and a medium.
Background
Currently, the process of re-executing computing code in a computing node in a computing platform to produce result data and deposit the result data into a data table is referred to as "data flushing". For example, a data table stores the last 30 days of result data, the computing node schedules operation every day and generates the last 1 day of data, and if the original computing code is modified and the data in the data table is required to be the latest computing code of the computing result data, the data needs to be refreshed, that is, the latest computing code after modification is used for recalculating and generating the last 30 days of result data.
In the data refreshing mode in the current stage, whether the data in the data table is missing or not needs to be checked manually, and if the data is missing, the result of refreshing the data is empty, so that the data refreshing needs to be performed again. Therefore, the method is time-consuming and labor-consuming, and is difficult to ensure that the inspection is error-free, and the accuracy of the finally obtained result data is low.
Disclosure of Invention
The embodiment of the invention provides a data processing method, a device, equipment and a medium, which are used for improving the data brushing efficiency.
In a first aspect, an embodiment of the present invention provides a method for determining a node, where the method may include:
acquiring a set of brushing nodes related to data updating and configuration information of the brushing nodes in the set of brushing nodes;
Determining a target data table according to the configuration information, wherein the target data table has a dependency relationship with the back-flushing node;
and when the target data table is detected to meet the first preset condition, displaying the target node corresponding to the target data table, so that a user adds the target node to the brushing node set.
In a second aspect, an embodiment of the present invention provides a node determining method, which may include:
acquiring configuration information related to the update data;
Displaying at least one node related to the configuration information;
Receiving an operation of selecting a target node from the at least one node by a user;
in response to the operation, the target node is determined to be a swiping node, and the swiping node is added to a swiping node set to update data in a target data table through the swiping nodes in the swiping node set, wherein the target data table has a dependency relationship with the swiping nodes.
In a third aspect, an embodiment of the present invention provides a data processing method, which may include:
acquiring a set of brushing nodes related to data updating and configuration information of the brushing nodes in the set of brushing nodes;
Determining a target data table according to the configuration information, wherein the target data table has a dependency relationship with the back-flushing node;
When the target data table is detected to meet the first preset condition, displaying a target node corresponding to the target data table;
When adding the target node to the brushing node set, obtaining a new brushing node set;
updating the data in the first data table through a new brushing node in the new brushing node set to obtain result data; wherein the first data table comprises a data table having a dependency relationship with the new brushing node.
In a fourth aspect, an embodiment of the present invention provides a node determining apparatus, which may include:
The acquisition module is used for acquiring configuration information related to the update data;
The display module is used for displaying at least one node related to the configuration information;
the receiving module is used for receiving the operation of selecting a target node from the at least one node by a user;
And the processing module is used for responding to the operation, determining the target node as a back-flushing node, adding the back-flushing node to a back-flushing node set, and updating data in a target data table through the back-flushing node in the back-flushing node set, wherein the target data table has a dependency relationship with the back-flushing node.
In a fifth aspect, an embodiment of the present invention provides a node determining apparatus, which may include:
The acquisition module is used for acquiring a brushing node set related to data updating and configuration information of brushing nodes in the brushing node set;
The processing module is used for determining a target data table according to the configuration information, wherein the target data table has a dependency relationship with the back-flushing node;
And the display module is used for displaying the target node corresponding to the target data table when the target data table is detected to meet the first preset condition, so that a user adds the target node to the back-flushing node set.
In a sixth aspect, an embodiment of the present invention provides a data processing apparatus, which may include:
The acquisition module is used for acquiring a brushing node set related to data updating and configuration information of brushing nodes in the brushing node set;
The processing module is used for determining a target data table according to the configuration information, wherein the target data table has a dependency relationship with the back-flushing node;
The display module is used for displaying the target node corresponding to the target data table when the target data table is detected to meet the first preset condition;
The updating module is used for obtaining a new brushing node set when the target node is added to the brushing node set;
the processing module is further used for updating the data in the first data table through a new brushing node in the new brushing node set to obtain result data; wherein the first data table comprises a data table having a dependency relationship with the new brushing node.
In a seventh aspect, embodiments of the present invention provide a computing device, the device comprising: a processor and a memory storing computer program instructions;
the processor, when executing the computer program instructions, implements the node determination method as provided in the first aspect, implements the node determination method as provided in the second aspect, or implements the data processing method as provided in the third aspect.
In an eighth aspect, embodiments of the present invention provide a computer storage medium having stored thereon computer program instructions which, when executed by a processor, implement the node determining method provided in the first aspect, implement the node determining method provided in the second aspect, or the data processing method provided in the second aspect.
The method provided by the embodiment of the invention is suitable for the data back flushing scene of the batch of nodes under the condition of a plurality of nodes. Firstly, acquiring a brushing node set related to data updating and configuration information of brushing nodes in the brushing node set; then, determining a target data table with a dependency relationship with the back-flushing node according to the configuration information; and then, when the automatic detection target data table meets the first preset, displaying the target nodes corresponding to the target data table, so that the user adds the target nodes to the back-flushing node set. Therefore, the back-flushing node can be determined through one-time node circle selection, and repeated work of a user is effectively avoided. In addition, the method provides a brushing tool for automatically determining the target node, and does not need to manually maintain the node so as to improve the efficiency of determining the brushing node, further ensure the efficiency of subsequent data brushing, and save labor cost and calculation cost.
Drawings
In order to more clearly illustrate the technical solution of the embodiments of the present invention, the drawings that are needed to be used in the embodiments of the present invention will be briefly described, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data processing interface provided by an embodiment of the present invention;
fig. 2 is a schematic flow chart of a method for determining a node according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a brush-back type interface according to an embodiment of the present invention;
Fig. 4 is a schematic structural diagram of a node determining apparatus according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a node determining apparatus according to an embodiment of the present invention;
FIG. 7 illustrates a schematic diagram of a computing device provided by an embodiment of the present invention.
Detailed Description
Features and exemplary embodiments of various aspects of the present invention will be described in detail below, and in order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely configured to illustrate the invention and are not configured to limit the invention. It will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the invention by showing examples of the invention.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
Currently, when large data development is performed, various requirements for data back flushing are often met. If the data is refreshed according to the data refresh mode at the present stage, a great deal of time and energy are consumed for data research and development, and various problems are frequent.
The data back-flushing mode in the current stage is mainly suitable for single-node back-flushing. The data back-flushing mode specifically comprises the following steps: and the data in the data table corresponding to the upstream node or the downstream node needs to be checked manually to determine whether the data is missing, and when the data in the data table is checked manually to be absent, the user selects the next node to continue data refreshing.
As can be seen from this, the application scenario of the data flushing scheme is limited, i.e., the data flushing scheme can be applied to a single node. If the data refreshing scene of the batch of nodes is faced, the process is needed to be performed for a plurality of times, and the process depends on manual detection of whether the data is missing or not and the refreshing sequence of the nodes is needed to be manually maintained. Therefore, the whole data back-flushing process is time-consuming and labor-consuming and has low efficiency. In addition, this process requires a significant amount of labor costs and computing resources. Furthermore, because of relying on manual detection of whether the data is missing or not, and the need to manually maintain the operations such as the sequence of the node's flushing, these operations are very prone to error, which results in errors in the flushing result data. In addition, if the data is refreshed by the refresh node without management and control, the cluster computing resource is occupied, and adverse effects are generated on other nodes which normally operate to influence the normal output of service data (such as sales statistics, click volume, inter-node data transmission among points, etc.).
For example, when the data corresponding to a certain data product module is refreshed by the data refreshing method, the nodes corresponding to the data product module have a plurality of nodes, have long period, are mutually dependent and complex, and when the upstream node is required to be refreshed, the next node is called up to continue the data refreshing under the condition that the manual inspection is not problematic after the completion of the data refreshing, thus consuming a great deal of time and being extremely easy to think about confusion, being difficult to ensure the correctness of the data refreshing sequence, and being more difficult to ensure the accuracy of the data. Or after the node code on a certain key node is adjusted, when the node related to the downstream part of the data product needs to be refreshed, the downstream node is difficult to be refreshed only by manual carding because of the number of the downstream nodes being thousands. In addition, the data flushing involves a large number of nodes, if the operation is submitted at the same time and is not managed, a large amount of computing resources of the computing platform are occupied, the other nodes which normally operate on the computing platform are adversely affected, and a large amount of computing resources are consumed by the large amount of data flushing.
Therefore, in order to solve the above problems, the embodiments of the present invention provide a data processing method, a node determining device, and a node determining medium, so that data flushing is simpler and more efficient, the risk of problems caused by data flushing is reduced, the working efficiency of data flushing is improved, and the labor cost and the computing cost are saved.
Fig. 1 shows a schematic diagram of a data processing interface according to an embodiment of the present invention. As shown in fig. 1, a user (such as a data research and development staff member) selects a data update related data refreshing node set on a terminal device according to the requirement of data research and development, wherein the data refreshing node set comprises at least one refreshing node (such as a refreshing node a, a refreshing node B and a refreshing node C) so as to perform data refreshing through the at least one refreshing node.
The terminal device may then display the target option according to the determined brushing node, so that the user selects or writes configuration information (such as task name, brushing period, scheduling type, concurrency, scheduling time, etc. in fig. 1) of the brushing node related to data brushing according to the target option.
And determining a target data table (such as a target data table X) with a dependency relationship with the back-flushing node according to the configuration information selected by the user, and detecting whether the first partition corresponding to the configuration information in the target data table has missing data. And under the condition that the data in the first partition is missing, displaying the target node corresponding to the first partition through the terminal equipment.
Therefore, the upstream node and/or the downstream node, namely the back-flushing node, which have an association relation with the node selected by the user can be determined through one-time node circle selection of the user, and the repeated work of the user is effectively avoided. In addition, the method provided by the embodiment of the invention can provide a brushing tool for automatically determining the target node for the user, and does not need to manually maintain the node so as to improve the efficiency of determining the brushing node, further ensure the efficiency of subsequent data brushing and save labor cost and calculation cost.
Then, when receiving that the user adds the target node to the brushing node set, updating the brushing node set selected by the user to obtain a new brushing node set; and then, updating the data in the first data table through a new refreshing node in the new refreshing node set to obtain result data so as to finish data refreshing. Wherein the first data table comprises a data table having a dependency relationship with the new brushing node.
It should be noted that, in the embodiment of the present invention, nodes such as a brushing node and a new brushing node refer to basic units describing a data analysis and processing process in a computing platform, and the nodes include information such as computing codes, node dependency relationships, node scheduling parameters, and the like. The target data table, the first data table and other data tables refer to data storage objects in a computing platform. The first partition, which may also be referred to as a partition table, designates a partition space when the table is created, that is, designates a certain number of fields in the table as partition columns; most of the data are partition tables in practical application; . The partition can be understood as classification, and different types of data are placed under different catalogues through classification; the classification standard is a partition field, and the partition field can be one or a plurality of partition fields; typically, a service date is used as a partition field, and each service date partition stores data corresponding to a corresponding service date.
In summary, the method provided by the embodiment of the invention is suitable for carrying out data refreshing on a batch of nodes under a plurality of conditions, and determining whether missing data exists in a first partition corresponding to configuration information in a target data table or not is automatically executed through terminal equipment; determining a target node according to the dependency relationship with the back-flushing node so as to automatically complement the back-flushing node; and the back flushing resource management and control and the concurrency control are automatically carried out, so that the risk of manual error of data back flushing is reduced, and the risk of influencing other normal nodes in the computing cluster is reduced. In addition, the embodiment of the invention provides some tooled support for the relevant configuration information, reduces the labor investment as much as possible, and is beneficial to reducing the consumption of computing resources, thereby saving the labor cost and the computing cost.
In addition to the above scenario, the method of the embodiment of the present invention may also be used in some data processing scenarios, such as in urban brain scenarios, where it is necessary to determine the history of some nodes. As shown in detail below, when the terminal obtains configuration information related to update data (or different functions according to different scenes, for example, data related to external services, or calculation data), at least one node related to the configuration information is displayed; receiving an operation of selecting a target node from at least one node by a user; in response to the operation, the target node is determined to be a swiping node, and the swiping node is added to the set of swiping nodes to update data in the target data table with the swiping nodes in the set of swiping nodes, the target data table having a dependency relationship with the swiping nodes. Or directly determines the target node as a functional node related to the scene, for example, a computing node related to computing data, or a service node related to data of external service, etc. Based on this, a data processing method in the embodiment of the present invention will be described in detail. The data processing method can comprise two links, one is a node determining link and the other is a scheduling link for data flushing. Therefore, the data refreshing of various user scenes can be met, and the user is helped to efficiently and safely perform the data refreshing. The node determining method is described in detail below with reference to fig. 2.
Fig. 2 is a flow chart illustrating a method for determining a node according to an embodiment of the present invention. As shown in fig. 2, the node determining method may include steps 210-230, which are specifically described as follows:
First, in step 210, a set of swiping nodes related to data update is obtained, and configuration information of the swiping nodes in the set of swiping nodes is obtained. Next, in step 220, the configuration information determines a target data table, where the target data table has a dependency relationship with the back-flushing node. Furthermore, in step 230, when it is detected that the target data table meets the first preset condition, the target node corresponding to the target data table is displayed, so that the user adds the target node to the set of back-flushing nodes.
Therefore, the back-flushing node can be determined through one-time node circle selection, and repeated work of a user is effectively avoided. In addition, the method provides a brushing tool for automatically determining the target node, and does not need to manually maintain the node so as to improve the efficiency of determining the brushing node, further ensure the efficiency of subsequent data brushing, and save labor cost and calculation cost.
The above steps are described below, respectively.
Firstly, referring to step 210, how to obtain a set of back-flushing nodes related to data update in different application scenarios, the embodiment of the present invention provides the following four modes according to different application scenarios, which are specifically shown as follows:
mode (1): a set of back-flushing nodes is determined from the single node.
Wherein, receiving a first input from a user selecting a single node in a computing platform; in response to the first data, a set of back-flushing nodes is generated from the single node selected by the user. It will be appreciated that only one back-flushing node is included in the set of back-flushing nodes.
Mode (2): and searching the nodes in batches.
And receiving a second input of selecting a plurality of nodes in the computing platform by the user, and responding to the second input, and generating a back-flushing node set according to the plurality of nodes selected by the user. It will be appreciated that the set of back-flushing nodes includes a plurality or even a large number of back-flushing nodes.
Mode (3): an intermediate link node is determined by the originating node and the terminating node.
And receiving a third input of selecting the starting node and the ending node from an interface displayed by the terminal equipment by a user, responding to the third input, screening a brushing node which is positioned between the starting node and the ending node and is related to data updating from a computing platform according to the acquired starting node and ending node, and generating a brushing node set according to the brushing node.
Here, the user only needs to determine the start node and the end node, and the terminal device may determine an intermediate link node existing between the start node and the end node according to the relationship between the two nodes. Thereby, the time for searching the intermediate link node of the user is reduced, and the user operation is performed. It should be noted that, in this scenario, the set of back-flushing nodes may include a start node and a termination node in addition to the intermediate link nodes existing between the start node and the termination node.
Mode (4) generates a set of back-flushing nodes according to upstream nodes and/or downstream nodes of a certain node.
Receiving a fourth input of a user for selecting a first node from a plurality of nodes of the computing platform according to user requirements, and responding to the fourth input, and screening upstream nodes and/or downstream nodes of the first node from the plurality of nodes; taking the upstream node and/or the downstream node as a back-flushing node; and generating a set of back-flushing nodes according to the back-flushing nodes.
Here, in a scenario where the user does not know the upstream node and/or the downstream node of the first node, the upstream node and/or the downstream node may be screened in this manner, and a set of back-flushing nodes may be generated. Therefore, the time for a user to record the upstream node and/or the downstream node of each node is reduced, the method is suitable for a scene of data refreshing by adopting a plurality of refreshing nodes, and the data refreshing efficiency is improved.
It should be noted that, in the process of selecting a swiping node from a plurality of nodes of the computing platform and adding the swiping node to the swiping node set, after the swiping node is added, the node can be searched again and added again, and the process is repeated. Here, after adding the swiping node, the terminal device may create a swiping job, that is, an abstract representation of the swiping node set, representing the node range of the data swiping. The back brushing operation can be repeatedly used in the back brushing operation, and the repeated circle selection of back brushing nodes is avoided.
Based on this, after determining the set of swiping nodes, some necessary information needs to be configured each time data swiping is performed. The embodiment of the invention also provides a mode for acquiring the configuration information of the back flushing nodes in the back flushing node set, which is specifically shown as follows:
Displaying target options corresponding to the configuration information to a user through terminal equipment; and responding to the selection input of the user on the target option, and acquiring configuration information corresponding to the target option, wherein the configuration information comprises a brushing cycle, a scheduling type, concurrency and scheduling time.
Wherein the back flush period is a time corresponding to the data update. The brushing cycle includes a plurality of time intervals that are discontinuous. For example, the user specifies which service time corresponding data to be refreshed based on the refresh data. A plurality of discontinuous service time intervals may be set, such as to recall data from 11 months of 2 years to 12 months of 2 years; or the back brushing period can be designated as [2 x-11 months-2 x-12 months, 2 x-9 months-2 x-10 months ]; still alternatively, the user may specify that only the data on one day per week or one day per month be refreshed, depending on the actual situation.
The scheduling type is used to determine a back-flushing form of the data update. For example, multiple-day parallelism or single-day serialization may be selected; if configured to be multiple days parallel, the data for different service dates will be concurrently flushed; if single day serialization is configured, then the data swiping will be performed serially from small to large on the service date.
Concurrency includes the maximum number of node instances allowed to run simultaneously at the time of data update. For example, when configuration data is updated, i.e. data is refreshed, the maximum number of node instances allowed to run simultaneously; when faster completion of data rollback is desired, the concurrency can be set to be greater to reduce overall rollback time. Here, the node instance refers to an object after the node is instantiated, and each time the node runs, a corresponding node instance is generated. The number of node instances refers to the number of node instances.
The scheduling time is a time interval for performing data update. For example, the configuration allows for a data update execution time interval, where the scheduling time may be configured to be 10:30-23:00, where the swiping task is only allowed to run for this time period every day, and the swiping task is not run for other time periods, so as to avoid preempting the computing resources of other normal computing tasks. The back-flushing task is an object after the back-flushing operation is instantiated, the back-flushing operation mainly defines a back-flushing node set to be back-flushed, the back-flushing task can be understood as one running of the back-flushing node set, and the back-flushing task can be run after configuration information such as a back-flushing period, a scheduling type, a task concurrency number, a priority and the like is added.
In addition, it should be noted that, in the method, the manner related to acquiring the initial set of back-flushing nodes may include:
Acquiring configuration information related to the update data; presenting at least one node associated with the configuration information; receiving an operation of selecting a target node from at least one node by a user; in response to the operation, the target node is determined to be a swiping node, and the swiping node is added to the set of swiping nodes to update data in the target data table with the swiping nodes in the set of swiping nodes, the target data table having a dependency relationship with the swiping nodes.
Next, referring to step 220, the target data table in the embodiment of the present invention may be directly determined according to the configuration information, or may be confirmed according to the configuration information and the data table on the back-flushing node.
Then, referring to step 230, in one possible embodiment, before this step, it may further include: and detecting whether the target data table meets a first preset condition.
Further, determining a first partition corresponding to the configuration information in a target data table; detecting whether missing data exists in the data in the first partition; if the data in the first partition is missing, the target data table meets a first preset condition, so that a user adds the target node to the back-flushing node set.
Here, the terminal device automatically checks whether the partition corresponding to the corresponding service flushing period of the target data table used in the flushing node code exists, and if the data in the partition is missing, the flushed result data may be empty or wrong. Thus, if the data in the partition is missing, the node corresponding to the data table comprising the partition can be determined as a target node, so that a user adds the target node into the brushing node set to perform a brushing task; or the target data table uses the data of the partition corresponding to the last service flushing period for data flushing.
Thus, in one possible embodiment, the target node determined based on the above manner may further include a process of scheduling data back-flushing in the data processing after step 230, and the following process is described in detail in the step 240 and the step 250, which are specifically shown as follows:
step 240, adding the target node to the set of back-flushing nodes to obtain a new set of back-flushing nodes.
Here, two ways are provided in the embodiment of the present invention to obtain the new set of back-flushing nodes.
In the mode (1), in a preset time period after the target node corresponding to the target data table is displayed to the user, if the operation of adding the target node to the brushing node set by the user is not received, the target node can be automatically added to the brushing node set to obtain a new brushing node set.
Mode (2) receives a fifth input from the user to add the target node to the set of swiping nodes, and in response to the fifth data, adds the target node to the set of swiping nodes to obtain a new set of swiping nodes.
And step 250, updating the data in the first data table through the new brushing node in the new brushing node set to obtain result data.
Wherein the first data table comprises a data table having a dependency relationship with the new brushing node.
Here, in one possible embodiment, the step may specifically include:
according to the configuration information, adjusting a brushing code of the new brushing node to obtain a target brushing code; and updating the data in the first data table according to the target back flushing code to obtain result data.
Further, some automatic optimization processes can be performed on codes (or back-flushing codes) of back-flushing nodes (or new back-flushing nodes), for example, multi-cycle merging back-flushing, column-level back-flushing or row-level back-flushing, so as to achieve the purposes of reducing risks and saving cost.
As shown in fig. 3, in the embodiment of the present invention, the brushing code of the new brushing node may be adjusted according to different manners to obtain a target brushing code, which is specifically as follows:
scene (1): multi-period merging back-flushing
And when the configuration information comprises data corresponding to a plurality of time intervals, adjusting the date partition field of the back flushing code to obtain a target back flushing code comprising a dynamic partition field.
For example: if a certain brushing node is to brush back the service data of a plurality of brushing cycles, the corresponding code of the brushing node is executed a plurality of times, so that a plurality of node instances are generated, and each node instance only uses the input data of a different brushing cycle. For multiple brush-back periods, the user may choose to employ multi-period merge brush-back. Thus, the date partition field of the back-flushing code can be adjusted, namely the original node code is automatically rewritten, and the service date partition field is used as a dynamic partition field. Thus, under the condition that the brushing codes comprising the dynamic partition fields are obtained, the brushing codes of the nodes are only needed to be executed once, and one node instance is generated to finish the service data brushing of a plurality of brushing periods, so that the consumption of computing resources and the time consumption for running can be reduced.
Scene (2): column level back brush
And when the configuration information comprises data indicating updating of at least one column of the first data table, adjusting the brushing code to obtain a target brushing code comprising the column code of the at least one column.
For example: for some data tables, only partial column data in the tables need to be refreshed, for example, newly added columns or partial column data have abnormality and need to be refreshed, a user can select column-level refresh, so that code rewriting can be automatically performed according to the original node code to generate codes for only refreshing partial column data, and the time consumption and the calculation resource consumption of data refresh are greatly reduced because only partial column data are calculated.
Scene (3): line level back brush
Wherein the method comprises adjusting the brushing code when the configuration information comprises data indicating updating at least one row of the first data table, resulting in a target brushing code comprising column codes of at least one row.
For example, in some cases, only data of part of the rows need to be refreshed, for example, a certain data table is transaction statistics data of 100 thousands of merchants, when data corresponding to 10 merchants is found to be problematic and needs to be refreshed, a user can select row-level refresh, so that code is automatically rewritten according to an original node code to generate a code for only refreshing part of the rows, thus the data amount participating in calculation is greatly reduced, and the time consumption and the calculation resource consumption for refreshing are greatly reduced as in column-level refresh.
Then, after the back-flushing task is created, corresponding instance resource information and node instance dependency relationships need to be generated, and scheduling execution is performed according to the instance relationship graph. The node instance information in the embodiment of the present invention may include instance dependency relationships and/or instance resource information, so that the embodiment of the present invention provides a manner how to determine instance dependency relationships and instance resource information, which is specifically shown as follows:
(1) Determining instance dependencies
Acquiring a first dependency relationship of a data table generated by actual operation of a new brushing node;
and determining the instance dependency relationship of the new brushing node according to the first dependency relationship and the second dependency relationship recorded in the new brushing node.
In this embodiment of the present invention, the dependency completion may be referred to as dependency completion, that is, the dependency relationship of the node instance is supplemented according to the dependency relationship of the data table generated by the actual operation of the swiping node in the past 30 days, in addition to the explicitly defined dependency relationship in the swiping node.
For example, for the last N Days (ND), the natural weeks (CW), the natural months (CM), and nodes that rely on the current service date minus the 2 days (T-2) data, typically have no cross-day dependencies, which in practice require data that depends on multiple service dates of upstream nodes, the order of execution of the node instances may not be logical if no node instance relationship completions are performed, resulting in erroneous final back-flushing result data.
(2) Determining instance resource information
Acquiring a node instance of a new brushing node operated when updating data;
When the resources occupied by the running node instance meet the second preset condition, determining that the information corresponding to the resources occupied by the node instance is instance resource information.
In this embodiment of the present invention, resource management may be referred to as resource control, that is, defining an upper limit of computing resources that can be used by data flushing according to a time period, and allocating computing resources according to historical resource consumption conditions of a flushing task. For example, the low-consumption task obtains more computing resources, so that efficiency is improved, and meanwhile, stability of a computing cluster is prevented from being influenced. In addition, concurrent control, namely control the node instance running at the same time, so as to reduce occupation of the scheduling resources of the computing platform, and submit the computing platform to execute when the node instance meets the running condition, and output result data.
In summary, the data processing method provided by the invention is suitable for the scene of batch node data refreshing in most cases, repeated work is avoided by one-time node circle selection, the node refreshing sequence is not required to be manually maintained, and the tool support of partition automatic inspection, multi-period merging refreshing, column-level or row-level refreshing is provided, so that the efficiency of data refreshing work is improved. In addition, the partition inspection, the automatic completion of the dependency relationship of the brushing nodes and the automatic control of brushing resources and concurrent control can be performed automatically, so that the risk of manual error of data brushing is reduced, and the risk of influencing other normal nodes in the computing cluster is reduced. Therefore, the embodiment of the invention provides a plurality of columns of tool support, reduces the labor investment as much as possible, and is beneficial to reducing the consumption of computing resources, thereby saving the labor cost and the computing cost.
Based on the above node determining method, the embodiment of the present invention further provides a node determining device, which can be specifically described with reference to fig. 4.
Fig. 4 is a schematic structural diagram of a node determining apparatus according to an embodiment of the present invention.
As shown in fig. 4, the node determining apparatus 40 may include:
An obtaining module 401, configured to obtain a set of brushing nodes related to data update, and configuration information of the brushing nodes in the set of brushing nodes;
a processing module 402, configured to determine a target data table according to the configuration information, where the target data table has a dependency relationship with the back-flushing node;
And the display module 403 is configured to display the target node corresponding to the target data table when it is detected that the target data table meets the first preset condition, so that the user adds the target node to the set of back-flushing nodes.
Therefore, the back-flushing node can be determined through one-time node circle selection, and repeated work of a user is effectively avoided. In addition, the method provides a brushing tool for automatically determining the target node, and does not need to manually maintain the node so as to improve the efficiency of determining the brushing node, further ensure the efficiency of subsequent data brushing, and save labor cost and calculation cost.
In one possible embodiment, the obtaining module 401 may be specifically configured to obtain a start node and a stop node in the plurality of nodes; screening a brushing node which is positioned between the starting node and the termination node and is related to data updating according to the starting node and the termination node; and generating a set of back-flushing nodes according to the back-flushing nodes.
In another possible embodiment, the obtaining module 401 may be specifically configured to obtain a first node of the plurality of nodes; screening upstream nodes and/or downstream nodes of the first node from a plurality of nodes; taking the upstream node and/or the downstream node as a back-flushing node; and generating a set of back-flushing nodes according to the back-flushing nodes.
In addition, the obtaining module 401 in the embodiment of the present invention may be further configured to display, through the display module 403, a target option corresponding to the configuration information; and responding to the selection input of the user on the target option, and acquiring configuration information corresponding to the target option, wherein the configuration information comprises a brushing cycle, a scheduling type, concurrency and scheduling time.
The data updating method comprises the steps of updating data, wherein a back brushing period is corresponding to data updating, and comprises a plurality of discontinuous time intervals; the scheduling type is used for determining a back-flushing form of the data update; concurrency includes the maximum number of node instances allowed to run simultaneously when data is updated; the scheduling time is a time interval for performing data update.
In addition, the node determining apparatus 40 in the embodiment of the present invention may further include a detecting module 404, configured to detect whether the target data table meets the first preset condition.
The detection module 404 may be specifically configured to determine a first partition corresponding to the configuration information in the target data table; detecting whether missing data exists in the data in the first partition; if the data in the first partition is missing, the target data table meets a first preset condition.
The processing module 402 in the embodiment of the present invention may also be configured to add the target node to the set of brushing nodes to obtain a new set of brushing nodes; updating the data in the first data table through a new brushing node in the new brushing node set to obtain result data; wherein the first data table comprises a data table having a dependency relationship with the new brushing node.
In one possible embodiment, the processing module 402 may be specifically configured to adjust, according to the configuration information, a brushing code of the new brushing node to obtain a target brushing code; and updating the data in the first data table according to the target back flushing code to obtain result data.
Further, the processing module 402 may be specifically configured to, when the configuration information includes data indicating updating corresponding to a plurality of time intervals, adjust a date partition field of the callback code to obtain a target callback code including a dynamic partition field; or when the configuration information comprises data indicating to update at least one column of the first data table, adjusting the back flush code to obtain a target back flush code comprising the column code of at least one column; or when the configuration information includes data indicating updating of at least one row of the first data table, adjusting the brushing code to obtain a target brushing code including a column code of at least one row.
In another possible embodiment, the processing module 402 may be specifically configured to determine node instance information of the new brushing node; and updating the data in the first data table according to the target back-flushing code and the node instance information to obtain result data.
Further, in the case where the node instance information includes an instance dependency, the processing module 402 may be specifically configured to obtain a first dependency of a data table generated by the actual operation of the new brushing node; and determining the instance dependency relationship of the new brushing node according to the first dependency relationship and the second dependency relationship recorded in the new brushing node.
In the case where the node instance information includes instance resource information, the processing module 402 may be specifically configured to obtain a node instance that runs a new brushing node when updating data; when the resources occupied by the running node instance meet the second preset condition, determining that the information corresponding to the resources occupied by the node instance is instance resource information.
In addition, based on the data processing method, the embodiment of the invention also provides a data processing device, which can be specifically described with reference to fig. 5. Fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention.
As shown in fig. 5, the data processing apparatus 50 may include:
the obtaining module 501 is configured to obtain a set of swiping nodes related to data update, and configuration information of the swiping nodes in the set of swiping nodes.
The processing module 502 is configured to determine a target data table according to the configuration information, where the target data table has a dependency relationship with the back-flushing node.
And a display module 503, configured to display a target node corresponding to the target data table when it is detected that the target data table meets the first preset condition.
An updating module 504 is configured to obtain a new set of swiping nodes when the target node is added to the set of swiping nodes.
The processing module 502 is further configured to update data in the first data table by using a new brushing node in the new brushing node set to obtain result data; wherein the first data table comprises a data table having a dependency relationship with the new brushing node.
Based on the above method, the embodiment of the present invention further provides a node determining device, which may be specifically described with reference to fig. 6. Fig. 6 shows a schematic structural diagram of a node determining apparatus according to an embodiment of the present invention.
As shown in fig. 6, the node determining apparatus 60 may include:
An obtaining module 601, configured to obtain configuration information related to the update data;
a display module 602, configured to display at least one node related to the configuration information;
A receiving module 603, configured to receive an operation of selecting a target node from at least one node by a user;
The processing module 604 is configured to determine, in response to the operation, the target node as a swiping node, and add the swiping node to the set of swiping nodes to update data in the target data table with the swiping node in the set of swiping nodes, the target data table having a dependency relationship with the swiping node. Thus, the data updating related set of the back flushing nodes and the configuration information of the back flushing nodes in the set of the back flushing nodes are obtained; then, determining a target data table with a dependency relationship with the back-flushing node according to the configuration information; and then, when the automatic detection target data table meets the first preset, displaying the target nodes corresponding to the target data table, so that the user adds the target nodes to the back-flushing node set. Therefore, the back-flushing node can be determined through one-time node circle selection, and repeated work of a user is effectively avoided. In addition, the method provides a brushing tool for automatically determining the target node, and does not need to manually maintain the node so as to improve the efficiency of determining the brushing node, further ensure the efficiency of subsequent data brushing, and save labor cost and calculation cost.
The data processing apparatus, the node determining apparatus, and the data processing method and the node determining method of the embodiments of the present invention described in connection with fig. 1 to 3 according to the embodiments of the present invention may be implemented by a computing device. The computing device, as shown in fig. 7, may include a processor 701 and a memory 702 storing computer program instructions.
In particular, the processor 701 may comprise a Central Processing Unit (CPU), or an Application SPECIFIC INTEGRATED Circuit (ASIC), or may be configured as one or more integrated circuits that implement embodiments of the present application.
Memory 702 may include mass storage for data or instructions. By way of example, and not limitation, memory 702 may include a hard disk drive (HARD DISK DRIVE, HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or universal serial bus (universal serial bus, USB) drive, or a combination of two or more of these. The memory 702 may include removable or non-removable (or fixed) media, where appropriate. The memory 702 may be internal or external to the integrated gateway device, where appropriate. In a particular embodiment, the memory 702 is a non-volatile solid state memory. In a particular embodiment, the memory 702 includes Read Only Memory (ROM). The ROM may be mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically Erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these, where appropriate.
The processor 701 implements the data processing method or the node determination method in the above-described embodiment by reading and executing the computer program instructions stored in the memory 702.
The transceiver 703 is mainly used for implementing the communication between each device and other devices in the embodiment of the present invention.
In one example, the device may also include a bus 704. As shown in fig. 7, the processor 701, the memory 702, and the transceiver 703 are connected by a bus 704 and communicate with each other.
Bus 704 includes hardware, software, or both. By way of example, and not limitation, the buses may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a multi-channel architecture (MCa) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus, or a combination of two or more of the above. Bus 703 may include one or more buses, where appropriate. Although embodiments of the application have been described and illustrated with respect to a particular bus, the application contemplates any suitable bus or interconnect.
In one possible embodiment, the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the data processing method or the node determining method of the embodiment of the present invention.
It should be clear that the invention is not limited to the specific arrangements and processes described in the foregoing embodiments and shown in the drawings. For convenience and brevity of description, detailed descriptions of known methods are omitted herein, and specific working processes of the systems, modules and units described above may refer to corresponding processes in the foregoing method embodiments, which are not repeated herein.
It will be apparent to those skilled in the art that the method process of the present invention is not limited to the specific steps described and illustrated, and that various changes, modifications and additions, or equivalent substitutions and order of steps within the scope of the present invention should be included within the scope of the present invention as will be appreciated by those skilled in the art after appreciating the spirit of the present invention.
Claims (14)
1. A node determination method, comprising:
Acquiring a set of back-flushing nodes related to data updating and configuration information of the back-flushing nodes in the set of back-flushing nodes;
determining a target data table according to the configuration information, wherein the target data table has a dependency relationship with the back-flushing node;
When the target data table is detected to meet a first preset condition, displaying a target node corresponding to the target data table, wherein the target data table meets the first preset condition that missing data exists in data in a first partition corresponding to the configuration information in the target data table;
When receiving that a user adds the target node to the back-flushing node set, adding the target node to the back-flushing node set to obtain a new back-flushing node set;
and updating data in a first data table through a new refreshing node in the new refreshing node set to obtain result data, wherein the first data table comprises a data table with a dependency relationship with the new refreshing node.
2. The method of claim 1, wherein the obtaining a set of back-flushing nodes related to data updates comprises:
Acquiring a starting node and a terminating node in a plurality of nodes;
Screening a brushing node which is positioned between the starting node and the termination node and is related to data updating according to the starting node and the termination node;
And generating the brushing node set according to the brushing nodes.
3. The method of claim 1, wherein the obtaining a set of back-flushing nodes related to data updates comprises:
acquiring a first node in a plurality of nodes;
screening out upstream nodes and/or downstream nodes of the first node from the plurality of nodes;
Taking the upstream node and/or the downstream node as the back-flushing node;
And generating the brushing node set according to the brushing nodes.
4. The method of claim 1, wherein obtaining configuration information for a swiping node in the set of swiping nodes comprises:
displaying target options corresponding to the configuration information;
Responding to the selection input of a user on a target option, and acquiring configuration information corresponding to the target option, wherein the configuration information comprises a brushing cycle, a scheduling type, concurrency and scheduling time; wherein,
The back brushing period is time corresponding to data updating, and comprises a plurality of discontinuous time intervals; the scheduling type is used for determining a back-flushing form of data update; the concurrency comprises the maximum number of node instances allowed to run simultaneously during data updating; the scheduling time is a time interval for executing data update.
5. The method of claim 1, wherein the updating the data in the first data table by the new brushing node in the new brushing node set to obtain the result data comprises:
according to the configuration information, adjusting the brushing codes of the new brushing nodes to obtain target brushing codes;
and updating the data in the first data table according to the target back flushing code to obtain result data.
6. The method of claim 5, wherein the adjusting the brushing code of the new brushing node according to the configuration information, to obtain a target brushing code, comprises:
When the configuration information comprises data corresponding to a plurality of time intervals, adjusting a date partition field of the brushing code to obtain a target brushing code comprising a dynamic partition field;
When the configuration information comprises data indicating updating of at least one column of the first data table, adjusting the brushing code to obtain a target brushing code comprising at least one column of column codes;
and when the configuration information comprises data indicating to update at least one row of the first data table, adjusting the brushing codes to obtain target brushing codes comprising column codes of at least one row.
7. The method of claim 5, wherein prior to said updating the data in the first data table according to the target backwash code to obtain result data, the method further comprises:
Determining node instance information of the new back-flushing node;
updating the data in the first data table according to the target brushing code to obtain result data, wherein the updating comprises the following steps:
and updating the data in the first data table according to the target back-flushing code and the node instance information to obtain result data.
8. The method of claim 7, wherein the node instance information includes instance dependencies; the determining node instance information of the new brushing node includes:
acquiring a first dependency relationship of a data table generated by the actual operation of the new brushing node;
and determining the instance dependency relationship of the new brushing node according to the first dependency relationship and the second dependency relationship recorded in the new brushing node.
9. The method of claim 7 or 8, wherein the node instance information includes instance resource information; the determining node instance information of the new brushing node includes:
Acquiring a node instance of the new brushing node running when updating data;
When the resources occupied by the node instance are operated to meet a second preset condition, determining information corresponding to the resources occupied by the node instance as instance resource information.
10. A node determination method, comprising:
acquiring configuration information related to the update data;
Displaying at least one node related to the configuration information;
Receiving operation of selecting a target node from the at least one node by a user, wherein the target node corresponds to a target data table, and data in a first partition corresponding to the configuration information in the target data table has missing data;
in response to the operation, the target node is determined to be a swiping node, and the swiping node is added to a swiping node set to update data in a target data table through the swiping nodes in the swiping node set, wherein the target data table has a dependency relationship with the swiping nodes.
11. A node determining apparatus comprising:
The acquisition module is used for acquiring a brushing node set related to data updating and configuration information of brushing nodes in the brushing node set;
The processing module is used for determining a target data table according to the configuration information, wherein the target data table has a dependency relationship with the back-flushing node;
The display module is used for displaying the target node corresponding to the target data table when the target data table is detected to meet a first preset condition, wherein the target data table meets the first preset condition that missing data exists in data in a first partition corresponding to the configuration information in the target data table;
The processing module is further configured to add the target node to the set of brushing nodes when receiving that the target node is added to the set of brushing nodes by a user, so as to obtain a new set of brushing nodes;
and updating data in a first data table through a new refreshing node in the new refreshing node set to obtain result data, wherein the first data table comprises a data table with a dependency relationship with the new refreshing node.
12. A node determining apparatus comprising:
The acquisition module is used for acquiring configuration information related to the update data;
The display module is used for displaying at least one node related to the configuration information;
the receiving module is used for receiving the operation of selecting a target node from the at least one node by a user, wherein the target node corresponds to a target data table, and the data in a first partition corresponding to the configuration information in the target data table has missing data;
And the processing module is used for responding to the operation, determining the target node as a back-flushing node, adding the back-flushing node to a back-flushing node set, and updating data in a target data table through the back-flushing node in the back-flushing node set, wherein the target data table has a dependency relationship with the back-flushing node.
13. A computing device, wherein the device comprises: a processor and a memory storing computer program instructions;
The processor, when executing the computer program instructions, implements the node determination method according to any one of claims 1-9 or the node determination method according to claim 10.
14. A computer storage medium having stored thereon computer program instructions which, when executed by a processor, implement the node determination method of any of claims 1-9 or the node determination method of claim 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010197455.2A CN113495805B (en) | 2020-03-19 | 2020-03-19 | Data processing method, node determining method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010197455.2A CN113495805B (en) | 2020-03-19 | 2020-03-19 | Data processing method, node determining method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113495805A CN113495805A (en) | 2021-10-12 |
| CN113495805B true CN113495805B (en) | 2024-06-07 |
Family
ID=77993488
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010197455.2A Active CN113495805B (en) | 2020-03-19 | 2020-03-19 | Data processing method, node determining method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113495805B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8812612B1 (en) * | 2012-04-20 | 2014-08-19 | Emc Corporation | Versioned coalescer |
| WO2016192604A1 (en) * | 2015-06-05 | 2016-12-08 | 阿里巴巴集团控股有限公司 | Visualization method, device and system for global task node dependence relationship |
| CN109254969A (en) * | 2018-08-31 | 2019-01-22 | 平安科技（深圳）有限公司 | Tables of data processing method, device, equipment and storage medium |
| CN109885786A (en) * | 2019-01-23 | 2019-06-14 | 聚好看科技股份有限公司 | Data buffer storage processing method, device, electronic equipment and readable storage medium storing program for executing |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6321238B1 (en) * | 1998-12-28 | 2001-11-20 | Oracle Corporation | Hybrid shared nothing/shared disk database system |
-
2020
- 2020-03-19 CN CN202010197455.2A patent/CN113495805B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8812612B1 (en) * | 2012-04-20 | 2014-08-19 | Emc Corporation | Versioned coalescer |
| WO2016192604A1 (en) * | 2015-06-05 | 2016-12-08 | 阿里巴巴集团控股有限公司 | Visualization method, device and system for global task node dependence relationship |
| CN109254969A (en) * | 2018-08-31 | 2019-01-22 | 平安科技（深圳）有限公司 | Tables of data processing method, device, equipment and storage medium |
| CN109885786A (en) * | 2019-01-23 | 2019-06-14 | 聚好看科技股份有限公司 | Data buffer storage processing method, device, electronic equipment and readable storage medium storing program for executing |
Non-Patent Citations (2)
| Title |
|---|
| 基于VMware vSAN集群节点损坏的数据恢复研究;张宇;;电脑知识与技术(01);全文 * |
| 种高效可靠传输方法研究;方协云;中国优秀硕士学位论文全文数据库(第6期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113495805A (en) | 2021-10-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7507602B2 (en) | Data Quality Analysis | |
| CN104866599B (en) | The production method and system of Visual Report Forms | |
| CN110209650A (en) | The regular moving method of data, device, computer equipment and storage medium | |
| CN109840248B (en) | Operation flow optimization method and device and storage medium | |
| CN110134689B (en) | Target group screening method and system based on main body object label change and computer equipment | |
| US7797680B2 (en) | Method and framework for test case management | |
| CN113760476A (en) | Task dependency processing method and related device | |
| CN109298929B (en) | Timed task execution time recommending method, device, equipment and storage medium | |
| CN109634989B (en) | HIVE task execution engine selection method and system | |
| CN111695877A (en) | Computer-implemented project resource management method, system, device and readable medium | |
| CN114139923A (en) | Task relevance analysis method and device and computer readable storage medium | |
| CN102981946A (en) | Extract Transform and Load (ETL) smoke test method | |
| CN113495805B (en) | Data processing method, node determining method, device, equipment and medium | |
| CN111767337B (en) | Block verification method, device and equipment | |
| US20090327020A1 (en) | Intelligent task Deactivation In Project Scheduling Application | |
| CN111461867A (en) | Batch operation simulation operation model establishing method and device | |
| US10409651B2 (en) | Incremental workflow execution | |
| US20140129879A1 (en) | Selection apparatus, method of selecting, and computer-readable recording medium | |
| CN114168661A (en) | Task dependence method and device based on data | |
| CN108681838B (en) | Scheduling plan determining method and device | |
| CN110716798B (en) | PHP (hypertext preprocessor) timing task management method and system | |
| CN113849404A (en) | Management method, device and storage medium for interface test related information | |
| CN113094095A (en) | Agile development progress determination method and device | |
| CN112559641A (en) | Processing method and device of pull chain table, readable storage medium and electronic equipment | |
| CN112131051A (en) | Data backup method and device based on parametric configuration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |