CN111045843B - Distributed data processing method with fault tolerance capability - Google Patents

Distributed data processing method with fault tolerance capability Download PDF

Info

Publication number
CN111045843B
CN111045843B CN201911057644.3A CN201911057644A CN111045843B CN 111045843 B CN111045843 B CN 111045843B CN 201911057644 A CN201911057644 A CN 201911057644A CN 111045843 B CN111045843 B CN 111045843B
Authority
CN
China
Prior art keywords
node
result
data
sub
cluster
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911057644.3A
Other languages
Chinese (zh)
Other versions
CN111045843A (en
Inventor
谢在鹏
张基
李博文
叶保留
毛莺池
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hohai University HHU
Original Assignee
Hohai University HHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hohai University HHU filed Critical Hohai University HHU
Priority to CN201911057644.3A priority Critical patent/CN111045843B/en
Publication of CN111045843A publication Critical patent/CN111045843A/en
Application granted granted Critical
Publication of CN111045843B publication Critical patent/CN111045843B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
    • G06F11/0709—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0793—Remedial or corrective actions

Abstract

The invention discloses a distributed data processing method with fault-tolerant capability, wherein each part of data is operated by three modules for realizing the same function at the same time, and the three modules are mutually independent, and the probability of the same error of the two modules is very small, so the reliability of the system can be greatly improved, and the error of a fault module is masked.

Description

Distributed data processing method with fault tolerance capability
Technical Field
The invention relates to the field of distributed computing, in particular to a distributed data processing method with fault tolerance capability.
Background
The proliferation of data volumes in the big data era has led to a rapid increase in the importance of distributed computing, and in many practical distributed systems, Transient faults (Transient faults) have led to abnormal behavior in computer systems, especially when smaller and smaller transistors and higher power densities of hardware are used to make such faults more frequent. In some large distributed systems, on average, 1% -2% of the nodes fail each day. Such error rates not only reduce the reliability of the distributed system but also affect the performance of the system, and it is important to provide effective fault tolerance in the distributed system.
Conventional fault tolerance techniques in distributed systems include replica techniques and coding techniques, however, these techniques cause a high communication load in implementing fault tolerance. For example, due to the distribution characteristic, the operation related to coding fault tolerance requires multiple nodes in the distributed cluster to cooperate with each other, which may result in a large amount of data transmission and thus high communication overhead. In the MapReduce-based distributed computing framework, a task is roughly divided into three phases of Map, Shuffle and Reduce. Firstly, each node in the Map stage calculates local input data according to a predefined Map function so as to obtain an intermediate result. These intermediate results are then exchanged between the various compute nodes, which use these intermediate values as inputs to the Reduce function to compute the output result. The performance of distributed computing is seriously affected by the high communication load of the Shuffle stage. Therefore, the communication load is a bottleneck for improving the performance of the distributed system, and the traditional scheme has the problem of excessively high communication load.
Disclosure of Invention
In order to solve the above problems, the present invention provides a distributed data processing method with fault tolerance capability.
In order to achieve the purpose of the invention, the invention provides a distributed data processing method with fault tolerance capability, which comprises the following steps:
s10, dividing the N nodes into
Figure BDA0002256948290000011
Sub-cluster to distribute computing tasks to
Figure BDA0002256948290000012
On the sub-clusters, each sub-cluster is subjected to parallel computation; wherein, the symbol
Figure BDA0002256948290000013
Represents rounding down, n represents the number of nodes in a sub-cluster;
s20, dividing the data set M into
Figure BDA0002256948290000014
Partial data sets, respectively sent to
Figure BDA0002256948290000015
On a sub-cluster, the data volume of each sub-cluster is made to be
Figure BDA0002256948290000016
Each sub-cluster divides the data set B of the sub-cluster into K' independent sub-data blocks, wherein each data block uses BiAnd i is more than or equal to 1 and less than or equal to K ', and copying 3 parts of data in each data block to generate 3K' parts of data；
S30, each sub-cluster distributes 3K' data to n nodes of the current sub-cluster according to the set rule, so that each data block in each node local data set is different, if the node is not the samekAnd a nodemWhen the subscript k + m is n +1, then the nodekAnd a nodemThe intersection of the owned data sets is empty, the union is the full set of the current subset cluster data set, and if k + m is not equal to n +1, the nodekAnd a nodemWith and only two identical redundant data
S50, randomly selecting a pair of check node nodes satisfying k + m-n +1 for each sub-clusterkAnd a nodemThe remaining nodes in the sub-cluster perform the operations of steps S51 and S52, which may be performed in parallel:
s51, using CDC algorithm to local and node by each other node in sub-clusterkCoding a first intermediate result corresponding to the redundantly stored data set to obtain a first coding result, and respectively sending the first coding result to the nodek；
S52, using CDC algorithm to local and node by each other node in sub-clustermCoding a second intermediate result corresponding to the redundantly stored data set to obtain a second coding result, and sending the second coding result to the nodem；
S60，nodekAfter receiving all the first encoding results, the first encoding results obtained in the decoding S51 obtain the first decoding result, node, of each piece of datamAfter receiving all the second encoding results, decoding the second encoding results obtained in S52 to obtain second decoding results;
S70，nodekverifying the correctness of the first encoding result of each piece of data according to the first decoding result and the first intermediate result, and if the first encoding result is not correct, returning to execute the step S50, or, the nodemVerifying the correctness of the second encoding result according to the second decoding result and the second intermediate result, and if the second encoding result is not correct, returning to execute the step S50; if the number of the correct coding results in the first coding result is more than or equal to 1 and the number of the correct coding results in the second coding result is positiveThe number of the code result is more than or equal to 1, the nodekCalculating the correct result of the corresponding node according to the correct coding result in the first coding result to obtain a first operation result, and the nodemCalculating the correct result of the corresponding node according to the correct coding result in the second coding result to obtain a second operation result;
and S90, determining the final result of the corresponding sub-cluster according to the corresponding first operation result and the second operation result by each sub-cluster.
In one embodiment, in step S30, when each sub-cluster allocates 3K' copies of data to n nodes of the current sub-cluster according to a set rule, the method further includes:
in each subgroup, each node can be divided into 3K'/n data, and the data set assigned to each node is BjRepresents; if nodekAnd a nodemIf the subscript k + m ≠ n +1, then the intersection of the data sets owned by the subscript k + m ≠ n +1 is empty, the union of the data sets owned by the subscript k + m ≠ n +1 is the full set of the current sub-cluster data set, and if k + m ≠ n +1, then the nodekAnd a nodemWith and only two identical redundant data bi,bjIf B isk∩Bm＝{bi,bjIs defined as
Figure BDA0002256948290000021
Wherein, the symbol
Figure BDA0002256948290000031
Meaning "defined as".
As an embodiment, before step S50, the method further includes:
s40, each node executes a Map function G (-) on each data block of the local data set, and obtains an intermediate result:
Figure BDA0002256948290000032
wherein i represents a data block serial number, t represents a node serial number, and the Map function can be customized by a user aiming at different calculation tasks.
In one embodiment, n has a value of 6 and K' has a value of 8.
As one example, step S50 includes:
each sub-cluster randomly selects a pair of check nodekAnd a nodemAnd k + m is 7, and the rest four nodes in each sub-cluster are recorded as: node(s)s，noden，nodep，nodeqIn each nodes，noden，nodep，nodeqAssociating local data sets with nodeskRedundantly stored data set Rs，k，Rn，k，Rp，k，Rq，kThe first intermediate result calculated at each node is coded by CDC to obtain the coding result u of the first intermediate results，k，un，k，up，k，uq，kDetermining a first coding result, and sending the first coding result to the nodek(ii) a Respectively at the nodes，noden，nodep，nodeqAssociating local data sets with nodesmRedundantly stored data set Rs，m，Rn，m，Rp，m，Rq，mThe second intermediate result calculated at each node is coded by CDC to obtain the coding result u of the second intermediate results，m，un，m，up，m，uq，mI.e., the second encoding result, transmits the second encoding result to the nodem。
As one example, step S70 includes:
if (v)2 s＝v2 k)∧(v3 n＝v3 k) Is formed by (v)1 p＝v1 k)∧(v4 q＝v4 k) Is true, indicating a nodekThe correct number of the coding result in the first coding result is more than or equal to 2 and the nodekAll local intermediate results are correctly calculated, and the operation result r of the current node data set is calculated through a Reduce functionk＝H(v1 k，v2 k，v3 k，v4 k) Obtaining a first operation result; if (v)6 s＝v6 m)∧(v7 n＝v7 m) Is true or (v)5 p＝v5 m)∧(v8 q＝v8 m) Is true, indicating a nodemThe correct number of the coding result in the second coding result is more than or equal to 2 and the nodemAll local intermediate results are correctly calculated, and the operation result r of the current node data set is calculated through a Reduce functionm＝H(v5 m，v6 m，v7 m，v8 m) Obtaining a second operation result; wherein v is2 sRepresents a nodesIntermediate result of data sequence number 2, same as v1 k，v2 k，v3 k，v4 kAll represent a nodekIntermediate result of the last corresponding data sequence number, v3 nRepresents a nodenIntermediate result of the upper data sequence number 3, v1 pRepresents a nodepIntermediate result of the upper data sequence number 1, v4 qRepresents a nodeqIntermediate result of the upper data sequence number 4, v6 sRepresents a nodesIntermediate result of data sequence number 6, same as v5 m，v6 m，v7 m，v8 mAll represent a nodemIntermediate result of the last corresponding data sequence number, v7 nRepresents a nodenIntermediate result of the upper data sequence number 7, v5 pRepresents a nodepIntermediate result of the upper data sequence number 5, v8 qRepresents a nodeqIntermediate results of the upper data sequence number 8, where H (-) denotes a function of the Reduce phase, the effect of which is to combine a plurality of intermediate results into one, the symbol Λ denotes an AND operation, v1 m，v2 m，v3 m，v4 mAll represent a nodemIntermediate result v obtained after calculation of self data block in Map stage5 m，v6 m，v7 m，v8 mAll represent a nodemTo self data in Map stageObtaining an intermediate result after block calculation;
if (v)2 s＝v2 k)∧(v3 n＝v3 k) Is false and (v)1 p＝v1 k)∧(v4 q＝v4 k) If not, then assume the nodekAt least one correct encoding result exists in the first encoding result; if (v)6 s＝v6 m)∧(v7 n＝v7 m) Is false and (v)5 p＝v5 m)∧(v8 q＝v8 m) If not, then assume the nodemAt least one correct encoding result exists in the second encoding result; if v is2 s≠v2 k，v3 n≠v3 k，v1 p≠v1 k，v4 q≠v4 kIf the first encoding result is not correct, the process returns to step S50, or if v is not correct6 s≠v6 m，v7 n≠v7 m，v5 p≠v5 m，v8 q≠v8 mIf it is determined that the second encoding result is not correct, the process returns to step S50.
As one example, after step S70, at nodekThe correct number of coding results in the first coding result of (2) is at least 1, and the nodemWhen the correct number of the encoding results in the second encoding result is at least 1, the method further comprises:
if v is2 s＝v2 kIndicates us，kIf the verification is successful, us，k＝uk，s，v1 k＝v1 s，v2 k＝v2 s(ii) a By passing
Figure BDA0002256948290000041
To obtain v3 p，
Figure BDA0002256948290000042
To obtain v4 qJudgment of
Figure BDA0002256948290000043
If true, it indicates that the error source is nodekLocal data blocks, nodeskThe first coding result sent by other nodes is received correctly to obtain the nodekOf correct result rk＝H(v1 k，v2 k，v3 p，v4 q) Obtaining a first operation result; wherein u isk，sRepresentation storage in nodekAnd ofsThe intermediate result corresponding to the redundant data is encoded by CDC
Figure BDA0002256948290000044
Represents a logical exclusive or;
if v is6 s＝v6 mIndicates us，mIf the verification is successful, us，m＝um，s，v6 m＝v6 s，v5 m＝v5 s(ii) a By passing
Figure BDA0002256948290000045
To obtain v7 p，
Figure BDA0002256948290000046
To obtain v8 qJudgment of
Figure BDA0002256948290000047
If true, it indicates that the error source is nodemLocal data block, nodemThe second coding result sent by other nodes is received correctly to obtain the nodemOf correct result rm＝H(v5 m，v6 m，v7 p，v8 q) Obtaining a second operation result; wherein u ism，sRepresentation storage in nodemAnd ofsThe intermediate result corresponding to the redundant data is encoded using CDC5 sRepresents a nodesIntermediate result of upper data sequence number 5.
The distributed data processing method with fault tolerance capability divides N nodes into
Figure BDA0002256948290000048
Sub-cluster to distribute computing tasks to
Figure BDA0002256948290000049
On a sub-cluster, the data set M is divided into
Figure BDA00022569482900000410
Partial data sets, respectively sent to
Figure BDA0002256948290000051
On the sub-clusters, each sub-cluster distributes 3K' parts of data to n nodes of the current sub-cluster according to a set rule, and thus each sub-cluster randomly selects a pair of check node nodes meeting the condition that K + m is n +1kAnd a nodemThe other nodes in the sub-cluster respectively use the local data set and the nodekRespectively coding the first intermediate result calculated by each node by using CDC (performance data control) of the first intermediate result calculated by each node of the redundantly stored data set to obtain a first coding result, and sending the first coding result to the nodek(ii) a The other nodes in the sub-cluster respectively use the local data set and the nodemRespectively coding second intermediate results calculated by the redundant stored data sets at the respective nodes by using CDC to obtain second coding results, and sending the second coding results to the nodesm(ii) a Make nodekReceiving a first encoding result, decoding said first encoding result to obtain a first decoding result, nodemReceiving a second encoding result, decoding the second encoding result to obtain a second encoding resultSecondly, decoding results; node(s)kVerifying the correctness of the first coding result according to the first decoding result and the first intermediate result, and if the first coding result is not correct, returning to execute each sub-cluster to randomly select a pair of check node nodes meeting the condition that k + m is equal to n +1kAnd a nodemOr a nodemVerifying the correctness of the second coding result according to the second decoding result and the second intermediate result, and if the second coding result is not correct, returning to execute each sub-cluster to randomly select a pair of check node nodes meeting the condition that k + m is equal to n +1kAnd a nodemThe process of (2); if the correct number of the coding results in the first coding result is greater than 1 and the correct number of the coding results in the second coding result is greater than 1, calculating the correct result of the corresponding node according to the correct coding result in the first coding result to obtain a first operation result, and calculating the correct result of the corresponding node according to the correct coding result in the second coding result to obtain a second operation result; therefore, each sub-cluster determines the final result of the corresponding sub-cluster according to the corresponding first operation result and the second operation result, the distributed computation not only keeps the lower communication load of the distributed coding computation, but also enables the backup maintenance and fault recovery efficiency to be close to the copy technology, thereby ensuring the high reliability of the distributed system and reducing the communication load.
Drawings
FIG. 1 is a flow diagram of a distributed data processing method with fault tolerance capabilities, according to an embodiment;
FIG. 2 is a diagram of various sub-clusters of one embodiment;
FIG. 3 is a model structure diagram of a sub-cluster of an embodiment;
FIG. 4 is a flow diagram of a distributed data processing method with fault tolerance capability according to another embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
Referring to fig. 1, fig. 1 is a flowchart illustrating a distributed data processing method with fault tolerance capability according to an embodiment, including the following steps:
s10, dividing the N nodes into
Figure BDA0002256948290000061
Sub-cluster to distribute computing tasks to
Figure BDA0002256948290000062
On the sub-clusters, each sub-cluster is subjected to parallel computation; wherein, the symbol
Figure BDA0002256948290000063
Indicating rounding down and n indicates the number of nodes in a sub-cluster.
Specifically, the value of N may be 6, and at this time, every 6 of the N nodes may be divided into a group, and each group forms a sub-cluster, and the sub-clusters share the same value
Figure BDA00022569482900000611
Sub-clusters, in one example, various sub-clusters may be as described with reference to FIG. 2. The sub-clusters are independent of each other, and the nodes within each sub-cluster may be physically distributed. Distributing computing tasks to
Figure BDA0002256948290000064
On the sub-clusters, each sub-cluster is computed in parallel. The 6 nodes in each sub-cluster can be respectively used as { node1，node2，node3，node4，node5，node6Represents it.
S20, dividing the data set M into
Figure BDA0002256948290000065
Partial data sets, respectively sent to
Figure BDA0002256948290000066
On a sub-cluster, the data volume of each sub-cluster is made to be
Figure BDA0002256948290000067
Each sub-cluster divides the data set B of the sub-cluster into K' independent sub-data blocks, wherein each data block uses BiAnd 1. ltoreq. i.ltoreq.K 'indicates that each data block is copied by 3 parts to generate 3K' parts of data.
The value of K' may be 8, and if n is 6, the data set M is divided into
Figure BDA0002256948290000068
Partial data sets, respectively sent to
Figure BDA0002256948290000069
On a sub-cluster. Each sub-cluster has an amount of data of
Figure BDA00022569482900000610
Meanwhile, each sub-cluster divides the data set B of the sub-cluster into 8 independent sub-data blocks, wherein each data block uses BiAnd i is more than or equal to 1 and less than or equal to 8. Each data block may further be replicated 3 copies (i.e., triple modular redundancy) resulting in 24 copies of data. The triple Modular redundancy TMR (triple Modular redundancy) is a common fault-tolerant technology in software and hardware systems, three modules for realizing the same function are used for operating simultaneously, and the final output of the system is selected by using a voting mechanism. Because the three modules are independent, the probability that the same error occurs in two modules is very small, the reliability of the system can be greatly improved, and the error of a fault module is masked.
S30, dividing each subcluster into 3K' partsDistributing the data blocks to n nodes of the current sub-cluster according to a set rule to ensure that the data blocks in the local data set of each node are different from each other, and if the node is not the samekAnd a nodemWhen the subscript k + m is n +1, then the nodekAnd a nodemThe intersection of the owned data sets is empty, the union is the full set of the current subset cluster data set, and if k + m is not equal to n +1, the nodekAnd a nodemThere are and only two identical redundant data.
Specifically, when the value of K' is 8 and the value of n is 6, each sub-cluster allocates 24 pieces of data to 6 nodes of the sub-cluster, each node can be divided into 4 pieces of data, and a data set owned by the node is represented by BtAnd t is more than or equal to 1 and less than or equal to 6. In the distribution process, the data sets among the nodes meet the following requirements, the data blocks in the local data sets of the nodes are different from each other, and if the nodes do not meet the requirementskAnd a nodemIf k + m ≠ 7, then the intersection of the datasets they have is empty, and the union is the full set of the current sub-cluster datasets, and if k + m ≠ 7, then the nodekAnd a nodemWith and only two identical redundant data bi，bjAt this time, there is Bk∩Bm＝{bi，bjIs defined as Rk，m＝Bk∩Bm(ii) a Where i and j denote the index of the data block. Such as B1＝{b1，b2，b3，b4}，B2＝{b1，b2，b5，b6}，B3＝{b1，b3，b5，b7}，B4＝{b2，b4，b6，b8}，B5＝{b3，b4，b7，b8}，B6＝{b5，b6，b7，b8}。R1，2＝{b1，b2R, e.g. R1，6Is an empty set. Correspondingly, each of the other sub-clusters respectively distributes 24 data to 6 nodes of the corresponding sub-cluster, each node can be divided into 4 data, and the data set owned by the node can also be used as BtT is more than or equal to 1 and less than or equal to 6, and the data sets of all the nodes are fullMeets the corresponding requirements. In an example, the model structure diagram of the sub-cluster may be as shown in fig. 3, where each surface of the solid graph in fig. 3 represents a node, each vertex represents a piece of data, and after the pieces of data are distributed to the nodes of the sub-cluster, local data sets of the nodes are different from each other, specifically, if mode is usedkAnd a nodemWhen the subscript k + m is n +1, then the nodekAnd a nodemThe intersection of the owned data sets is empty, the union is the full set of the current subset cluster data set, and if k + m is not equal to n +1, the nodekAnd a nodemThere are and only two identical redundant data.
In an example, after the step S30, the method further includes: s40, each node executes a Map function G (-) on each data block of the local data set, and obtains an intermediate result:
Figure BDA0002256948290000071
wherein i represents a data block serial number, t represents a node serial number, and the Map function can be customized by a user aiming at different calculation tasks; so that each node can obtain the required intermediate results.
S50, randomly selecting a pair of check node nodes satisfying k + m-n +1 for each sub-clusterkAnd a nodemThe remaining nodes in the sub-cluster perform the operations of steps S51 and S52, which may be performed in parallel:
s51, using CDC algorithm to local and node by each other node in sub-clusterkCoding a first intermediate result corresponding to the redundantly stored data set to obtain a first coding result, and respectively sending the first coding result to the nodek；
S52, using CDC algorithm to local and node by each other node in sub-clustermCoding a second intermediate result corresponding to the redundantly stored data set to obtain a second coding result, and sending the second coding result to the nodem。
The CDC coding is a coding technique for creating coding opportunities using redundancy, such as a node1Encoding the result
Figure BDA0002256948290000072
Is sent to the node2，node2By locally stored redundant data blocks b1Corresponding intermediate result v1 2Decoding of coded data packets
Figure BDA0002256948290000073
The remaining intermediate results v in the coded data packet are obtained2 1(
Figure BDA0002256948290000074
Refers to an exclusive or operation).
In particular, the above steps may be directed to a nodekAnd a nodemThe same operation is performed in both nodes, so that both obtain corresponding encoding results and intermediate results.
S60，nodekAfter receiving all the first encoding results, the first encoding results obtained in the decoding S51 obtain the first decoding result, node, of each piece of datamAfter receiving all the second encoding results, the second encoding results obtained in S52 are decoded to obtain second decoding results.
In the above step, the nodekAnd a nodemWill receive the encoded data packet (e.g., node)kReceived first coding result and nodemThe received second encoded result) is decoded, resulting in an intermediate result therein. node(s)kReceiving a coded packet of data as us，k，un，k，up，k，uq，kWherein u iss，kRepresents a nodesTo nodekEncoding result of (1), un，kRepresents a nodenTo nodekEncoding result of (1), up，kRepresents a nodepTo nodekEncoding result of (1), uq，kRepresents a nodeqTo nodekThe result of the encoding of (1). And a nodekLocally calculates a first intermediate result v1 k，v2 k，v3 k，v4 k. If the following relationship holds: s + n is 7, p + q is 7, since
Figure BDA0002256948290000081
Figure BDA0002256948290000082
Then nodek performs the decoding steps as follows:
Figure BDA0002256948290000083
Figure BDA0002256948290000084
to obtain v2 s，v3 n，v1 p，v4 q；nodemReceiving a coded packet of data as us，m，un，m，up，m，uq，mWherein u iss，mRepresents a nodesTo nodemEncoding result of (1), un，mRepresents a nodenTo nodemEncoding result of (1), up，mRepresents a nodepTo nodemEncoding result of (1), uq，mRepresents a nodeqTo nodemThe result of the encoding of (1). And a nodemLocally calculates a first intermediate result v5 m，v6 m，v7 m，v8 m. Due to the fact that
Figure BDA0002256948290000085
Then nodemThe decoding steps are performed as follows:
Figure BDA0002256948290000086
to obtain v6 s，v7 n，v5 p，v8 q(ii) a Wherein v is1 s，v2 s，v5 s，v6 sRespectively represent a nodesIntermediate results, v, of local storage3 n，v4 n，v7 n，v8 nRespectively represent a nodenIntermediate results, v, of local storage1 p，v3 p，v5 p，v7 pRespectively represent a nodepIntermediate results, v, of local storage2 q，v4 q，v6 q，v8 qRespectively represent a nodeqIntermediate results stored locally.
S70，nodekVerifying the correctness of the first encoding result of each piece of data according to the first decoding result and the first intermediate result, and if the first encoding result is not correct, returning to execute the step S50, or, the nodemVerifying the correctness of the second encoding result according to the second decoding result and the second intermediate result, and if the second encoding result is not correct, returning to execute the step S50; if the number of the correct coding results in the first coding result is more than or equal to 1 and the number of the correct coding results in the second coding result is more than or equal to 1, the nodekCalculating the correct result of the corresponding node according to the correct coding result in the first coding result to obtain a first operation result, and the nodemAnd calculating the correct result of the corresponding node according to the correct coding result in the second coding result to obtain a second operation result.
The above steps can be respectively carried out on the nodekAnd a nodemThe correctness of the coding result is verified by comparing the intermediate result obtained by decoding (namely the corresponding decoding result) with the intermediate result corresponding to the local data set, and meanwhile, the correct intermediate result is calculated.
And S90, determining the final result of the corresponding sub-cluster according to the corresponding first operation result and the second operation result by each sub-cluster.
In particular, when the nodekAnd a nodemCan pass the verification and obtain the correct operation result, i.e. the first operation result rkAnd a second operation result rmThen, each sub-cluster can obtain the final result r ═ H (r) of the corresponding sub-cluster through the Reduce functionk，rm)。
Specifically, the Map function and the Reduce function are two functions of a MapReduce-based distributed computing framework, and two groups of functions can be customized by a user for specific computing tasks respectively so as to perform corresponding processing on big data, so that the distributed coding computing process keeps a low communication load.
The distributed data processing method with fault tolerance capability divides N nodes into
Figure BDA0002256948290000091
Sub-cluster to distribute computing tasks to
Figure BDA0002256948290000092
On a sub-cluster, the data set M is divided into
Figure BDA0002256948290000093
Partial data sets, respectively sent to
Figure BDA0002256948290000094
On the sub-clusters, each sub-cluster distributes 3K' parts of data to n nodes of the current sub-cluster according to a set rule, and thus each sub-cluster randomly selects a pair of check node nodes meeting the condition that K + m is n +1kAnd a nodemThe other nodes in the sub-cluster respectively use the local data set and the nodekRespectively coding the first intermediate result calculated by each node by using CDC (performance data control) of the first intermediate result calculated by each node of the redundantly stored data set to obtain a first coding result, and sending the first coding result to the nodek(ii) a The other nodes in the sub-cluster respectively use the local data set and the nodemRespectively coding second intermediate results calculated by the redundant stored data sets at the respective nodes by using CDC to obtain second coding results, and sending the second coding results to the nodesm(ii) a Make nodekReceiving a first encoding result, decoding said first encoding result to obtain a first decoding result, nodemReceiving a second encoding result, and decoding the second encoding result to obtain a second decoding result; node(s)kVerifying a first coding junction based on the first decoding result and a first intermediate resultIf the first coding result is not correct, returning to execute each sub-cluster to randomly select a pair of check nodes meeting the condition that k + m is equal to n +1kAnd a nodemOr a nodemVerifying the correctness of the second coding result according to the second decoding result and the second intermediate result, and if the second coding result is not correct, returning to execute each sub-cluster to randomly select a pair of check node nodes meeting the condition that k + m is equal to n +1kAnd a nodemThe process of (2); if the correct number of the coding results in the first coding result is more than or equal to 1 and the correct number of the coding results in the second coding result is more than or equal to 1, calculating the correct result of the corresponding node according to the correct coding result in the first coding result to obtain a first operation result, and calculating the correct result of the corresponding node according to the correct coding result in the second coding result to obtain a second operation result; therefore, each sub-cluster determines the final result of the corresponding sub-cluster according to the corresponding first operation result and the second operation result, the distributed computation not only keeps the lower communication load of the distributed coding computation, but also enables the backup maintenance and fault recovery efficiency to be close to the copy technology, thereby ensuring the high reliability of the distributed system and reducing the communication load.
In one embodiment, in step S30, when each sub-cluster allocates 3K' copies of data to n nodes of the current sub-cluster according to a set rule, the method further includes:
in each subgroup, each node can be divided into 3K'/n data, and the data set assigned to each node is BjRepresents; if nodekAnd a nodemIf the subscript k + m ≠ n +1, then the intersection of the data sets owned by the subscript k + m ≠ n +1 is empty, the union of the data sets owned by the subscript k + m ≠ n +1 is the full set of the current sub-cluster data set, and if k + m ≠ n +1, then the nodekAnd a nodemWith and only two identical redundant data bi，bjIf B isk∩Bm＝{bi，bjIs defined as
Figure BDA0002256948290000104
Wherein, the symbol
Figure BDA0002256948290000101
Meaning "defined as".
Specifically, when the value of K' is 8 and the value of n is 6, each sub-cluster allocates 24 pieces of data to 6 nodes of the corresponding sub-cluster, each node can be divided into 4 pieces of data, and a data set owned by the node uses BjJ is more than or equal to 1 and less than or equal to 6. In the distribution process, the data sets among the nodes meet the following requirements, the data blocks in the local data sets of each node are different from each other, and if the nodes do not meet the requirementskAnd a nodemIf k + m ≠ 7, then the intersection of the datasets they have is empty, and the union is the full set of the current sub-cluster datasets, and if k + m ≠ 7, then the nodekAnd a nodemWith and only two identical redundant data bi，bjAt this time, there is Bk∩Bm＝{bi，bjIs defined as Rk，m＝Bk∩Bm(ii) a Where i, j denotes the data block index.
Such as B1＝{b1，b2，b3，b4}，B2＝{b1，b2，b5，b6)，B3＝{b1，b3，b5，b7}，B4＝{b2，b4，b6，b8}，B5＝{b3，b4，b7，b8}，B6＝{b5，b6，b7，b8}。R1，2＝{b1，b2R, e.g. R1，6Is an empty set.
In one embodiment, before step S50, the method further includes:
s40, each node executes a Map function G (-) on each data block of the local data set, and obtains an intermediate result:
Figure BDA0002256948290000102
wherein i represents a data block serial number, t represents a node serial number, and the Map function can be customized by a user aiming at different calculation tasks.
In this embodiment, each node of the sub-cluster may be for each data block b of the local data setiExecuting Map function G (b)i) Get a corresponding set of intermediate results as
Figure BDA0002256948290000103
So that each node can smoothly obtain the required intermediate result. Specifically, the Map function and the Reduce function are functions of MapReduce, the MapReduce is a calculation model for large data parallel processing, specifically, a data set is distributed to each node in the Map stage to realize division of tasks, each node calculates its own data set, the calculation process is represented by a function G (·), and a result obtained after calculation is an intermediate result v. The Reduce phase may merge intermediate results produced by the Map phase. After the Map stage is calculated, the intermediate result needs to be sent to the Reduce node for corresponding operation, so as to obtain the required operation result.
In one embodiment, n has a value of 6 and K' has a value of 8.
In one embodiment, step S50 includes:
each sub-cluster randomly selects a pair of check nodekAnd a nodemAnd k + m is 7, and the rest four nodes in each sub-cluster are recorded as: node(s)s，noden，nodep，nodeqIn each nodes，noden，nodep，nodeqAssociating local data sets with nodeskRedundantly stored data set Rs，k，Rn，k，Rp，k，Rq，kThe first intermediate result calculated at each node is coded by CDC to obtain the coding result u of the first intermediate results，k，un，k，up，k，uq，kDetermining a first coding result, and sending the first coding result to the nodek(ii) a Respectively at the nodes，noden，nodep，nodeqAssociating local data sets with nodesmRedundantly stored data set Rs，m，Rn，m，Rp，m，Rq，mThe second intermediate result calculated at each node is coded by CDC to obtain the coding result u of the second intermediate results，m，un，m，up，m，uq，mI.e., the second encoding result, transmits the second encoding result to the nodem。
The first intermediate result comprises a nodes，noden，nodep，nodeqAccording to the local data set and the node respectivelykIntermediate results determined from redundantly stored data sets, the first encoding result comprising a nodes，noden，nodep，nodeqRespectively using CDC coding to code the obtained result according to the corresponding first intermediate result, wherein us，kRepresents a nodesCorresponding first coding result, un，kRepresents a nodenCorresponding first coding result, up，kRepresents a nodepCorresponding first coding result, uq，kRepresents a nodeqAnd the corresponding first coding result. The second intermediate result comprises a nodes，noden，nodep，nodeqAccording to the local data set and the node respectivelymIntermediate results determined from redundantly stored data sets, the second encoding result comprising a nodes，noden，nodep，nodeqRespectively using CDC coding to code the result according to the corresponding second intermediate result, wherein us，mRepresents a nodesCorresponding second coding result, un，mRepresents a nodenCorresponding second coding result, up，mRepresents a nodepCorresponding second coding result, uq，mRepresents a nodeqAnd the corresponding second encoding result.
As one example, step S70 includes:
if (v)2 s＝v2 k)∧(v3 n＝v3 k) Is true or (v)1 p＝v1 k)∧(v4 q＝v4 k) Is true, indicating a nodekThe correct number of the coding result in the first coding result is greater thanIs equal to 2 and nodekAll intermediate results of the local data set are correctly calculated, and the operation result r of the current node data set is calculated through a Reduce functionk＝H(v1 k，v2 k，v3 k，v4 k) Obtaining a first operation result; if (v)6 s＝v6 m)∧(v7 n＝v7 m) Is true or (v)5 p＝v5 m)∧(v8 q＝v8 m) Is true, indicating a nodemThe correct number of the coding result in the second coding result is more than or equal to 2 and the nodemAll intermediate results of the local data set are correctly calculated, and the operation result r of the current node data set is calculated through a Reduce functionm＝H(v5 m，v6 m，v7 m，v8 m) Obtaining a second operation result; wherein v is2 sRepresents a nodesIntermediate result of the upper data sequence number 2, v accordingly1 k，v2 k，v3 k，v4 kAll represent a nodekIntermediate result of the last corresponding data sequence number, v3 nRepresents a nodenIntermediate result of the upper data sequence number 3, v1 pRepresents a nodepIntermediate result of the upper data sequence number 1, v4 qRepresents a nodeqIntermediate result of the upper data sequence number 4, v6 sRepresents a nodesIntermediate result of the upper data sequence number (i.e. the sequence number of the data block) 6, like v5 m，v6 m，v7 m，v8 mAll represent a nodemIntermediate result of the last corresponding data sequence number, v7 nRepresents a nodenIntermediate result of the upper data sequence number 7, v5 pRepresents a nodepIntermediate result of the upper data sequence number 5, v8 qRepresents a nodeqThe intermediate result of the upper data sequence number 8, H (-) represents a function of the Reduce phase, which functions to merge multiple intermediate results into one,symbol Λ represents the logical operation and, v1 k，v2 k，v3 k，v4 kAll represent a nodemIntermediate result v obtained after calculation of self data block in Map stage5 m，v6 m，v7 m，v8 mAll represent a nodemCalculating self data blocks in a Map stage to obtain an intermediate result;
if (v)2 s＝v2 k)∧(v3 n＝v3 k) Is false and (v)1 p＝v1 k)∧(v4 q＝v4 k) If not, then assume the nodekAt least one correct coding result exists in the first coding result, i.e. the assumed nodekAt least one of the received 4 encoded data packets is verified as correct, e.g. v2 s＝v2 k，v3 n≠v3 k，v1 p≠v1 k，v4 q≠v4 k(ii) a If (v)6 s＝v6 m)∧(v7 n＝v7 m) Is false and (v)5 p＝v5 m)∧(v8 q＝v8 m) If not, then assume the nodemThere is at least one correct encoding result of the second encoding result, i.e. the hypothetical nodemAt least one of the received 4 encoded data packets is verified as correct, e.g. v6 s＝v6 m，v8 n≠v8 m，v7 p≠v7 m，v5 q≠v5 m；
If v is2 s≠v2 k，v3 n≠v3 k，v1 p≠v1 k，v4 q≠v4 kJudgment ofThe first coding result is not correct, i.e. u meanss，k，un，k，up，k，uq，kIf all the verifications fail, the method returns to execute the step S50, and/or if v fails6 s≠v6 m，v7 n≠v7 m，v5 p≠v5 m，v8 q≠v8 mIf the second coding result is judged to be incorrect, that means u iss，n，un，m，up，m，uq，mIf all the verifications fail, the process returns to step S50. So that the corresponding sub-cluster randomly selects a pair of check node nodes satisfying k + m-n +1kAnd a nodemAnd corresponding data processing is carried out again.
As one example, after step S70, at nodekThe correct number of coding results in the first coding result of (2) is at least 1, and the nodemWhen the correct number of the encoding results in the second encoding result is at least 1, the method further comprises:
if v is2 s＝v2 kIndicates us，kIf the verification is successful, us，k＝uk，s，v1 k＝v1 s，v2 k＝v2 s(ii) a By passing
Figure BDA0002256948290000131
To obtain v3 p，
Figure BDA0002256948290000132
To obtain v4 qJudgment of
Figure BDA0002256948290000133
If true, it indicates that the error source is nodekLocal data blocks, nodeskThe first coding result sent by other nodes is received correctly to obtain the nodekOf correct result rk＝H(v1 k，v2 k，v3 p，v4 q) Obtaining a first operation result; wherein u isk，sRepresentation storage in nodekAnd ofsThe intermediate result corresponding to the redundant data is encoded by CDC
Figure BDA0002256948290000134
Represents a logical exclusive or;
if v is6 s＝v6 mIndicates us，mIf the verification is successful, us，m＝um，s，v6 m＝v6 s，v5 m＝v5 s(ii) a By passing
Figure BDA0002256948290000135
To obtain v7 p，
Figure BDA0002256948290000136
To obtain v8 qJudgment of
Figure BDA0002256948290000137
If true, it indicates that the error source is nodemLocal data block, nodemThe second coding result sent by other nodes is received correctly to obtain the nodemOf correct result rm＝H(v5 m，v6 m，v7 p，v8 q) Obtaining a second operation result; wherein u ism，sRepresentation storage in nodemAnd ofsThe intermediate result corresponding to the redundant data is encoded using CDC5 sRepresents a nodesIntermediate result of upper data sequence number 5.
In this example, in the nodekIn due to v2 s＝v2 kI.e. us，kVerification is successful then us，k＝uk，s，v1 k＝v1 s，v2 k＝v2 s. Then pass through
Figure BDA0002256948290000138
To obtain v3 p，
Figure BDA0002256948290000139
To obtain v4 q. Then judge
Figure BDA00022569482900001310
If true, it indicates that the error source is nodekThe local data block, but the received coded data packet sent by other nodes is correct, and at this time, the correct result r of the current node can still be obtainedk＝H(v1 k，v2 k，v3 p，v4 q) I.e., the result of the first operation, S90 is further performed. Accordingly, in the nodemCan be processed the same for the corresponding data to obtain rm＝H(v5 m，v6 m，v7 p，v8 q) And determining the result of the second operation, and further executing S90.
In one embodiment, it is assumed that in a distributed cluster containing N-6 nodes, the amount of data to be processed is M. The execution process of the distributed data processing method with fault tolerance capability described above can be shown with reference to fig. 4, and includes the following steps:
step 1: the 6 nodes are grouped into a sub-cluster, and the 6 nodes can be respectively used as { node1，node2，node3，node4，node5，node6Represents it.
Step 2: dividing a data set M into
Figure BDA00022569482900001311
A subset of data sets, here
Figure BDA00022569482900001312
I.e. the data set M is sent to the current sub-cluster. Data of the sub-clusterSet M is partitioned into 8 independent sub-data blocks b1，b2，b3，b4，b5，b6，b7，b8}. Each block of data is then replicated 3 copies (i.e., triple modular redundancy) resulting in 24 copies of data.
And step 3: distributing the 24 data to 6 nodes of the current sub-cluster according to a set rule, wherein each node can be divided into 4 data, and B1＝{b1，b2，b3，b4}，B2＝{b1，b2，b5，b6}，B3＝{b1，b3，b5，b7}，B4＝{b2，b4，b6，b8}，B5＝{b3，b4，b7，b8}，B6＝{b5，b6，b7，b8}。
And 4, step 4: each node pairs each data block b of the local data setiExecuting Map function G (b)i) To obtain a corresponding set of intermediate results, which may be denoted as VjWhere j is 1, 2, 3, 4, 5 or 6, then V1＝{v1 1，v2 1，v3 1，v4 1}，V2＝{v1 2，v2 2，v5 2，v6 2}，V3＝{v1 3，v3 3，v5 3，v7 3}，V4＝{v2 4，v4 4，v6 4，v8 4}，V5＝{v3 5，v4 5，v7 5，v8 5}，V6＝{v5 6，v6 6，v7 6，v8 6}. Considering node1Calculated in Map phase
Figure BDA0002256948290000141
And
Figure BDA0002256948290000142
is a fault tolerant procedure in case of errors.
And 5: randomly selecting a pair of check node1And a node6The remaining four nodes { node } in the sub-cluster2，node3，node4，node5It is coupled with a node1Redundantly stored data set R2，1，R3，1，R4，1，R5，1And a node6Redundantly stored data set R2，6，R3，6，R4，6，R5，6The corresponding intermediate results are respectively coded by CDC to obtain corresponding coding results u2，1，u3，1，u4，1，u5，1Is sent to the node1，u2，6，u3，6，u4，6，u5，6Is sent to the node6。
The results obtained in step 5 using CDC encoding were each
Figure BDA0002256948290000143
Figure BDA0002256948290000144
And
Figure BDA0002256948290000145
Figure BDA0002256948290000146
step 6: node(s)1And a node6And decoding the received coded data packet to obtain an intermediate result. node(s)1Receive from
Figure BDA0002256948290000147
By decoding
Figure BDA0002256948290000148
Figure BDA0002256948290000149
To obtain v2 2，v3 5，v1 3，v4 4Node of the same theory6。
And 7: node(s)1And verifying the correctness of the coding result by comparing the intermediate result obtained by decoding with the intermediate result corresponding to the local data set, and simultaneously obtaining the correct intermediate result. Step 4 suppose that
Figure BDA00022569482900001410
And
Figure BDA00022569482900001411
a calculation error of then (v)2 2＝v2 1)∧(v3 5＝v3 1) And (v)1 3＝v1 1)∧(v4 4＝v4 1) Are all false while v2 2＝v2 1，v3 5≠v3 1，v1 3≠v1 1，v4 4≠v4 1Step 8 is executed.
And 8: due to v2 2＝v2 1I.e. u2，1Verification is successful, then v1 1＝v1 2，v2 1＝v2 2. Then pass through
Figure BDA0002256948290000151
To obtain v3 3，
Figure BDA0002256948290000152
To obtain v4 4. At the same time
Figure BDA0002256948290000153
If true, then node is indicated1The local data block is calculated incorrectly, but the received coded data packet is calculated correctly, and at the moment, the correct result r of the current node can be obtained1＝H(v1 1,v2 1,v3 3,v4 4) Step 9 is executed.
And step 9: node(s)1And a node6All pass the verification and obtain the correct operation result r1And r6Then, the final result r ═ H (r) of the current sub-cluster can be obtained by Reduce function1,r6)。
The distributed coding computation-based triple modular redundancy fault-tolerant algorithm provided by the embodiment can keep the low communication load of the distributed coding computation, and can make the backup maintenance and fault recovery efficiency close to the copy technology, thereby ensuring the high reliability of the distributed system under the low communication load.
The terms "comprising" and "having" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device that comprises a list of steps or modules is not limited to the listed steps or modules but may alternatively include other steps or modules not listed or inherent to such process, method, product, or device.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (7)

1. A distributed data processing method with fault tolerance capability is characterized by comprising the following steps:
s10, dividing the N nodes into
Figure FDA0003188355650000011
Sub-cluster to distribute computing tasks to
Figure FDA0003188355650000012
On the sub-clusters, each sub-cluster is subjected to parallel computation; wherein, the symbol
Figure FDA0003188355650000013
Represents rounding down, n represents the number of nodes in a sub-cluster;
s20, dividing the data set M into
Figure FDA0003188355650000014
Partial data sets, respectively sent to
Figure FDA0003188355650000016
On a sub-cluster, the data volume of each sub-cluster is made to be
Figure FDA0003188355650000015
Each sub-cluster divides the data set B of the sub-cluster into K' independent sub-data blocks, wherein each data block uses BiAnd i is more than or equal to 1 and less than or equal to K 'represents that each data block is copied by 3 parts to generate 3K' parts of data;
s30, each sub-cluster distributes 3K' data to n nodes of the current sub-cluster according to the set rule, so that each data block in each node local data set is different, if the node is not the samekAnd a nodemWhen the subscript k + m is n +1, then the nodekAnd a nodemThe intersection of the owned data sets is empty, the union is the full set of the current subset cluster data set, and if k + m is not equal to n +1, the nodekAnd a nodemThere are and only two identical redundant data;
s50, randomly selecting a pair of check node nodes satisfying k + m-n +1 for each sub-clusterkAnd a nodemThe remaining nodes in the sub-cluster perform the operations of steps S51 and S52, which are two stepsThe steps may be performed in parallel:
s51, using CDC algorithm to local and node by each other node in sub-clusterkCoding a first intermediate result corresponding to the redundantly stored data set to obtain a first coding result, and sending the first coding result to the nodek；
S52, using CDC algorithm to local and node by each other node in sub-clustermCoding a second intermediate result corresponding to the redundantly stored data set to obtain a second coding result, and sending the second coding result to the nodem；
S60，nodekAfter receiving all the first encoding results, the first encoding results obtained in the decoding S51 obtain the first decoding result, node, of each piece of datamAfter receiving all the second encoding results, decoding the second encoding results obtained in S52 to obtain second decoding results;
S70，nodekverifying the correctness of the first encoding result of each piece of data according to the first decoding result and the first intermediate result, and if the first encoding result is not correct, returning to execute the step S50, or, the nodemVerifying the correctness of the second encoding result according to the second decoding result and the second intermediate result, and if the second encoding result is not correct, returning to execute the step S50; if the number of the correct coding results in the first coding result is more than or equal to 1 and the number of the correct coding results in the second coding result is more than or equal to 1, the nodekCalculating the correct result of the corresponding node according to the correct coding result in the first coding result to obtain a first operation result, and the nodemCalculating the correct result of the corresponding node according to the correct coding result in the second coding result to obtain a second operation result;
and S90, determining the final result of the corresponding sub-cluster according to the corresponding first operation result and the second operation result by each sub-cluster.
2. The distributed data processing method with fault tolerance according to claim 1, wherein in step S30, when each sub-cluster distributes 3K' parts of data to n nodes of the current sub-cluster according to the set rule, the method further comprises:
in each subgroup, each node can be divided into 3K'/n data, and the data set assigned to each node is BjRepresents; if nodekAnd a nodemIf the subscript k + m ≠ n +1, then the intersection of the data sets owned by the subscript k + m ≠ n +1 is empty, the union of the data sets owned by the subscript k + m ≠ n +1 is the full set of the current sub-cluster data set, and if k + m ≠ n +1, then the nodekAnd a nodemWith and only two identical redundant data bi,bjIf B isk∩Bm＝{bi,bjIs defined as
Figure FDA0003188355650000021
Wherein, the symbol
Figure FDA0003188355650000022
The representation is defined as.
3. The distributed data processing method with fault tolerance capability according to claim 2, wherein before step S50, further comprising:
s40, each node executes a Map function G (-) on each data block of the local data set, and obtains an intermediate result:
Figure FDA0003188355650000023
wherein i represents a data block serial number, t represents a node serial number, and the Map function can be customized by a user aiming at different calculation tasks.
4. The distributed data processing method with fault tolerance of claim 1, wherein the value of n is 6 and the value of K' is 8.
5. The distributed data processing method with fault tolerance capability according to claim 4, wherein the step S50 includes:
each sub-cluster randomly selects a pair of check nodekAnd a nodemAnd k + m is 7, and the rest four nodes in each sub-cluster are recorded as: node(s)s,noden,nodep,nodeqIn each nodes,noden,nodep,nodeqAssociating local data sets with nodeskRedundantly stored data set Rs,k,Rn,k,Rp,k,Rq,kThe first intermediate result calculated at each node is coded by CDC to obtain the coding result u of the first intermediate results,k,un,k,up,k,uq,kDetermining a first coding result, and sending the first coding result to the nodek(ii) a Respectively at the nodes,noden,nodep,nodeqAssociating local data sets with nodesmRedundantly stored data set Rs,m,Rn,m,Rp,m,Rq,mThe second intermediate result calculated at each node is coded by CDC to obtain the coding result u of the second intermediate results,m,un,m,up,m,uq,mI.e., the second encoding result, transmits the second encoding result to the nodem。
6. The distributed data processing method with fault tolerance capability according to claim 5, wherein step S70 includes:
if (v)2 s＝v2 k)∧(v3 n＝v3 k) Is formed by (v)1 p＝v1 k)∧(v4 q＝v4 k) Is true, indicating a nodekThe correct number of the coding result in the first coding result is more than or equal to 2 and the nodekAll local intermediate results are correctly calculated, and the operation result r of the current node data set is calculated through a Reduce functionk＝H(v1 k,v2 k,v3 k,v4 k) Obtaining a first operation result; if (v)6 s＝v6 m)∧(v7 n＝v7 m) Is true or (v)5 p＝v5 m)∧(v8 q＝v8 m) Is true, indicating a nodemThe correct number of the coding result in the second coding result is more than or equal to 2 and the nodemAll local intermediate results are correctly calculated, and the operation result r of the current node data set is calculated through a Reduce functionm＝H(v5 m,v6 m,v7 m,v8 m) Obtaining a second operation result; wherein v is2 sRepresents a nodesIntermediate result of data sequence number 2, same as v1 k,v2 k,v3 k,v4 kAll represent a nodekIntermediate result of the last corresponding data sequence number, v3 nRepresents a nodenIntermediate result of the upper data sequence number 3, v1 pRepresents a nodepIntermediate result of the upper data sequence number 1, v4 qRepresents a nodeqIntermediate result of the upper data sequence number 4, v6 sRepresents a nodesIntermediate result of data sequence number 6, same as v5 m,v6 m,v7 m,v8 mAll represent a nodemIntermediate result of the last corresponding data sequence number, v7 nRepresents a nodenIntermediate result of the upper data sequence number 7, v5 pRepresents a nodepIntermediate result of the upper data sequence number 5, v8 qRepresents a nodeqIntermediate results of the upper data sequence number 8, where H (-) denotes a function of the Reduce phase, the effect of which is to combine a plurality of intermediate results into one, the symbol Λ denotes the logical operation AND, v1 k,v2 k,v3 k,v4 kAll represent a nodemIntermediate result v obtained after calculation of self data block in Map stage5 m,v6 m,v7 m,v8 mAll represent a nodemCalculating self data blocks in a Map stage to obtain an intermediate result;
if (v)2 s＝v2 k)∧(v3 n＝v3 k) Is false and (v)1 p＝v1 k)∧(v4 q＝v4 k) If not, it indicates the nodekAt least one correct encoding result exists in the first encoding result; if (v)6 s＝v6 m)∧(v7 n＝v7 m) Is false and (v)5 p＝v5 m)∧(v8 q＝v8 m) If not, it indicates the nodemAt least one correct encoding result exists in the second encoding result; if v is2 s≠v2 k,v3 n≠v3 k,v1 p≠v1 k,v4 q≠v4 kIf the first encoding result is not correct, the process returns to step S50, or if v is not correct6 s≠v6 m,v7 n≠v7 m,v5 p≠v5 m,v8 q≠v8 mIf it is determined that the second encoding result is not correct, the process returns to step S50.
7. The distributed data processing method with fault tolerance of claim 6, wherein step S70 is followed by nodekThe correct number of coding results in the first coding result of (2) is at least 1, and the nodemWhen the correct number of the encoding results in the second encoding result is at least 1, the method further comprises:
if v is2 s＝v2 kIndicates us,kIf the verification is successful, us,k＝uk,s，v1 k＝v1 s，v2 k＝v2 s(ii) a By passing
Figure FDA0003188355650000031
To obtain v3 p，
Figure FDA0003188355650000032
To obtain v4 qJudgment of
Figure FDA0003188355650000033
If true, it indicates that the error source is nodekLocal data blocks, nodeskThe first coding result sent by other nodes is received correctly to obtain the nodekOf correct result rk＝H(v1 k,v2 k,v3 p,v4 q) Obtaining a first operation result; wherein u isk,sRepresentation storage in nodekAnd ofsThe intermediate result corresponding to the redundant data is encoded by CDC
Figure FDA0003188355650000041
Represents a logical exclusive or;
if v is6 s＝v6 mIndicates us,mIf the verification is successful, us,m＝um,s，v6 m＝v6 s，v5 m＝v5 s(ii) a By passing
Figure FDA0003188355650000042
To obtain v7 p，
Figure FDA0003188355650000043
To obtain v8 qJudgment of
Figure FDA0003188355650000044
If true, it indicates that the error source is nodemLocal data block, nodemThe second coding result sent by other nodes is received correctly to obtain the nodemOf correct result rm＝H(v5 m,v6 m,v7 p,v8 q) Obtaining a second operation result; wherein u ism,sRepresentation storage in nodemAnd ofsThe intermediate result corresponding to the redundant data is encoded using CDC5 sRepresents a nodesIntermediate result of upper data sequence number 5.
CN201911057644.3A 2019-11-01 2019-11-01 Distributed data processing method with fault tolerance capability Active CN111045843B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911057644.3A CN111045843B (en) 2019-11-01 2019-11-01 Distributed data processing method with fault tolerance capability

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911057644.3A CN111045843B (en) 2019-11-01 2019-11-01 Distributed data processing method with fault tolerance capability

Publications (2)

Publication Number Publication Date
CN111045843A CN111045843A (en) 2020-04-21
CN111045843B true CN111045843B (en) 2021-09-28

Family

ID=70232883

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911057644.3A Active CN111045843B (en) 2019-11-01 2019-11-01 Distributed data processing method with fault tolerance capability

Country Status (1)

Country Link
CN (1) CN111045843B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111490795B (en) * 2020-05-25 2021-09-24 南京大学 Intermediate value length isomerism-oriented encoding MapReduce method
CN111858721B (en) * 2020-08-03 2023-07-21 南京大学 Distributed computing method based on priority coding
CN113434299B (en) * 2021-07-05 2024-02-06 广西师范大学 Coding distributed computing method based on MapReduce framework

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101834899A (en) * 2010-04-29 2010-09-15 中科院成都信息技术有限公司 Distributed adaptive coding and storing method
CN103064728A (en) * 2012-12-13 2013-04-24 河南工业大学 Fault-tolerant scheduling method of Map Reduce task
CN108052679A (en) * 2018-01-04 2018-05-18 焦点科技股份有限公司 A kind of Log Analysis System based on HADOOP
US10114581B1 (en) * 2016-12-27 2018-10-30 EMC IP Holding Company LLC Creating a virtual access point in time on an object based journal replication

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101834899A (en) * 2010-04-29 2010-09-15 中科院成都信息技术有限公司 Distributed adaptive coding and storing method
CN103064728A (en) * 2012-12-13 2013-04-24 河南工业大学 Fault-tolerant scheduling method of Map Reduce task
US10114581B1 (en) * 2016-12-27 2018-10-30 EMC IP Holding Company LLC Creating a virtual access point in time on an object based journal replication
CN108052679A (en) * 2018-01-04 2018-05-18 焦点科技股份有限公司 A kind of Log Analysis System based on HADOOP

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
A fundamental tradeoff between computation and;LI Songze, MADDAH-ALI M A, QIAN Yu等;《IEEE Transactions on Information Theory》;20180131;全文 *

Also Published As

Publication number Publication date
CN111045843A (en) 2020-04-21

Similar Documents

Publication Publication Date Title
CN111045843B (en) Distributed data processing method with fault tolerance capability
US10594340B2 (en) Disaster recovery with consolidated erasure coding in geographically distributed setups
CN109643258B (en) Multi-node repair using high-rate minimal storage erase code
WO2020047707A1 (en) Data coding, decoding and repairing method for distributed storage system
CN109491835B (en) Data fault-tolerant method based on dynamic block code
CN107003933B (en) Method and device for constructing partial copy code and data restoration method thereof
KR20130012591A (en) State history storage for synchronizing redundant processors
CN102843212B (en) Coding and decoding processing method and device
US11467905B1 (en) Stripe merging method and system based on erasure codes
Esmaili et al. The core storage primitive: Cross-object redundancy for efficient data repair & access in erasure coded storage
CN109828723B (en) Distributed storage system and accurate data restoration method and device thereof
CN113541870A (en) Recovery optimization method for erasure code storage single node failure
CN107103360B (en) Reliable application distribution distributed genetic method based on mixed cloud
CN109344009B (en) Mobile cloud system fault-tolerant method based on hierarchical check points
CN110888761A (en) Fault-tolerant method based on active backup of key task part and stream processing platform
CN107153661A (en) A kind of storage, read method and its device of the data based on HDFS systems
WO2022246952A1 (en) Fault tolerance method and system based on multi-master-node master-slave distributed architecture
CN114281508A (en) Data batch-flow fusion offline calculation method
CN111459710B (en) Erasure code memory recovery method, device and memory system capable of sensing heat degree and risk
WO2018119976A1 (en) Efficient data layout optimization method for data warehouse system
CN111045853A (en) Method and device for improving erasure code recovery speed and background server
US10599520B2 (en) Meta-copysets for fault-tolerant data storage
CN112732203B (en) Regeneration code construction method, file reconstruction method and node repair method
CN110908835B (en) Data redundancy method and system supporting private label in distributed system
JP2012033169A (en) Method and device for supporting live check pointing, synchronization, and/or recovery using coding in backup system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant