CN110457161A - A kind of efficiently highly reliable big data storage system, method, computer program - Google Patents
A kind of efficiently highly reliable big data storage system, method, computer program Download PDFInfo
- Publication number
- CN110457161A CN110457161A CN201910681574.2A CN201910681574A CN110457161A CN 110457161 A CN110457161 A CN 110457161A CN 201910681574 A CN201910681574 A CN 201910681574A CN 110457161 A CN110457161 A CN 110457161A
- Authority
- CN
- China
- Prior art keywords
- data
- array
- module
- node
- piecemeal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 238000013500 data storage Methods 0.000 title claims abstract description 31
- 238000004590 computer program Methods 0.000 title claims abstract description 12
- 238000011084 recovery Methods 0.000 claims abstract description 42
- 238000003860 storage Methods 0.000 claims abstract description 35
- 238000007726 management method Methods 0.000 claims abstract description 4
- 239000011159 matrix material Substances 0.000 claims description 75
- 230000008569 process Effects 0.000 claims description 28
- 238000010276 construction Methods 0.000 claims description 9
- 230000005540 biological transmission Effects 0.000 claims description 8
- 238000001514 detection method Methods 0.000 claims description 7
- 238000005457 optimization Methods 0.000 claims description 6
- 230000006399 behavior Effects 0.000 claims description 3
- 230000000903 blocking effect Effects 0.000 claims description 3
- 230000008859 change Effects 0.000 claims description 3
- 238000005520 cutting process Methods 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 7
- 238000012545 processing Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 5
- 230000001965 increasing effect Effects 0.000 description 5
- 241000208340 Araliaceae Species 0.000 description 3
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 3
- 235000003140 Panax quinquefolius Nutrition 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 239000004744 fabric Substances 0.000 description 3
- 235000008434 ginseng Nutrition 0.000 description 3
- 230000001568 sexual effect Effects 0.000 description 3
- 235000013399 edible fruits Nutrition 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- FNMKZDDKPDBYJM-UHFFFAOYSA-N 3-(1,3-benzodioxol-5-yl)-7-(3-methylbut-2-enoxy)chromen-4-one Chemical compound C1=C2OCOC2=CC(C2=COC=3C(C2=O)=CC=C(C=3)OCC=C(C)C)=C1 FNMKZDDKPDBYJM-UHFFFAOYSA-N 0.000 description 1
- 241000700605 Viruses Species 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 229910002056 binary alloy Inorganic materials 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1004—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's to protect a block of data words, e.g. CRC or checksum
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Detection And Correction Of Errors (AREA)
Abstract
The invention belongs to information data processing technology field, a kind of efficiently highly reliable big data storage system, method, computer program are disclosed, the data for obtaining user and uploading are set according to configuration center；Array module calculates the array of coding；The array that array module calculates is read, is encoded according to array；Host node and from heartbeat message is sent between node mutually, reads the heartbeat delay time of configuration center；Parse failure node information；Carry out data recovery；The position of each initial data piecemeal is inquired according to the data ID to be downloaded；The setting of configuration center is read, the newborn node that the piecemeal memory node and data after being responsible for allocated code select when restoring；Configuration center, the various parameters being responsible in management system；Piecemeal after storage coding.The present invention provides a kind of storage efficiencies to be optimal, efficiently, high reliability, the big data storage solution easily extended.
Description
Technical field
The invention belongs to information data processing technology fields more particularly to a kind of efficiently highly reliable big data to store system
System, method, computer program.
Background technique
Currently, the immediate prior art:
As information technology is in the continuous development of various industries and field, data volume shows the trend of exponential growth,
The rapid growth bring data storing reliability problem of data volume and the concurrent efficiency of data access are also following.
Usually effective method be building by multiple back end (back end, which can be a PC or server etc., can be used as data
The equipment of storage) composition distributed memory system.The sustainable growth of every profession and trade field storage data quantity leads to distributed storage
The scale of system is increasing, and number of nodes is continuously increased.Existing some enterprises have possessed multiple more than 3000 nodes
Storage system.There is the dispersibility of region and some characteristics of network, distributed storage due to distributed memory system
System is also faced with many hardware and software failures, the test such as virus attack and natural calamity.These factors are likely to cause more
A data node failure, so that loss of data.Therefore, how for distributed memory system the environment of a safety is provided, allows it
With high reliability, data have high availability, become an important topic of distributed security storage at this stage.It is high
Fault-tolerant ability is closely related with data reliability, becomes the indispensable technology of distributed memory system, so for dividing
More fault-toleranr techniques of cloth storage system are most important to safeguards system data safety.
Currently, fault tolerant most commonly seen in distributed memory system is mainly more replication policies.The strategy
It is data to be replicated n-1 copy to be respectively stored on n different nodes, to realize redundancy backup.Redundant data is n-1 secondary
Notebook data can effectively restore data after occurring n-1 node while failure simultaneously.Most of commercial storage system uses
Data reliability Enhancement Method be more fault-tolerant strategies.Famous distributed memory system GFS, hadoop uses this side
Method.This method does not need special coding and restructing algorithm, and error resilience performance is preferable, but space utilization rate is low.If fault-tolerant energy
When power is n-1, space utilization rate only has 1/n.In large-scale distributed storage system of today, with the promotion of fault-tolerant ability
And the storage efficiency and ever-increasing update cost constantly declined becomes its great drawback.
Storage system fault-tolerance approach (abbreviation correcting and eleting codes strategy) based on correcting and eleting codes is that one kind is quite paid attention to by industry in recent years
Enhancing storage system reliability method.Compared with more replication policies, the sharpest edges of correcting and eleting codes strategy are to protect
In the case where demonstrate,proving fault-tolerant ability, it can be effectively reduced update cost, improve storage efficiency.Therefore, correcting and eleting codes strategy becomes and mentions
The research hotspot of the reliability method of high distributed memory system.To using correcting and eleting codes strategy as the appearance of distributed memory system
Mainly there are a Plank team of University of Tennessee and the Blaum team of IBM in wrong Mechanism Study, foreign countries, the country have Tsinghua University relax after
Military team and Xu Yinlong team, Chinese University of Science and Technology etc..
Currently, the research for correcting and eleting codes strategy, mainly in two broad aspects: first is that RS (Reed-Solomon) correcting and eleting codes.
RS correcting and eleting codes are the MDS codes that can entangle any mistake.Theoretically fault-tolerant ability is unrestricted for RS correcting and eleting codes, imitates with optimal storage
Rate, but it calculates the operation being related in polynary finite field, and computation complexity is high (multiplying in especially polynary finite field),
It is excessively huge that this is calculated as in large scale distributed system.To solve this problem, some scholars also further investigate this, and
It is proposed some improved methods, wherein most typical method be by by polynary finite field operations be transformed on two element field carry out to
Improve arithmetic speed.Other scholars also have outstanding contributions on improving polynary finite field operations efficiency, such as Plank proposition
GF-complete improves the arithmetic speed in polynary finite field by modification instruction set.Second is that array code.The volume of array code
Code process usually only uses simple binary system XOR operation, possesses operation efficiency height, realizes the advantages that process is simple.These are excellent
Point is so that array code is highly suitable to be applied for large-scale distributed storage system.What array code was classified according to storage efficiency
Words, can be divided into MDS code and non-MDS code.There are two fault-tolerant EVENODD codes and X code, three fault-tolerant star in typical MDS code
Code.But there are no MDS array code of the fault-tolerant ability greater than 3 occur so far.In order to there is better fault-tolerant ability, work people
Member designs some how fault-tolerant (fault-tolerant ability is greater than 3) non-MDS code, wherein more typical in the case where sacrificing certain storage efficiency
Use Weaver code, Grid code etc..Although this kind of array code increases in fault-tolerant ability, is applied and be distributed
Variety of problems can be faced in formula storage system, first is that band or stick limited amount system, need band or stick be prime number or
That could be realized with the wired sexual intercourse of prime number, this be undoubtedly the expansion bands of distributed memory system are come with very big limitation, with point
The expansibility of cloth storage system is not inconsistent.Second is that storage efficiency can be reduced with the increase of fault-tolerant number.Such as design one
A to hold 10 wrong Weaver codes, its storage efficiency is less than 20%.So these problems cause array code in distribution
Storage system field practicability is not high.
In conclusion problem of the existing technology is: fault-toleranr technique space utilization rate in current distributed memory system
Low, storage efficiency is low, and poor reliability, scalability is poor, and practicability is not high.
Solve the difficulty of above-mentioned technical problem:
First is that band or stick limited amount system, need band or stick be prime number or with the wired sexual intercourse of prime number
It is able to achieve, this, which is undoubtedly, carrys out very big limitation to the expansion bands of distributed memory system, the easy extension with distributed memory system
Property is not inconsistent.Second is that storage efficiency can be reduced with the increase of fault-tolerant number.Such as design one can hold 10 it is wrong
Weaver code, its storage efficiency is less than 20%.So these problems cause array code practical in distributed memory system field
Property is not high.
Solve the meaning of above-mentioned technical problem:
First is that the numerical value in prime number is bigger, the interval between two adjacent prime numbers is also increasing.Dividing on a large scale
When cloth storage system extends, if be still extended according to prime number, need to increase more nodes or dummy node,
More nodes mean the increased costs of primary system extension, and more dummy nodes then need to waste more calculating, shadow
Ring efficiency.It releases and the quantity of band and stick is limited, user can be allowed to be extended according to the demand of itself, not may require that increasing
If more nodes and dummy node.
Second is that correcting and eleting codes used by the fault-toleranr technique of this system are MDS codes, storage efficiency is optimal, can be identical
Memory space, store more effective data.
Summary of the invention
In view of the problems of the existing technology, the present invention provides a kind of efficiently highly reliable big data storage systems, side
Method, computer program.
The invention is realized in this way a kind of efficiently highly reliable big data storage system includes:
The data for obtaining user and uploading are arranged according to configuration center, and transfer data to coding mould for data uploading module
Block；
Array module calculates the array of coding according to the parameter of configuration center, and stores；
Coding module receives the data of data uploading module transmission, the array that array module calculates is read, according to array
It is encoded；
Heart beat detection module, host node and from heartbeat message is sent between node mutually, the heartbeat for reading configuration center is prolonged
The slow time, if it exceeds the time, then judge the node failure, and send information to data recovery module；
Data recovery module parses failure node information according to the feedback information of heartbeat inspecting module, reads in configuration
The data of the heart restore relevant parameter, parsing failure node corresponding location information in an array, transmit to decoder module, into
Row data are restored；
Decoder module obtains the failure node location information of data recovery module transmission, then carries out data recovery；
Data download module inquires the position of each initial data piecemeal according to the data ID to be downloaded, if each original
Beginning deblocking, which exists, to be lost, then advanced row data are restored, if all existed, all initial data piecemeals are downloaded,
It carries out being assembled into initial data；
Load balancing module reads the setting of configuration center, piecemeal memory node and data after being responsible for allocated code
The newborn node selected when recovery；
Configuration center, the various parameters being responsible in management system；
Distributed memory system, the piecemeal after storing coding.
Further, the array module includes:
Parameter set unit is used for plan of establishment parameter, and parameter includes: initial data number of blocks n, verifies number of blocks m, has
Confinement size GF (2w)；
Matrix construction unit, for constructing finite field gf (2w)And corresponding binary matrix and Cauchy matrix, and by Ke
Western matrix-expand is at binary matrix；
Array Construction unit constructs array by binary matrix；
Operation optimizes unit, carries out operation statistics to entire array.
Further, the coding module includes:
Initial data loading unit takes out corresponding data according to array request from the file to be encoded；
Redundant data computing unit calculates the value of each replacement formula according to replacement list, goes out further according to array computation
Redundant data.
Further, the decoder module includes；
Decoding unit, for according to Array Construction decoding matrix；
Data Computation Unit, for calculating the data for losing data recovery needs；
Data recovery unit, the data block that all loss data that are restored need.Further according to the data of these data blocks
Exclusive or obtains corresponding loss data.
Another object of the present invention is to provide a kind of efficient height for efficiently highly reliable big data storage system can
The big data storage method leaned on, the efficiently highly reliable big data storage method include that data upload process, data were restored
Journey and data downloading process.
The array module of the efficiently highly reliable big data storage method realizes that process is as follows:
Step 1: plan of establishment parameter, parameter include: initial data number of blocks n, verify number of blocks m, finite field size GF
(2w)；
Step 2: building finite field gf (2w) and corresponding binary matrix；
If root is primitive polynomial g (x)=x of αa+xb…+xc+ 1, (a > b > ... > c) generates a finite field gf
(2w), i.e. αa=αb+…αc+ 1, construction process is as follows:
α0=1
α1=α
α2=α2
αa=αb+…+αc+1
αa+1=αa× α=αb+1+…+αc+1+α
It is the multinomial element representation in finite field above.
Coefficient vector V (x)=(f of each finite field multinomial element f (x) is obtained again0,f1,…,fw-1)；
Then, the corresponding binary matrix β (e) of building V (x).For arbitrary element e ∈ GF (2w), β (e) be a w ×
The binary matrix of w, wherein i-th is classified as xiThe coefficient vector of emodp (x)；
Step 3: building Cauchy matrix；
From GF (2w) m element and n element composition X={ x are chosen respectively1,x2,…,xm, Y={ y1,y2,…,ynStructure
Cauchy matrix is built, the Cauchy matrix of building is as follows:
Step 4: Cauchy matrix is extended to binary matrix；
The corresponding binary matrix β (e) that element in Cauchy matrix obtained in step 3 is obtained according to step 2,
The Cauchy matrix of m × n is extended to the binary matrix of mw × nw；
Binary matrix after being expanded is as follows:
Step 5: building array；
Firstly, each data block in n data block is cut into w parts, such as D1,D2,D3,…,DnIt is cut into D1,1,
D1,2,…,D1,w,D2,1,…,Dn,w；
Obtain array:
Step 6: operation optimization
Operation statistics is carried out to entire array, in statistic array(i1,
I2, i3 ∈ { 1,2 ..., n }, j1, j2, j3 ∈ { 1,2 ..., w }) number that occurs of pattern.It is more than primary formula by number
Individually take out, according toThe sub- number of pattern multiplies 2,The sub- number of pattern multiply 3 after knot
Fruit sorts from large to small；Array is replaced according to ranking results, is replaced successfully labeled as Si, i is replacement serial number.Replacement is completed
Alternate form sublist S and last optimization array Z2 are obtained afterwards；
If the parameter being arranged does not change, that array module need to only be executed once, obtain array Z1 and Z2, and replace
Formula list S is changed, does not need to repeat；
The coding module of the efficiently highly reliable big data storage method realizes that process is as follows:
Step 1: load initial data
Nw parts of data are taken out from the file to be encoded according to array request, respectively correspond D in array1,1,…,D1,w,
D2,1,…,Dn,w；
Step 2: computing redundancy data
According to replacement list S, the value of each replacement formula is calculated.And then according to array Z2, calculate redundant digit
According to completion coding；
The decoder module of the efficiently highly reliable big data storage method realizes that process is as follows:
Step 1: building decoding matrix
Firstly, being divided into (n+m) w block according to array Z1, so creation (n+m) w × (n+m) w matrix B, structure areI is the unit matrix of nw × nw, and O is the null matrix of nw × mw, and H is the check matrix of Z1；
Step 2 calculates and loses the data that data restore needs
It (1) is the loss of data of entire data column because when node failure, so when restoring data, according to loss
Node create corresponding loss element list L；
(2) it loops through the element e lost in element list L and carries out (3) if e belongs to data element, if e belongs to
In redundant elements, then the judgement of next element is carried out；
(3) r is the set for the row that e train value is 1 in check matrix H.If not having element in r, e row is set 0；If r
In have element, that just from be chosen to remove in r lose element after the smallest a line of Hamming weight as rl, element is corresponding in r
Capable and e row successively with rlXOR operation is carried out, finally by rlSet 0；
(4) after data element operations all in L are complete, then the corresponding column of the redundant elements in L are all set 0；
Step 3 restores data；
After completing step 2, if the corresponding behavior r in corresponding matrix B of the element e in loss element list Le, ce
It is in reThe set for the column that intermediate value is 1；ceIn the corresponding data block of element be restore element e need data block；According to losing
Element list L and matrix B are lost, the data block that all loss data that are restored need；It is different further according to the data of these data blocks
Or obtain corresponding loss data.
Further, the data upload process specifically includes:
Data uploading module receives the data uploaded, and the data of upload are sent to coding module；
Array module reads the parameter in relation to array in configuration center；
Array module polls are under corresponding parameter, if having corresponding array, if it is not, according to parameter, meter
Encoding array is calculated, and the array being finally calculated is sent to coding module, if existing, array is directly sent to volume
Code module；
Coding module reads the parameter of the related coding of configuration center, and receive the initial data that data uploading module is sent with
And the encoding array that array module is sent, cutting, pretreatment, coding meter then are carried out to initial data according to encoding array
It calculates, obtains redundant data；
Load balancing module reads the related related parameter of load balancing in configuration center；
Each node load state of load balancing module Querying Distributed storage system；
Load balancing module chooses the memory node list of piecemeal after coding according to parameter and each node load state, will
It is sent to coding module；
The piecemeal memory node list that coding module balancing received load module is chosen, then according to node listing, respectively
Corresponding piecemeal is stored in distributed memory system.
Further, the data recovery procedure specifically includes:
Heart beat detection module moment control host node and from sending heartbeat message between node, if within the setting time,
The heartbeat message of node is not received, then judges the node failure, and the information of the node failure is sent to data and restores mould
Block；
Data recovery module reads the parameter that related data are restored in configuration center；
Data recovery module restores related parameter, parsing according to node failure information or piecemeal fail message and data
Node or the piecemeal corresponding position in encoding array are lost out, send it to decoder module；
Load balancing module reads the related related parameter of load balancing in configuration center；
Load balancing module chooses newborn node listing, sends it to decoding according to parameter and each node load state
Module；
Decoder module chooses decoding scheme, according to side according to node or the piecemeal corresponding position in encoding array is lost
Case reads residue block data needed for HDFS distributed memory system；
Decoder module is decoded calculating according to remaining block data, obtains the data for losing piecemeal, equal according to load
The newborn node listing that the module that weighs is chosen, the piecemeal recovered is stored in the newborn node of distributed memory system.
Further, the data downloading process specifically includes:
Data download module will if piecemeal is intact according to the state of the data ID to be downloaded inquiry each piecemeal of data
Piecemeal assembles, and downloads to client, loses if there is piecemeal, then will lose blocking information and be sent to data recovery module；
Data recovery module reads the parameter that related data are restored in configuration center；
Data recovery module restores related parameter, parsing according to node failure information or piecemeal fail message and data
Node or the piecemeal corresponding position in encoding array are lost out, send it to decoder module；
Load balancing module reads the related related parameter of load balancing in configuration center；
Load balancing module chooses newborn node listing, sends it to decoding according to parameter and each node load state
Module；
Decoder module chooses decoding scheme, according to side according to node or the piecemeal corresponding position in encoding array is lost
Case reads residue block data needed for distributed memory system；
Decoder module is decoded calculating according to remaining block data, obtains the data for losing piecemeal, equal according to load
The newborn node listing that the module that weighs is chosen, the piecemeal recovered is stored in the newborn node of distributed memory system；
Data download module reads the data of each piecemeal, and block data is assembled into, downloads to client.
Another object of the present invention is to provide a kind of computer program, the computer program has for executing efficiently
The program coding of highly reliable big data storage method, wherein the computer program is run on computers.
In conclusion advantages of the present invention and good effect are as follows:
Storage efficiency of the present invention is optimal: the correcting and eleting codes that the fault-toleranr technique of this system uses are MDS codes, are theoretically had
There is optimal space utilization rate, fault-tolerant ability adds one to need to increase a redundant block.
The present invention is efficient: the correcting and eleting codes that the fault-toleranr technique of this system uses are turned by calculating the complicated finite field of RS code
It changes simple XOR operation into, effectively increases code efficiency.
Reliability of the present invention: the correcting and eleting codes that the fault-toleranr technique of this system uses are transformed by RS code, with RS code, In
Theoretically fault-tolerant ability is unrestricted.
The present invention easily extends: being directed to same efficient array code, the correcting and eleting codes that the fault-toleranr technique of this system uses are to item
There is no limit do not need band or stick be prime number or can be realized as with the wired sexual intercourse of prime number for band or number of blocks.With
It family can expansion system according to their own needs.
The present invention is low for fault-toleranr technique space utilization rate in current distributed memory system, low efficiency, poor reliability, expands
The status of malleability difference provides a kind of storage efficiency and is optimal and (has MDS property), and efficiently, high reliability easily extends
Big data solution.
Detailed description of the invention
Fig. 1 is efficiently highly reliable big data memory system architecture schematic diagram provided in an embodiment of the present invention.
Fig. 2 is data upload process schematic diagram provided in an embodiment of the present invention.
Fig. 3 is data recovery procedure provided in an embodiment of the present invention and data downloading process schematic diagram.
Fig. 4 is correcting and eleting codes of the present invention provided in an embodiment of the present invention and EVENODD code operation efficiency contrast schematic diagram.
Fig. 5 is the operation efficiency contrast schematic diagram of RA code of the present invention and Star code provided in an embodiment of the present invention.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to embodiments, to the present invention
It is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not used to
Limit the present invention.
The present invention is low for fault-toleranr technique space utilization rate in current distributed memory system, low efficiency, poor reliability, expands
The status of malleability difference, provide a kind of storage efficiency be optimal and (there is MDS property), efficiently, high reliability, easily extend it is big
Data solution.
As shown in Figure 1, efficiently highly reliable big data storage system provided in an embodiment of the present invention includes: that data upload mould
Block 101, array module 102, coding module 103, heart beat detection module 104, data recovery module 105, decoder module 106,
Data download module 107, load balancing module 108, configuration center 109, distributed memory system 110.
The data for obtaining user and uploading are arranged according to configuration center, and transfer data to coding for data uploading module 101
Module；
Array module 102 calculates the array of coding according to the parameter of configuration center, and stores；
Coding module 103 receives the data of data uploading module transmission, reads the array that array module calculates, according to
Array is encoded；
Heart beat detection module 104, host node and from heartbeat message is sent between node mutually read the heartbeat of configuration center
Delay time, if it exceeds the time, then judge the node failure, and send information to data recovery module；
Data recovery module 105 parses failure node information according to the feedback information of heartbeat inspecting module, and reading is matched
The data for setting center restore relevant parameter, parse failure node corresponding location information in an array, transmit and give decoding mould
Block carries out data recovery；
Decoder module 106 obtains the failure node location information of data recovery module transmission, then carries out data recovery；
Data download module 107 inquires the position of each initial data piecemeal according to the data ID to be downloaded, if each
Initial data piecemeal, which exists, to be lost, then advanced row data are restored, will be under the downloading of all initial data piecemeals if all existed
Come, carries out being assembled into initial data；
Load balancing module 108 reads the setting of configuration center, the piecemeal memory node after being responsible for allocated code, and
The newborn node that data select when restoring；
Configuration center 109, the various parameters being responsible in management system；
Distributed memory system 110, the piecemeal after storing coding.
Further, array module 102 realizes that process is as follows:
Step 1: plan of establishment parameter, parameter include: initial data number of blocks n, verify number of blocks m, finite field size GF
(2w)。
Step 2: building finite field gf (2w) and corresponding binary matrix.
If root is primitive polynomial g (x)=x of αa+xb…+xc+ 1, (a > b > ... > c) generates a finite field gf
(2w), i.e. αa=αb+…αc+ 1, construction process is as follows:
α0=1
α1=α
α2=α2
αa=αb+…+αc+1
αa+1=αa× α=αb+1+…+αc+1+α
It is the multinomial element representation in finite field above.
Next, obtaining coefficient vector V (x)=(f of each finite field multinomial element f (x) again0,f1,…,fw-1)。
Then, the corresponding binary matrix β (e) of building V (x).For arbitrary element e ∈ GF (2w), β (e) be a w ×
The binary matrix of w, wherein i-th is classified as xiThe coefficient vector of emodp (x).
Step 3: building Cauchy matrix.
From GF (2w) m element and n element composition X={ x are chosen respectively1,x2,…,xm, Y={ y1,y2,…,ynStructure
Cauchy matrix is built, the Cauchy matrix of building is as follows:
Step 4: Cauchy matrix is extended to binary matrix.
The corresponding binary matrix β (e) that element in Cauchy matrix obtained in step 3 is obtained according to step 2.
The Cauchy matrix of m × n is extended to the binary matrix of mw × nw.
Binary matrix after being expanded is as follows:
Step 5: building array.
Firstly, each data block in n data block is cut into w parts, such as D1,D2,D3,…,DnIt is cut into D1,1,
D1,2,…,D1,w,D2,1,…,Dn,w。
Obtain array:
Step 6: operation optimization
Operation statistics is carried out to entire array, in statistic array(i1,
I2, i3 ∈ { 1,2 ..., n }, j1, j2, j3 ∈ { 1,2 ..., w }) number that occurs of pattern.It is more than primary formula by number
Individually take out, according toThe sub- number of pattern multiplies 2,The sub- number of pattern multiply 3 after knot
Fruit sorts from large to small.Array is replaced according to ranking results, is replaced successfully labeled as Si, i is replacement serial number.Replacement is completed
Alternate form sublist S and last optimization array Z2 are obtained afterwards.
If the parameter being arranged does not change, that array module need to only be executed once, obtain array Z1 and Z2, and replace
Formula list S is changed, does not need to repeat.
Further, coding module 103 realizes that process is as follows:
Step 1: load initial data
Nw parts of data are taken out from the file to be encoded according to array request, respectively correspond D in array1,1,…,D1,w,
D2,1,…,Dn,w
Step 2: computing redundancy data
According to replacement list S, the value of each replacement formula is calculated.And then according to array Z2, calculate redundant digit
According to completion coding.
Further, decoder module 106 realizes that process is as follows:
Step 1: building decoding matrix
Firstly, being divided into (n+m) w block according to array Z1, so creation (n+m) w × (n+m) w matrix B, structure areI is the unit matrix of nw × nw, and O is the null matrix of nw × mw, and H is the check matrix of Z1.
Step 2: calculating and lose the data that data restore needs
Step 2-1: being the loss of data of entire data column because when node failure, so when restoring data, according to
The node of loss creates corresponding loss element list L.
Step 2-2: it loops through the element e lost in element list L and carries out step 2- if e belongs to data element
3, if e belongs to redundant elements, carry out the judgement of next element.
Step 2-3:r is the set for the row that e train value is 1 in check matrix H.If not having element in r, e row is set 0
(theoretically the element can not restore explanation).If there is element in r, that is just from the Hamming weight being chosen to remove after losing element in r
The smallest a line is measured as rl, in r the corresponding row of element and e row successively with rlXOR operation is carried out, finally by rlSet 0.
Step 2-4: 0 is all set after data element operations all in L are complete, then by the corresponding column of the redundant elements in L.
Step 3: restoring data
After completing step 2, if the corresponding behavior r in corresponding matrix B of the element e in loss element list Le, ceIt is
In reThe set for the column that intermediate value is 1.ceIn the corresponding data block of element be restore element e need data block.According to loss
Element list L and matrix B, it is available to restore all data blocks losing data and needing.Further according to the data of these data blocks
Exclusive or obtains corresponding loss data.
Another object of the present invention is to provide a kind of efficient height for efficiently highly reliable big data storage system can
The big data storage method leaned on, the efficiently highly reliable big data storage method include that data upload process, data were restored
Journey and data downloading process.
As shown in Fig. 2, data upload process provided in an embodiment of the present invention specifically includes:
Step 1: data uploading module receives the data uploaded, and the data of upload are sent to coding module.
Step 2: array module reads the parameter in relation to array in configuration center.
Step 3: array module polls are under corresponding parameter, if having corresponding array.If it is not, according to ginseng
Number, calculation code array, and the array being finally calculated is sent to coding module.If existing, directly array is sent out
Give coding module.
Step 4: coding module reads parameter of the configuration center in relation to coding, and receives the original of data uploading module transmission
The encoding array that data and array module are sent.Then cutting is carried out to initial data according to encoding array, pre-processed, compiled
Code calculates, and obtains redundant data.
Step 5: load balancing module reads the related related parameter of load balancing in configuration center.
Step 6: each node load state of load balancing module inquiry HDFS distributed memory system.
Step 7: load balancing module chooses the memory node column of piecemeal after coding according to parameter and each node load state
Table sends it to coding module.
Step 8: then the piecemeal memory node list that coding module balancing received load module is chosen is arranged according to node
Table stores corresponding piecemeal in HDFS distributed memory system respectively.
Wherein HDFS distributed memory system can be replaced with other distributed memory systems.
As shown in figure 3, data recovery procedure provided in an embodiment of the present invention specifically includes:
A1 step: the heart beat detection module moment controls host node and from heartbeat message is sent between node, if being arranged
In time, the heartbeat message of node is not received, then judges the node failure.And the information of the node failure is sent to data
Recovery module.
A2 step: data recovery module reads the parameter that related data are restored in configuration center.
A3 step: data recovery module restores related ginseng according to node failure information or piecemeal fail message and data
Number parses and loses node or the piecemeal corresponding position in encoding array, sends it to decoder module.
A4 step: load balancing module reads the related related parameter of load balancing in configuration center.
A5 step: load balancing module is chosen newborn node listing, is sent to according to parameter and each node load state
To decoder module.
A6 step: decoder module chooses decoding scheme according to node or the piecemeal corresponding position in encoding array is lost.
According to scheme, residue block data needed for HDFS distributed memory system is read.
A7 step: decoder module is decoded calculating according to remaining block data, obtains the data for losing piecemeal.It presses
According to the newborn node listing that load balancing module is chosen, the piecemeal recovered is stored in the newborn node of HDFS.
Data downloading process specifically includes:
B1 step: data download module inquires the state of each piecemeal of data according to the data ID to be downloaded, if piecemeal is complete
It is good, then piecemeal is assembled, downloads to client.It is lost if there is piecemeal, then losing blocking information, to be sent to data extensive
Multiple module.
A2 step: data recovery module reads the parameter that related data are restored in configuration center.
A3 step: data recovery module restores related ginseng according to node failure information or piecemeal fail message and data
Number parses and loses node or the piecemeal corresponding position in encoding array, sends it to decoder module.
A4 step: load balancing module reads the related related parameter of load balancing in configuration center.
A5 step: load balancing module is chosen newborn node listing, is sent to according to parameter and each node load state
To decoder module.
A6 step: decoder module chooses decoding scheme according to node or the piecemeal corresponding position in encoding array is lost.
According to scheme, residue block data needed for HDFS distributed memory system is read.
A7 step: decoder module is decoded calculating according to remaining block data, obtains the data for losing piecemeal.It presses
According to the newborn node listing that load balancing module is chosen, the piecemeal recovered is stored in the newborn node of HDFS.
B8 step: data download module reads the data of each piecemeal, and block data is assembled into, downloads to client.
Wherein HDFS distributed memory system can be replaced with other distributed memory systems.
Fig. 4 is n=5, the correcting and eleting codes and n=5 of the present invention of m=2 (data number of blocks is 5, fault-tolerant ability 2)
EVENODD code, (each document No. 10 times removes most for the scramble time comparison encoded respectively to the file of 1MB-10MB
Average value after big minimum value).Fig. 5 is n=5, the correcting and eleting codes of the present invention of m=3 (data number of blocks is 5, fault-tolerant ability 3) with
The Star code of n=5, scramble time comparison (each document No. 10 times, the removing that the file of 1MB-10MB is encoded respectively
Average value after maximin).
Correcting and eleting codes of the present invention are transformed by calculating complicated RS code, from two figures it can be seen that correcting and eleting codes of the present invention exist
It is slightly above on operation efficiency with the EVENODD code and Star code in efficiently famous array code, and the appearance of correcting and eleting codes of the present invention
There is no limit this has also further embodied the practicability of correcting and eleting codes of the present invention for wrong ability and stripe size.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Made any modifications, equivalent replacements, and improvements etc., should all be included in the protection scope of the present invention within mind and principle.
Claims (10)
1. a kind of efficiently highly reliable big data storage system, which is characterized in that the efficiently highly reliable big data stores system
System includes:
The data for obtaining user and uploading are arranged according to configuration center, and transfer data to coding module for data uploading module；
Array module calculates the array of coding according to the parameter of configuration center, and stores；
Coding module receives the data of data uploading module transmission, reads the array that array module calculates, carries out according to array
Coding；
Heart beat detection module, host node and from heartbeat message is sent between node mutually, when reading the heartbeat delay of configuration center
Between, if it exceeds the time, then judge the node failure, and send information to data recovery module；
Data recovery module parses failure node information, reads configuration center according to the feedback information of heartbeat inspecting module
Data restore relevant parameter, parse failure node corresponding location information in an array, transmit to decoder module, counted
According to recovery；
Decoder module obtains the failure node location information of data recovery module transmission, then carries out data recovery；
Data download module inquires the position of each initial data piecemeal according to the data ID to be downloaded, if each original number
Exist according to piecemeal and lose, then advanced row data are restored, if all existed, all initial data piecemeals are downloaded, and carry out
It is assembled into initial data；
Load balancing module reads the setting of configuration center, and the piecemeal memory node and data after being responsible for allocated code restore
When the newborn node that selects；
Configuration center, the various parameters being responsible in management system；
Distributed memory system, the piecemeal after storing coding.
2. efficiently highly reliable big data storage system as described in claim 1, which is characterized in that the array module packet
It includes:
Parameter set unit is used for plan of establishment parameter, and parameter includes: initial data number of blocks n, verifies number of blocks m, finite field
Size GF (2w)；
Matrix construction unit, for constructing finite field gf (2w) and corresponding binary matrix and Cauchy matrix, and by Cauchy's square
Battle array is extended to binary matrix；
Array Construction unit constructs array by binary matrix；
Operation optimizes unit, carries out operation statistics to entire array.
3. efficiently highly reliable big data storage system as described in claim 1, which is characterized in that the coding module packet
It includes:
Initial data loading unit takes out corresponding data according to array request from the file to be encoded；
Redundant data computing unit calculates the value of each replacement formula according to replacement list, goes out redundancy further according to array computation
Data.
4. efficiently highly reliable big data storage system as described in claim 1, which is characterized in that the decoder module packet
It includes；
Decoding unit, for according to Array Construction decoding matrix；
Data Computation Unit, for calculating the data for losing data recovery needs；
Data recovery unit, the data block that all loss data that are restored need；Further according to the data exclusive or of these data blocks
Obtain corresponding loss data.
5. efficiently highly reliable big data storage system is efficient highly reliable described in a kind of operation claim 1-4 any one
Big data storage method, which is characterized in that the efficiently highly reliable big data storage method includes data upload process, data
Recovery process and data downloading process；
The array module of the efficiently highly reliable big data storage method realizes that process is as follows:
Step 1: plan of establishment parameter, parameter include: initial data number of blocks n, verify number of blocks m, finite field size GF (2w)；
Step 2: building finite field gf (2w) and corresponding binary matrix；
If root is primitive polynomial g (x)=x of αa+xb…+xc+ 1, (a > b > ... > c) generates a finite field gf (2w), i.e.,
αa=αb+…αc+ 1, construction process is as follows:
α0=1
α1=α
α2=α2
αa=αb+…+αc+1
αa+1=αa× α=αb+1+…+αc+1+α
It is the multinomial element representation in finite field above；
Coefficient vector V (x)=(f of each finite field multinomial element f (x) is obtained again0,f1,…,fw-1)；
Then, the corresponding binary matrix β (e) of building V (x), for arbitrary element e ∈ GF (2w), β (e) is the two of a w × w
System matrix, wherein i-th is classified as xiThe coefficient vector of e mod p (x)；
Step 3: building Cauchy matrix；
From GF (2w) m element and n element composition X={ x are chosen respectively1,x2,…,xm, Y={ y1,y2,…,ynBuilding Ke
The Cauchy matrix of western matrix, building is as follows:
Step 4: Cauchy matrix is extended to binary matrix；
The corresponding binary matrix β (e) that element in Cauchy matrix obtained in step 3 is obtained according to step 2, by m
The Cauchy matrix of × n is extended to the binary matrix of mw × nw；
Binary matrix after being expanded is as follows:
Step 5: building array；
Firstly, each data block in n data block is cut into w parts, such as D1,D2,D3,…,DnIt is cut into D1,1,D1,2,…,
D1,w,D2,1,…,Dn,w；
Obtain array:
Step 6: operation optimization
Operation statistics is carried out to entire array, in statistic array(i1,i2,i3
∈ { 1,2 ..., n }, j1, j2, j3 ∈ { 1,2 ..., w }) number that occurs of pattern；It is more than that primary formula individually takes by number
Out, according toThe sub- number of pattern multiplies 2,The sub- number of pattern multiply 3 after result from
Small sequence is arrived greatly；Array is replaced according to ranking results, is replaced successfully labeled as Si, i is replacement serial number, after the completion of replacement
To alternate form sublist S and last optimization array Z2；
If the parameter being arranged does not change, that array module need to only be executed once, obtain array Z1 and Z2 and alternate form
Sublist S does not need to repeat；
The coding module of the efficiently highly reliable big data storage method realizes that process is as follows:
Step 1: load initial data
Nw parts of data are taken out from the file to be encoded according to array request, respectively correspond D in array1,1,…,D1,w,D2,1,…,
Dn,w；
Step 2: computing redundancy data
According to replacement list S, the value of each replacement formula is calculated；And then according to array Z2, redundant data is calculated, it is complete
At coding；
The decoder module of the efficiently highly reliable big data storage method realizes that process is as follows:
Step 1: building decoding matrix
Firstly, being divided into (n+m) w block according to array Z1, so creation (n+m) w × (n+m) w matrix B, structure areI is the unit matrix of nw × nw, and O is the null matrix of nw × mw, and H is the check matrix of Z1；
Step 2 calculates and loses the data that data restore needs
It (1) is the loss of data of entire data column because when node failure, so when restoring data, according to the section of loss
Point creates corresponding loss element list L；
(2) loop through lose element list L in element e carry out (3) if e belongs to data element, if e belong to it is superfluous
Remaining element then carries out the judgement of next element；
(3) r is the set for the row that e train value is 1 in check matrix H；If not having element in r, e row is set 0；If had in r
Element, that is just from the smallest a line of Hamming weight being chosen to remove after losing element in r as rl, the corresponding row of element and e in r
Row successively with rlXOR operation is carried out, finally by rlSet 0；
(4) after data element operations all in L are complete, then the corresponding column of the redundant elements in L are all set 0；
Step 3 restores data；
After completing step 2, if the corresponding behavior r in corresponding matrix B of the element e in loss element list Le, ceBe
reThe set for the column that intermediate value is 1；ceIn the corresponding data block of element be restore element e need data block；It is first according to losing
Plain list L and matrix B, the data block that all loss data that are restored need；It is obtained further according to the data exclusive or of these data blocks
To corresponding loss data.
6. efficiently highly reliable big data storage method as claimed in claim 5, which is characterized in that the data upload process
It specifically includes:
Data uploading module receives the data uploaded, and the data of upload are sent to coding module；
Array module reads the parameter in relation to array in configuration center；
Array module polls are under corresponding parameter, if having corresponding array, if it is not, calculating and compiling according to parameter
Code array, and the array being finally calculated is sent to coding module, if existing, array is directly sent to coding mould
Block；
Coding module reads parameter of the configuration center in relation to coding, and receives the initial data and battle array of data uploading module transmission
Then the encoding array that columnization module is sent carries out cutting, pretreatment to initial data according to encoding array, coding is calculated, obtained
To redundant data；
Load balancing module reads the related related parameter of load balancing in configuration center；
Each node load state of load balancing module Querying Distributed storage system；
Load balancing module is chosen the memory node list of piecemeal after coding, is sent out according to parameter and each node load state
Give coding module；
The piecemeal memory node list that coding module balancing received load module is chosen stores respectively then according to node listing
Corresponding piecemeal is in distributed memory system.
7. efficiently highly reliable big data storage method as claimed in claim 5, which is characterized in that the data recovery procedure
It specifically includes:
The heart beat detection module moment controls host node and from heartbeat message is sent between node, if do not had within the setting time
The heartbeat message of node is received, then judges the node failure, and the information of the node failure is sent to data recovery module；
Data recovery module reads the parameter that related data are restored in configuration center；
Data recovery module restores related parameter according to node failure information or piecemeal fail message and data, parses and loses
Disloyal point or piecemeal corresponding position in encoding array, send it to decoder module；
Load balancing module reads the related related parameter of load balancing in configuration center；
Load balancing module chooses newborn node listing, sends it to decoder module according to parameter and each node load state；
Decoder module chooses decoding scheme according to node or the piecemeal corresponding position in encoding array is lost, and according to scheme, reads
Take residue block data needed for HDFS distributed memory system；
Decoder module is decoded calculating according to remaining block data, the data for losing piecemeal is obtained, according to load balancing mould
The newborn node listing that block is chosen, the piecemeal recovered is stored in the newborn node of distributed memory system.
8. efficiently highly reliable big data storage method as claimed in claim 5, which is characterized in that the data downloading process
It specifically includes:
Data download module inquires the state of each piecemeal of data according to the data ID to be downloaded, if piecemeal is intact, by piecemeal
It assembles, downloads to client, lost if there is piecemeal, then will lose blocking information and be sent to data recovery module；
Data recovery module reads the parameter that related data are restored in configuration center；
Data recovery module restores related parameter according to node failure information or piecemeal fail message and data, parses and loses
Disloyal point or piecemeal corresponding position in encoding array, send it to decoder module；
Load balancing module reads the related related parameter of load balancing in configuration center；
Load balancing module chooses newborn node listing, sends it to decoder module according to parameter and each node load state；
Decoder module chooses decoding scheme according to node or the piecemeal corresponding position in encoding array is lost, and according to scheme, reads
Take residue block data needed for distributed memory system；
Decoder module is decoded calculating according to remaining block data, the data for losing piecemeal is obtained, according to load balancing mould
The newborn node listing that block is chosen, the piecemeal recovered is stored in the newborn node of distributed memory system；
Data download module reads the data of each piecemeal, and block data is assembled into, downloads to client.
9. a kind of computer program, which is characterized in that the computer program has for any in perform claim requirement 5 to 8
The program coding of efficient highly reliable big data storage method described in, wherein the computer program is run on computers.
10. a kind of storage medium for being stored with computer program as claimed in claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910681574.2A CN110457161A (en) | 2019-07-26 | 2019-07-26 | A kind of efficiently highly reliable big data storage system, method, computer program |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910681574.2A CN110457161A (en) | 2019-07-26 | 2019-07-26 | A kind of efficiently highly reliable big data storage system, method, computer program |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110457161A true CN110457161A (en) | 2019-11-15 |
Family
ID=68483648
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910681574.2A Pending CN110457161A (en) | 2019-07-26 | 2019-07-26 | A kind of efficiently highly reliable big data storage system, method, computer program |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110457161A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143108A (en) * | 2019-12-09 | 2020-05-12 | 成都信息工程大学 | Encoding and decoding method and device for reducing array code Xcode repair |
| CN111539870A (en) * | 2020-02-25 | 2020-08-14 | 成都信息工程大学 | New media image tampering recovery method and device based on erasure codes |
| CN111585581A (en) * | 2020-05-14 | 2020-08-25 | 成都信息工程大学 | Coding method based on binary domain operation and supporting any code distance |
| CN111585582A (en) * | 2020-05-14 | 2020-08-25 | 成都信息工程大学 | Coding method based on array operation and freely determined code distance |
| CN111679793A (en) * | 2020-06-16 | 2020-09-18 | 成都信息工程大学 | Single-disk fault rapid recovery method based on STAR code |
| CN111858169A (en) * | 2020-07-10 | 2020-10-30 | 山东云海国创云计算装备产业创新中心有限公司 | Data recovery method, system and related components |
| CN112052114A (en) * | 2020-08-27 | 2020-12-08 | 江苏超流信息技术有限公司 | Data storage and recovery method, coder-decoder and coder-decoder system |
| CN112417527A (en) * | 2020-11-26 | 2021-02-26 | 南京邮电大学 | Energy internet data reliable storage system considering attack |
| CN113051428A (en) * | 2019-12-28 | 2021-06-29 | 浙江宇视科技有限公司 | Method and device for storing and backing up front end of camera |
| CN113391946A (en) * | 2021-05-25 | 2021-09-14 | 杭州电子科技大学 | Coding and decoding method for erasure code in distributed storage |
| WO2022110691A1 (en) * | 2020-11-24 | 2022-06-02 | 深圳市大数据研究院 | Coding method, decoding method, electronic device and storage medium |
| CN114595092A (en) * | 2022-04-28 | 2022-06-07 | 阿里云计算有限公司 | Distributed storage system, data reconstruction method, device and storage medium |
| CN116560915A (en) * | 2023-07-11 | 2023-08-08 | 北京谷数科技股份有限公司 | Data recovery method and device, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105530294A (en) * | 2015-12-04 | 2016-04-27 | 中科院成都信息技术股份有限公司 | Mass data distributed storage method |
| CN107086870A (en) * | 2017-03-16 | 2017-08-22 | 东莞理工学院 | Repair the MDS array codes coding and coding/decoding method of more piece point failure |
-
2019
- 2019-07-26 CN CN201910681574.2A patent/CN110457161A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105530294A (en) * | 2015-12-04 | 2016-04-27 | 中科院成都信息技术股份有限公司 | Mass data distributed storage method |
| CN107086870A (en) * | 2017-03-16 | 2017-08-22 | 东莞理工学院 | Repair the MDS array codes coding and coding/decoding method of more piece point failure |
Non-Patent Citations (1)
| Title |
|---|
| WEI YUAN 等: "Research on Multi-fault-tolerant MDS Array Erasure code", 《JOURNAL OF PHYSICS: CONFERENCE SERIES》 * |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143108A (en) * | 2019-12-09 | 2020-05-12 | 成都信息工程大学 | Encoding and decoding method and device for reducing array code Xcode repair |
| CN111143108B (en) * | 2019-12-09 | 2023-05-02 | 成都信息工程大学 | Coding and decoding method and device for reducing array code Xcode repair |
| CN113051428A (en) * | 2019-12-28 | 2021-06-29 | 浙江宇视科技有限公司 | Method and device for storing and backing up front end of camera |
| CN113051428B (en) * | 2019-12-28 | 2024-04-05 | 浙江宇视科技有限公司 | Method and device for back-up storage at front end of camera |
| CN111539870B (en) * | 2020-02-25 | 2023-07-14 | 成都信息工程大学 | Tamper recovery method and device for new media image based on erasure codes |
| CN111539870A (en) * | 2020-02-25 | 2020-08-14 | 成都信息工程大学 | New media image tampering recovery method and device based on erasure codes |
| CN111585582A (en) * | 2020-05-14 | 2020-08-25 | 成都信息工程大学 | Coding method based on array operation and freely determined code distance |
| CN111585581A (en) * | 2020-05-14 | 2020-08-25 | 成都信息工程大学 | Coding method based on binary domain operation and supporting any code distance |
| CN111585582B (en) * | 2020-05-14 | 2023-04-07 | 成都信息工程大学 | Coding method based on array operation and freely determined code distance |
| CN111585581B (en) * | 2020-05-14 | 2023-04-07 | 成都信息工程大学 | Coding method based on binary domain operation and supporting any code distance |
| CN111679793A (en) * | 2020-06-16 | 2020-09-18 | 成都信息工程大学 | Single-disk fault rapid recovery method based on STAR code |
| CN111679793B (en) * | 2020-06-16 | 2023-03-14 | 成都信息工程大学 | Single-disk fault rapid recovery method based on STAR code |
| CN111858169A (en) * | 2020-07-10 | 2020-10-30 | 山东云海国创云计算装备产业创新中心有限公司 | Data recovery method, system and related components |
| CN112052114A (en) * | 2020-08-27 | 2020-12-08 | 江苏超流信息技术有限公司 | Data storage and recovery method, coder-decoder and coder-decoder system |
| CN112052114B (en) * | 2020-08-27 | 2024-05-07 | 江苏超流信息技术有限公司 | Data storage and recovery method, coder and decoder and coding and decoding system |
| WO2022110691A1 (en) * | 2020-11-24 | 2022-06-02 | 深圳市大数据研究院 | Coding method, decoding method, electronic device and storage medium |
| CN112417527B (en) * | 2020-11-26 | 2022-11-11 | 南京邮电大学 | Energy internet data reliable storage system considering attack |
| CN112417527A (en) * | 2020-11-26 | 2021-02-26 | 南京邮电大学 | Energy internet data reliable storage system considering attack |
| CN113391946B (en) * | 2021-05-25 | 2022-06-17 | 杭州电子科技大学 | Coding and decoding method for erasure codes in distributed storage |
| CN113391946A (en) * | 2021-05-25 | 2021-09-14 | 杭州电子科技大学 | Coding and decoding method for erasure code in distributed storage |
| CN114595092B (en) * | 2022-04-28 | 2022-09-20 | 阿里云计算有限公司 | Distributed storage system, data reconstruction method, device and storage medium |
| CN114595092A (en) * | 2022-04-28 | 2022-06-07 | 阿里云计算有限公司 | Distributed storage system, data reconstruction method, device and storage medium |
| CN116560915A (en) * | 2023-07-11 | 2023-08-08 | 北京谷数科技股份有限公司 | Data recovery method and device, electronic equipment and storage medium |
| CN116560915B (en) * | 2023-07-11 | 2023-09-19 | 北京谷数科技股份有限公司 | Data recovery method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110457161A (en) | A kind of efficiently highly reliable big data storage system, method, computer program | |
| Huang et al. | Pyramid codes: Flexible schemes to trade space for access efficiency in reliable data storage systems | |
| Greenan et al. | Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs | |
| Silberstein et al. | Lazy means smart: Reducing repair bandwidth costs in erasure-coded distributed storage | |
| CN109643258B (en) | Multi-node repair using high-rate minimal storage erase code | |
| Xin et al. | Reliability mechanisms for very large storage systems | |
| US11500725B2 (en) | Methods for data recovery of a distributed storage system and storage medium thereof | |
| CN104364765A (en) | Method of data storing and maintenance in a distributed data storage system and corresponding device | |
| US20150127974A1 (en) | Method of storing a data item in a distributed data storage system, corresponding storage device failure repair method and corresponding devices | |
| Zhang et al. | Tip-code: A three independent parity code to tolerate triple disk failures with optimal update complextiy | |
| CN107086870A (en) | Repair the MDS array codes coding and coding/decoding method of more piece point failure | |
| CN107844272A (en) | A kind of cross-packet coding and decoding method for improving error correcting capability | |
| CN105518996B (en) | A kind of data decoding method based on binary field reed-solomon code | |
| CN106484559A (en) | A kind of building method of check matrix and the building method of horizontal array correcting and eleting codes | |
| CN107689983B (en) | Cloud storage system and method based on low repair bandwidth | |
| CN103412799A (en) | Data recovery method, data recovery equipment and distributed storage system | |
| Ramkumar et al. | Codes for distributed storage | |
| Qiu et al. | Ec-fusion: An efficient hybrid erasure coding framework to improve both application and recovery performance in cloud storage systems | |
| Zhang et al. | Aggrecode: Constructing route intersection for data reconstruction in erasure coded storage | |
| Song et al. | A Low complexity design of reed solomon code algorithm for advanced RAID system | |
| Xie et al. | AZ-Recovery: An efficient crossing-AZ recovery scheme for erasure coded cloud storage systems | |
| Li | Enabling low degraded read latency and fast recovery for erasure coded cloud storage systems | |
| Wei et al. | expanCodes: Tailored LDPC codes for big data storage | |
| CN110781025B (en) | Symmetrical partial repetition code construction and fault node repairing method based on complete graph | |
| Wang et al. | An erasure code with low recovery-overhead based on a particular three-hierarchical redundancy structure |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191115 |