CN103577546A - Method and equipment for data backup, and distributed cluster file system - Google Patents
Method and equipment for data backup, and distributed cluster file system Download PDFInfo
- Publication number
- CN103577546A CN103577546A CN201310476405.8A CN201310476405A CN103577546A CN 103577546 A CN103577546 A CN 103577546A CN 201310476405 A CN201310476405 A CN 201310476405A CN 103577546 A CN103577546 A CN 103577546A
- Authority
- CN
- China
- Prior art keywords
- node
- cluster
- file
- backup
- machine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/184—Distributed file systems implemented as replicated file system
- G06F16/1844—Management specifically adapted to replicated file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a method and equipment for data backup, and a distributed cluster file system, wherein the method comprises the steps of scanning the file range in the charge of a node in a backup transfer table by a background backup thread of each node in a first cluster; for a scanned written-in file, writing the written-in file into a second cluster by the background backup thread of the node in the first cluster. The files can be automatically backed up into other clusters by the embodiment of the invention, and the safety of the backup files is improved.
Description
Technical field
The present invention relates to field of computer technology, relate in particular to a kind of method, equipment and distributed cluster file system of data backup.
Background technology
Cassandra is the database of the non-relation of a mixed type, the principal feature of Cassandra be exactly it be not a database, but the distributed network services jointly being formed by a pile database node, to Cassandra write operation, can be copied to other nodes gets on, read operation to Cassandra, also can be routed to certain and above node, go to read.For a Cassandra, troop, scalability is fairly simple thing, adds node by all means just passable in the inside of trooping.
Yet, contrast this quasi-tradition database of existing MYSQL (Relational DBMS), Cassandra cluster lacks Database Automatic Backup mechanism, at present the backup instrument of Cassandra data is limited in cluster the data backup of a node to another node, is not directed to the backup scenario of whole Cassandra cluster data.By the backing up to another cluster and obviously can greatly improve the safety and reliability of data of the data integrity in a cluster, thereby how efficiently data backup to be become to a problem demanding prompt solution to another cluster.
Summary of the invention
In view of the above problems, the present invention has been proposed to a kind of method, equipment and distributed cluster file system of a kind of data backup that overcomes the problems referred to above or address the above problem are at least in part provided, can be by file automated back-up in other clusters.
According to an aspect of the present invention, a kind of method of data backup is provided, for distributed cluster file system, described distribution cluster file system at least comprises: the first cluster and the second cluster, and described method comprises: the backstage backup thread of each node in described the first cluster scans the responsible file extent of this node in backup transfer table; The backstage backup thread of the node in described the first cluster is written to the file writing scanning in described the second cluster.
Alternatively, the backstage backup thread of each node in described the first cluster is to before the step that in backup transfer table, the responsible file extent of this node scans, and described method also comprises: the access node in described the first cluster stores the file allocation writing to the respective nodes of described the first cluster; The file of said write and the write request information corresponding with the file of said write are synchronously write to backup transfer table.
Alternatively, after the backstage backup thread of the node in described the first cluster is written to the step in described the second cluster by the file of said write, described method also comprises: the file of said write and write request information thereof are deleted from described backup transfer table.
Alternatively, wherein, described backup transfer table comprises: the cryptographic hash of the filename of the file of said write and write request information, and wherein said write request information comprises: the file of said write is positioned at the address information of the first cluster and the/the second cluster;
The step that access node in described the first cluster stores the file allocation writing to the respective nodes of described the first cluster comprises: the access node in described the first cluster is given corresponding node in described the first cluster according to the cryptographic hash of the filename of said write by the file allocation of said write;
The backstage backup thread of each node in described the first cluster comprises the step that in described backup transfer table, the responsible file extent of this node scans: the backstage backup thread of each node in described the first cluster, according to the cryptographic hash of the filename writing, scans the responsible file extent of this node in described backup transfer table.
Alternatively, the backstage backup thread of each node in described the first cluster is to before the step that in described backup transfer table, the responsible file extent of this node scans, and described method also comprises: detect the node that whether has the machine of delaying in described the first cluster;
When the node of machine is delayed in existence, the backstage backup thread of each node in described the first cluster comprises the step that in described backup transfer table, the responsible file extent of this node scans: described in the delay scope of the responsible file of the node of the machine of delaying described in the next node scanning of node of machine;
The step that the backstage backup thread of the node in described the first cluster is written to the file of said write in described the second cluster comprises: described in the delay backstage backup thread of next node of node of machine the responsible file of the node of the described machine of delaying is written in described the second cluster.
Alternatively, in described the first cluster of described detection, whether exist the step of the node of the machine of delaying to comprise: to detect each node in described the first cluster and regularly to some or all other nodes, send heartbeat message; When first node surpassed for first schedule time while not receiving the heartbeat message of Section Point, judge the described Section Point machine of delaying; Described first node is by the delay next node of Section Point described in the information notification of machine of described Section Point.
Alternatively, in described the first cluster of described detection, whether exist the step of the node of the machine of delaying also to comprise: the next node of the Section Point in described the first cluster is delayed the message stores of machine at local internal memory by described Section Point; When the next node of described Section Point is not intercepted described Section Point and delayed the message of machine by background thread, the internal memory of next node quantitative check this locality of described Section Point, detects the described Section Point machine of whether delaying.
Alternatively, the next node of described second node adopts an other background thread, and second responsible scope of node described in described backup transfer table scanned.
Alternatively, the step that access node in described the first cluster stores the file allocation writing to the respective nodes of described the first cluster is: the access node in described the first cluster stores the file allocation writing to the respective nodes of described the first cluster, and is stored in the next node of described respective nodes or in lower two nodes.
According to another aspect of the present invention, also provide a kind of equipment of data backup, be arranged in the first cluster of distributed cluster file system, described equipment comprises: scan module, for the responsible file extent of backup transfer table is scanned; Backup module, for being written to the file writing scanning the second cluster of described distributed cluster file system.
Alternatively, described equipment also comprises: memory module, for storing the file allocation writing to the respective nodes of described the first cluster; Backup transfer table writing module, for synchronously writing backup transfer table by the file of said write and the write request information corresponding with the file of said write.
Alternatively, described equipment also comprises: removing module, and for the file of said write and write request information thereof are deleted from described backup transfer table.
Alternatively, described backup transfer table comprises: the cryptographic hash of the filename of the file of said write and write request information, and wherein said write request information comprises: the file of said write is positioned at the address information of the first cluster and the/the second cluster;
Described memory module is further used for by the file allocation of said write, giving corresponding node in described the first cluster according to the cryptographic hash of the filename of said write;
Described scan module is further used for, according to the cryptographic hash of the filename writing, the responsible file extent of this node in described backup transfer table being scanned.
Alternatively, described equipment also comprises: the machine testing module of delaying, for detection of whether there being the node of the machine of delaying in described the first cluster;
Described scan module be further used for delaying described in the scanning scope of the responsible file of the node of machine;
Described backup module is further used for the responsible file of the node of the described machine of delaying to be written in described the second cluster.
Alternatively, the machine testing module of delaying described in is further used for detecting each node in described the first cluster and regularly to some or all other nodes, sends heartbeat message; When surpassing for first schedule time while not receiving the heartbeat message of Section Point, judge the described Section Point machine of delaying; By the delay next node of Section Point described in the information notification of machine of described Section Point.
Alternatively, the machine testing module of delaying described in is also further used for described Section Point to delay the message stores of machine at local internal memory; When not intercepting described Section Point and delay the message of machine, the internal memory of quantitative check this locality, detects the described Section Point machine of whether delaying.
According to a further aspect of the invention, also provide a kind of distributed cluster file system, wherein, comprised the equipment of data backup as above.
As shown from the above technical solution, embodiments of the invention have following beneficial effect: in an embodiment of the present invention, the backstage backup thread of each node in the first cluster scans the responsible file extent of this node in backup transfer table; The backstage backup thread of the node in the first cluster is written to the file writing scanning in described the second cluster, realizes file automated back-up in other clusters, effectively improves the security of backup file.
Above-mentioned explanation instrument is the general introduction of technical solution of the present invention, in order to better understand technological means of the present invention, and can be implemented according to the content of instructions, and for above and other objects of the present invention, feature and advantage can be become apparent, below especially exemplified by the specific embodiment of the present invention.
Accompanying drawing explanation
By reading below detailed description of the preferred embodiment, various other advantage and benefits will become cheer and bright for those of ordinary skills.Accompanying drawing instrument is used for the object of preferred implementation is shown, and does not think limitation of the present invention.And in whole accompanying drawing, by identical reference symbol, represent identical parts.In the accompanying drawings:
Fig. 1 shows according to the process flow diagram of the method 100 of the data backup of an embodiment of the invention;
Fig. 2 shows the structural representation of cluster in distribution cluster file system according to the embodiment of the present invention; And
Fig. 3 shows the block diagram of equipment 300 of the data backup of another embodiment of the invention.
Embodiment
Exemplary embodiment of the present disclosure is described below with reference to accompanying drawings in more detail.Although shown exemplary embodiment of the present disclosure in accompanying drawing, yet should be appreciated that and can realize the disclosure and the embodiment that should do not set forth limits here with various forms.On the contrary, it is in order more thoroughly to understand the disclosure that these embodiment are provided, and can by the scope of the present disclosure complete convey to those skilled in the art.
Referring to Fig. 1, wherein schematically show the process flow diagram of the method 100 of data backup according to an embodiment of the invention, the method 100 is for distributed cluster file system, and the cluster file system that wherein distributes at least comprises: the first cluster and the second cluster.As shown in Figure 2, be the structural representation of cluster in the cluster file system that distributes in embodiments of the invention, there are 4 nodes (node1～4).Each node is responsible for this node to the middle region (also claiming the master file scope that node is responsible) of clockwise next node.Each node in cluster can be as access node (access contact can optionally).Access node to the key (key) of read-write (for example: the filename of the file that store), calculate the hash value of key, and then can navigate to the node under this file, realize the access (filename of the file that storage will be stored and file content) of file.
Alternatively, in an embodiment of the present invention, above-mentioned calculating hash value-based algorithm is as follows: the MD5 value of calculation document name first, then by the integer of the 0～2^127 in MD5 value conversion side calculating, and then can navigate to the position of this file in cluster.
What certainly can understand is, embodiments of the invention are not limited to use the cluster of ring topology, other for example centre type topological structure also can use similar scheme to carry out the backup across cluster, even embodiments of the invention also can be extended to relevant database cluster, other embodiment and embodiments of the invention are similar, at this, no longer apply and state.
As shown in Figure 1, described method 100 comprises step S110, S130, and method 100 starts from step S110, and wherein, the backstage backup thread of each node in the first cluster scans the responsible file extent of this node in backup transfer table.
Alternatively, in an embodiment of the present invention, backup transfer table comprises: the cryptographic hash of the filename of the file writing and write request information, wherein write request information comprises: the file writing is positioned at the address information of the first cluster and the/the second cluster.
See table, the form for the backup transfer table in embodiments of the invention, certainly can understand, and does not limit in an embodiment of the present invention the concrete form of backup transfer table.
| ? | Column1 | Column2 | ... |
| Key1 (filename) | Value1 (file content) | Value12 (file content) | ? |
| Key2 (filename) | Value2 (file content) | Value22 (file content) | ? |
Wherein, column (row) is separated with ' _ ' by the keyspace asking (database name), columnfamily (name of table), columnname (row name), is spliced.
In upper table, it shows that Key1 is stored in two clusters, and these two clusters can be specified by user.The filename that will store in node and file content are all recorded in backup transfer table, that is to say the backup transfer table of the common use of all nodes in cluster.
Alternatively, in an embodiment of the present invention, step S110 is specially: the backstage backup thread of each node in the first cluster, according to the cryptographic hash of the filename writing, scans the responsible file extent of this node in backup transfer table.
In an embodiment of the present invention, backup transfer table can be stored in the first cluster on each node, and each node is responsible for the data that a storage part is about to backup.Certain file specifically by which node is responsible for, and is also to decide according to the calculating MD5 value of this document name.Backup transfer table is stored part at each node, when it upgrades to whole node broadcasts.
Subsequently, in step S130, for the file writing scanning, the backstage backup thread of the node in the first cluster is written to the file writing in the second cluster.
Each node starts background thread, the responsible file extent of this node in backup transfer scanned, and during scanning, be the MD5 that describes each filename in backup transfer table, all to calculate corresponding interior MD5 at every turn.To the filename scanning, first local (from backup transfer table) reads out real value, then calls relevant interface, and (also can be described as backup cluster, backup cluster can be any nosql database or distributed file system to be written to the second cluster.
In an embodiment of the present invention, above-mentioned interface, backs up the incoming interface of writing of cluster.Physical interface depends on the classification that backs up cluster, such as using hbase interface, and the thrift data writing interface that interface provides for hbase.
According to embodiments of the invention, the method 100 of described data backup can also comprise one or more optional step, to realize extra or additional function, yet these optional steps are not indispensable for realizing object of the present invention, the method 100 of data backup can, in the situation that there is no these optional steps, realize object of the present invention completely according to an embodiment of the invention.These optional steps are not shown in Figure 1, but priority between they and above steps is carried out and can according to following instruction, easily be drawn by those skilled in the art.It is pointed out that unless otherwise specified, these optional steps can be selected according to actual needs together with the execution sequence of above-mentioned steps.
Alternatively, in an embodiment of the present invention, before step S110, method 100 also comprises: step S150 and step S170, in step S150, the access node in the first cluster stores the file allocation writing to the respective nodes of the first cluster.Particularly, the access node in the first cluster is given corresponding node in the first cluster according to the cryptographic hash of the filename writing by the file allocation writing.
Subsequently, in step S170, the file writing and the write request information corresponding with the file writing are synchronously write to backup transfer table.
Alternatively, after step S130, method 100 also comprises: step S190 in step S190, deletes the file writing and write request information thereof from backup transfer table.
Alternatively, in an embodiment of the present invention, before step S110, method 100 also comprises: determine the node that whether has the machine of delaying in the first cluster.
When the node of machine is delayed in existence, the backstage backup thread of each node in the first cluster comprising the step (S110) that in backup transfer table, the responsible file extent of this node scans: the delay scope of the responsible file of the node of machine of the next node scanning of the node of the machine of delaying; The backstage backup thread of next node of node of machine of delaying is written to the responsible file of the node of the machine of delaying in the second cluster.
Alternatively, in an embodiment of the present invention, determine in the first cluster and whether exist the mode of the node of the machine of delaying to comprise:
Mode one, detect in the first cluster each node and regularly to some or all other nodes, send heartbeat message; When first node surpassed for first schedule time while not receiving the heartbeat message of Section Point, judge the Section Point machine of delaying; First node is by the delay next node of information notification Section Point of machine of Section Point.
The next node of the Section Point in mode two, the first cluster is delayed the message stores of machine at local internal memory by Section Point; When the next node of Section Point is not intercepted Section Point and delayed the message of machine by background thread, the internal memory of next node quantitative check this locality of Section Point, determines the Section Point machine of whether delaying.
Continuation is referring to Fig. 2, and all nodes in cluster, can get by current mechanism the state of other nodes in cluster.Suppose that Section Point is the clockwise next node of first node.During any one below meeting in two conditions, Section Point is responsible for taking over the backup tasks of first node.
A) Section Point is received the delay message of machine of a node (first node) on it
Each node of cluster can keep a long connection with other all nodes.Node can regularly send heartbeat message to some other nodes.When the 3rd node surpasses certain hour, do not receive after the heartbeat message of first node, will think that first node delayed machine.At this moment the 3rd node can notify other all node first nodes to delay machine, and Section Point has just been received the machine of the delaying message of first node like this.
B) background thread that Section Point starts, regularly detects the first node machine of delaying
The 3rd node is sent out message to Section Point, and notice first node has been delayed machine, and Section Point can be stored in local internal storage structure S (because Section Point is directly known the delay message of machine of first node) the machine information of delaying of first node.Backup thread (being yet background thread), may, due to abnormal cause, not intercept this message.At this moment, just by backup thread, regularly go to look into lower local S structure, detect first node and whether delayed machine.
Alternatively, in an embodiment of the present invention, the next node of second node adopts an other background thread, and second responsible scope of node in described backup transfer table scanned.
Alternatively, in an embodiment of the present invention, the file that each node is stored is also stored in its next node or in lower two nodes.
Below in conjunction with concrete example, introduce embodiments of the invention, it comprises three phases, (1) normal write phase; (2) Normal Backups stage and (3) node backup phase of machine when abnormal of delaying
(1) normal write phase:
Cluster in distributed cluster file system comprises: first node, Section Point, the 3rd node and the 4th node, and the responsible file extent of each node is:
First node:
[0?42535295865117307932921825928971026432]
Section Point:
(425352958651173079329218259289710264325070591730234615865843651857942052864]
The 3rd node:
(507059173023461586584365185794205286427605887595351923798765477786913079296]
The 4th node:
(127605887595351923798765477786913079296170141183460469231731687303715884105728]
The maximal value of 170141183460469231731687303715884105728 (being 2^127) MD5 theory wherein.Namely the filename of each file (key) calculates after MD5 value, can fall the responsible file extent of certain node.
Such as the filename (key) of file is that decimal value after 5, MD5 calculates is 39610130254249000831137752363030282240, belong to the responsible master file scope of above-mentioned first node.Therefore this file can be write in first node, is recorded in the backup transfer table of first node simultaneously.For reliability, also can in clockwise Section Point and the 3rd node, respectively deposit portion.
(2) Normal Backups node:
The backstage backup thread of first node, in scanning backup transfer table, it is responsible for the key (being that MD5 value is at the key of [042535295865117307932921825928971026432] scope) of file extent, when sweeping to 5 these key.First from local backup transfer table, read value, then write long-range to backup cluster.
(3) the node backup node of machine when abnormal of delaying
When the first node machine of extremely delaying.Section Point can receive or quantitative check to the delay message of machine of first node.And then adapter first node is responsible for the backup of the key of file extent.Now Section Point is except scanning (42,535,295,865,117,307,932,921,825,928,971,026,432,507,059,173,023,461,586 5843651857942052864] this is responsible for file extent, also can scan the key of first responsible file extent [0 42535295865117307932921825928971026432] scope).Complete the mainly backup transfer of responsible key of Section Point and first node.
According to a second aspect of the invention, corresponding with the method 100 of data backup according to an embodiment of the invention as above, the present invention also provides a kind of equipment 300 of data backup, is arranged in the first cluster of distributed cluster file system.
As shown in Figure 3, the equipment 300 of data backup according to an embodiment of the invention can mainly comprise scan module 310, backup module 330.The annexation instrument that should be appreciated that modules represented in Fig. 3 is example, and those skilled in the art can adopt other annexation completely, as long as modules also can be realized function of the present invention under such annexation.
In this manual, the function of modules can by with specialized hardware or the hardware that can combine with suitable software to carry out processing realize.Such hardware or specialized hardware can comprise special IC (ASIC), various other circuit, various processors etc.When being realized by processor, this function can be provided by single application specific processor, single shared processing device or a plurality of independently processor (wherein some may be shared).In addition, processor should not be understood to the hardware that special finger can executive software, but can impliedly comprise and be not limited to digital signal processor (DSP) hardware, be used for ROM (read-only memory) (ROM), random access memory (RAM) and the non-volatile memory apparatus of storing software.
In an embodiment of the present invention, scan module 310, for scanning backup transfer table responsible file extent in the first cluster;
Backup module 330, for the file writing for scanning, is written to the file writing in the second cluster of distributed cluster file system.
According to embodiments of the invention, the equipment 300 of described data backup can also comprise one or more optional module, to realize extra or additional function, yet these optional modules are not indispensable for realizing object of the present invention, the equipment 300 of data backup can, in the situation that there is no these optional modules, be realized object of the present invention completely according to an embodiment of the invention.Although these optional modules are not shown in Figure 3, the annexation between they and above-mentioned each module can easily be drawn according to following instruction by those skilled in the art.
Alternatively, in the bright embodiment of this law, described equipment 300 also comprises:
Memory module 350, for storing the file allocation writing to the respective nodes of described the first cluster;
Backup transfer table writing module 370, for synchronously writing backup transfer table by the file of said write and the write request information corresponding with the file of said write.
Alternatively, in the bright embodiment of this law, described equipment 300 also comprises:
Removing module 390, for deleting the file of said write and write request information thereof from described backup transfer table.
Alternatively, in the bright embodiment of this law, described backup transfer table comprises: the cryptographic hash of the filename of the file of said write and write request information, and wherein said write request information comprises: the file of said write is positioned at the address information of the first cluster and the/the second cluster;
Described memory module is further used for by the file allocation of said write, giving corresponding node in described the first cluster according to the cryptographic hash of the filename of said write;
Described scan module is further used for, according to the cryptographic hash of the filename writing, the responsible file extent of this node in described backup transfer table being scanned.
Alternatively, in the bright embodiment of this law, described equipment 300 also comprises: the machine testing module of delaying, for detection of whether there being the node of the machine of delaying in described the first cluster;
Described scan module be further used for delaying described in the scanning scope of the responsible file of the node of machine;
Described backup module is further used for the responsible file of the node of the described machine of delaying to be written in described the second cluster.
Alternatively, in the bright embodiment of this law, described in the machine testing module of delaying be further used for detecting each node in described the first cluster and regularly to some or all other nodes, send heartbeat message; When surpassing for first schedule time while not receiving the heartbeat message of Section Point, judge the described Section Point machine of delaying; By the delay next node of Section Point described in the information notification of machine of described Section Point.
Alternatively, in the bright embodiment of this law, described in the machine testing module of delaying be also further used for described Section Point to delay the message stores of machine at local internal memory; When not intercepting described Section Point and delay the message of machine, the internal memory of quantitative check this locality, detects the described Section Point machine of whether delaying.
According to a third aspect of the invention we, also provide a kind of distributed cluster file system, described distributed cluster file system comprises as above according to the equipment of the data backup of various embodiments of the present invention.
B9. according to the method described in any one in claim 2-8, wherein, the step that the access node in described the first cluster stores the file allocation writing to the respective nodes of described the first cluster is:
Access node in described the first cluster stores the file allocation writing to the respective nodes of described the first cluster, and is stored in the next node of described respective nodes or in lower two nodes.
B11. equipment according to claim 10, described equipment also comprises:
Memory module, for storing the file allocation writing to the respective nodes of described the first cluster;
Backup transfer table writing module, for synchronously writing backup transfer table by the file of said write and the write request information corresponding with the file of said write.
B12. equipment according to claim 11, described equipment also comprises:
Removing module, for deleting the file of said write and write request information thereof from described backup transfer table.
B13. equipment according to claim 11, described backup transfer table comprises: the cryptographic hash of the filename of the file of said write and write request information, and wherein said write request information comprises: the file of said write is positioned at the address information of the first cluster and the/the second cluster;
Described memory module is further used for by the file allocation of said write, giving corresponding node in described the first cluster according to the cryptographic hash of the filename of said write;
Described scan module is further used for, according to the cryptographic hash of the filename writing, the responsible file extent of this node in described backup transfer table being scanned.
B14. equipment according to claim 10, described equipment also comprises:
The machine testing module of delaying, for detection of whether there being the node of the machine of delaying in described the first cluster;
Described scan module be further used for delaying described in the scanning scope of the responsible file of the node of machine;
Described backup module is further used for the responsible file of the node of the described machine of delaying to be written in described the second cluster.
B15. equipment according to claim 14, wherein, described in the machine testing module of delaying be further used for detecting each node in described the first cluster and regularly to some or all other nodes, send heartbeat message; When surpassing for first schedule time while not receiving the heartbeat message of Section Point, judge the described Section Point machine of delaying; By the delay next node of Section Point described in the information notification of machine of described Section Point.
B16. equipment according to claim 14, wherein, described in the machine testing module of delaying be also further used for described Section Point to delay the message stores of machine at local internal memory; When not intercepting described Section Point and delay the message of machine, the internal memory of quantitative check this locality, detects the described Section Point machine of whether delaying.
The algorithm providing at this is intrinsic not relevant to any certain computer, virtual system or miscellaneous equipment with demonstration.Various general-purpose systems also can with based on using together with this teaching.According to description above, it is apparent constructing the desired structure of this type systematic.In addition, the present invention is not also for any certain programmed language.It should be understood that and can utilize various programming languages to realize content of the present invention described here, and the description of above language-specific being done is in order to disclose preferred forms of the present invention.
In the instructions that provided herein, a large amount of details have been described.Yet, can understand, embodiments of the invention can not put into practice in the situation that there is no these details.In some instances, be not shown specifically known method, structure and technology, so that not fuzzy understanding of this description.
Similarly, be to be understood that, in order to simplify the disclosure and to help to understand one or more in each inventive aspect, in the above in the description of exemplary embodiment of the present invention, each feature of the present invention is grouped together into single embodiment, figure or sometimes in its description.Yet, the method for the disclosure should be construed to the following intention of reflection: the present invention for required protection requires than the more feature of feature of clearly recording in each claim.Or rather, as reflected in claims below, inventive aspect is to be less than all features of disclosed single embodiment above.Therefore, claims of following embodiment are incorporated to this embodiment thus clearly, and wherein each claim itself is as independent embodiment of the present invention.
Those skilled in the art are appreciated that and can the module in the equipment in embodiment are adaptively changed and they are arranged in one or more equipment different from this embodiment.Module in embodiment or unit or assembly can be combined into a module or unit or assembly, and can put them into a plurality of submodules or subelement or sub-component in addition.At least some in such feature and/or process or unit are mutually repelling, and can adopt any combination to combine all processes or the unit of disclosed all features in this instructions (comprising claim, summary and the accompanying drawing followed) and disclosed any method like this or equipment.Unless clearly statement in addition, in this instructions (comprising claim, summary and the accompanying drawing followed) disclosed each feature can be by providing identical, be equal to or the alternative features of similar object replaces.
In addition, those skilled in the art can understand, although embodiment more described herein comprise some feature rather than further feature included in other embodiment, the combination of the feature of different embodiment means within scope of the present invention and forms different embodiment.For example, in the following claims, the one of any of embodiment required for protection can be used with array mode arbitrarily.
All parts embodiment of the present invention can realize with hardware, or realizes with the software module moved on one or more processor, or realizes with their combination.It will be understood by those of skill in the art that and can use in practice microprocessor or digital signal processor (DSP) to realize the some or all functions according to the some or all parts in the browser client of the embodiment of the present invention.The present invention for example can also be embodied as, for carrying out part or all equipment or device program (, computer program and computer program) of method as described herein.Realizing program of the present invention and can be stored on computer-readable medium like this, or can there is the form of one or more signal.Such signal can be downloaded and obtain from internet website, or provides on carrier signal, or provides with any other form.
It should be noted above-described embodiment the present invention will be described rather than limit the invention, and those skilled in the art can design alternative embodiment in the situation that do not depart from the scope of claims.In the claims, any reference symbol between bracket should be configured to limitations on claims.Word " comprises " not to be got rid of existence and is not listed as element or step in the claims.Being positioned at word " " before element or " one " does not get rid of and has a plurality of such elements.The present invention can be by means of including the hardware of some different elements and realizing by means of the computing machine of suitably programming.In having enumerated the unit claim of some devices, several in these devices can be to carry out imbody by same hardware branch.The use of word first, second and C grade does not represent any order.Can be title by these word explanations.
Claims (10)
1. a method for data backup, for distributed cluster file system, described distribution cluster file system at least comprises: the first cluster and the second cluster, described method comprises:
The backstage backup thread of each node in described the first cluster scans the responsible file extent of this node in backup transfer table;
The backstage backup thread of the node in described the first cluster is written to the file writing scanning in described the second cluster.
2. method according to claim 1, wherein, the backstage backup thread of each node in described the first cluster is to before the step that in backup transfer table, the responsible file extent of this node scans, and described method also comprises:
Access node in described the first cluster stores the file allocation writing to the respective nodes of described the first cluster;
The file of said write and the write request information corresponding with the file of said write are synchronously write to backup transfer table.
3. method according to claim 1, after the backstage backup thread of the node in described the first cluster is written to the step in described the second cluster by the file of said write, described method also comprises:
The file of said write and write request information thereof are deleted from described backup transfer table.
4. method according to claim 2, described backup transfer table comprises: the cryptographic hash of the filename of the file of said write and write request information, and wherein said write request information comprises: the file of said write is positioned at the address information of the first cluster and the/the second cluster;
The step that access node in described the first cluster stores the file allocation writing to the respective nodes of described the first cluster comprises:
Access node in described the first cluster is given corresponding node in described the first cluster according to the cryptographic hash of the filename of said write by the file allocation of said write;
The backstage backup thread of each node in described the first cluster comprises the step that in described backup transfer table, the responsible file extent of this node scans:
The backstage backup thread of each node in described the first cluster, according to the cryptographic hash of the filename writing, scans the responsible file extent of this node in described backup transfer table.
5. method according to claim 1, the backstage backup thread of each node in described the first cluster is to before the step that in described backup transfer table, the responsible file extent of this node scans, and described method also comprises:
Detect the node that whether has the machine of delaying in described the first cluster;
When the node of machine is delayed in existence, the backstage backup thread of each node in described the first cluster comprises the step that in described backup transfer table, the responsible file extent of this node scans:
The scope of the file that the node of machine of delaying described in the next node scanning of the node of the described machine of delaying is responsible;
The step that the backstage backup thread of the node in described the first cluster is written to the file of said write in described the second cluster comprises:
The backstage backup thread of the next node of the node of the described machine of delaying is written to the responsible file of the node of the described machine of delaying in described the second cluster.
6. whether method according to claim 5, wherein, exist the step of the node of the machine of delaying to comprise in described the first cluster of described detection:
Detect each node in described the first cluster and regularly to some or all other nodes, send heartbeat message;
When first node surpassed for first schedule time while not receiving the heartbeat message of Section Point, judge the described Section Point machine of delaying;
Described first node is by the delay next node of Section Point described in the information notification of machine of described Section Point.
7. whether method according to claim 5, wherein, exist the step of the node of the machine of delaying also to comprise in described the first cluster of described detection:
The next node of the Section Point in described the first cluster is delayed the message stores of machine at local internal memory by described Section Point;
When the next node of described Section Point is not intercepted described Section Point and delayed the message of machine by background thread, the internal memory of next node quantitative check this locality of described Section Point, detects the described Section Point machine of whether delaying.
8. according to the method described in claim 6 or 7, wherein, the next node of described second node adopts an other background thread, and second responsible scope of node described in described backup transfer table scanned.
9. an equipment for data backup, is arranged in the first cluster of distributed cluster file system, and described equipment comprises:
Scan module, for scanning the responsible file extent of backup transfer table;
Backup module, for being written to the file writing scanning the second cluster of described distributed cluster file system.
10. a distributed cluster file system, wherein, comprises the equipment of data backup as claimed in claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310476405.8A CN103577546B (en) | 2013-10-12 | 2013-10-12 | A kind of method of data backup, equipment and distributed cluster file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310476405.8A CN103577546B (en) | 2013-10-12 | 2013-10-12 | A kind of method of data backup, equipment and distributed cluster file system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103577546A true CN103577546A (en) | 2014-02-12 |
| CN103577546B CN103577546B (en) | 2017-06-09 |
Family
ID=50049322
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310476405.8A Active CN103577546B (en) | 2013-10-12 | 2013-10-12 | A kind of method of data backup, equipment and distributed cluster file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103577546B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104346479A (en) * | 2014-11-26 | 2015-02-11 | 北京奇虎科技有限公司 | Database synchronization method and database synchronization device |
| CN105516097A (en) * | 2015-11-30 | 2016-04-20 | 上海爱数信息技术股份有限公司 | Mixed-architecture message system and message transmission method based on Thrift data format |
| CN105608167A (en) * | 2015-12-18 | 2016-05-25 | 金蝶软件（中国）有限公司 | KDFS system-based file backup method and system |
| CN106250273A (en) * | 2016-08-01 | 2016-12-21 | 浪潮软件集团有限公司 | Method for realizing data remote disaster recovery based on Hadoop file system |
| CN108063782A (en) * | 2016-11-08 | 2018-05-22 | 北京国双科技有限公司 | Node is delayed machine adapting method and device, node group system |
| CN108123976A (en) * | 2016-11-30 | 2018-06-05 | 阿里巴巴集团控股有限公司 | Data back up method, apparatus and system between cluster |
| CN108132858A (en) * | 2017-12-22 | 2018-06-08 | 周川 | A kind of disaster-tolerant backup method |
| CN109561125A (en) * | 2017-09-27 | 2019-04-02 | 阿里巴巴集团控股有限公司 | The method and device of replicate data in a kind of data transmission method, distributed system |
| CN109684128A (en) * | 2018-11-16 | 2019-04-26 | 深圳证券交易所 | Cluster overall failure restoration methods, server and the storage medium of message-oriented middleware |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7941404B2 (en) * | 2006-03-08 | 2011-05-10 | International Business Machines Corporation | Coordinated federated backup of a distributed application environment |
| CN102780763A (en) * | 2012-06-29 | 2012-11-14 | 华中科技大学 | Distributed home subscriber server (HSS) data storage method and distributed HSS data extraction method |
| CN102890716A (en) * | 2012-09-29 | 2013-01-23 | 南京中兴新软件有限责任公司 | Distributed file system and data backup method thereof |
| US20130110779A1 (en) * | 2010-05-03 | 2013-05-02 | Panzura, Inc. | Archiving data for a distributed filesystem |
| CN103176860A (en) * | 2011-12-21 | 2013-06-26 | 腾讯科技（深圳）有限公司 | Data backup method and data backup system |
| CN103209091A (en) * | 2013-01-18 | 2013-07-17 | 中兴通讯股份有限公司 | Hot backup method and system for clustering system |
-
2013
- 2013-10-12 CN CN201310476405.8A patent/CN103577546B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7941404B2 (en) * | 2006-03-08 | 2011-05-10 | International Business Machines Corporation | Coordinated federated backup of a distributed application environment |
| US20130110779A1 (en) * | 2010-05-03 | 2013-05-02 | Panzura, Inc. | Archiving data for a distributed filesystem |
| CN103176860A (en) * | 2011-12-21 | 2013-06-26 | 腾讯科技（深圳）有限公司 | Data backup method and data backup system |
| CN102780763A (en) * | 2012-06-29 | 2012-11-14 | 华中科技大学 | Distributed home subscriber server (HSS) data storage method and distributed HSS data extraction method |
| CN102890716A (en) * | 2012-09-29 | 2013-01-23 | 南京中兴新软件有限责任公司 | Distributed file system and data backup method thereof |
| CN103209091A (en) * | 2013-01-18 | 2013-07-17 | 中兴通讯股份有限公司 | Hot backup method and system for clustering system |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104346479A (en) * | 2014-11-26 | 2015-02-11 | 北京奇虎科技有限公司 | Database synchronization method and database synchronization device |
| CN105516097B (en) * | 2015-11-30 | 2019-02-19 | 上海爱数信息技术股份有限公司 | Mixed architecture message system and method for message transmission based on Thrift data format |
| CN105516097A (en) * | 2015-11-30 | 2016-04-20 | 上海爱数信息技术股份有限公司 | Mixed-architecture message system and message transmission method based on Thrift data format |
| CN105608167A (en) * | 2015-12-18 | 2016-05-25 | 金蝶软件（中国）有限公司 | KDFS system-based file backup method and system |
| CN105608167B (en) * | 2015-12-18 | 2019-10-08 | 金蝶软件（中国）有限公司 | A kind of file backup method and system based on KDFS system |
| CN106250273B (en) * | 2016-08-01 | 2019-04-09 | 浪潮软件集团有限公司 | Method for realizing data remote disaster recovery based on Hadoop file system |
| CN106250273A (en) * | 2016-08-01 | 2016-12-21 | 浪潮软件集团有限公司 | Method for realizing data remote disaster recovery based on Hadoop file system |
| CN108063782A (en) * | 2016-11-08 | 2018-05-22 | 北京国双科技有限公司 | Node is delayed machine adapting method and device, node group system |
| CN108123976A (en) * | 2016-11-30 | 2018-06-05 | 阿里巴巴集团控股有限公司 | Data back up method, apparatus and system between cluster |
| CN108123976B (en) * | 2016-11-30 | 2020-11-20 | 阿里巴巴集团控股有限公司 | Method, device and system for data backup among clusters |
| CN109561125A (en) * | 2017-09-27 | 2019-04-02 | 阿里巴巴集团控股有限公司 | The method and device of replicate data in a kind of data transmission method, distributed system |
| CN108132858A (en) * | 2017-12-22 | 2018-06-08 | 周川 | A kind of disaster-tolerant backup method |
| CN109684128A (en) * | 2018-11-16 | 2019-04-26 | 深圳证券交易所 | Cluster overall failure restoration methods, server and the storage medium of message-oriented middleware |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103577546B (en) | 2017-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103577546A (en) | Method and equipment for data backup, and distributed cluster file system | |
| US9715507B2 (en) | Techniques for reconciling metadata and data in a cloud storage system without service interruption | |
| CN102043686B (en) | Disaster tolerance method, backup server and system of memory database | |
| CN105843702B (en) | A kind of method and device for data backup | |
| CN103744628B (en) | SSTable file storage method and device | |
| CN105550229A (en) | Method and device for repairing data of distributed storage system | |
| WO2016149006A1 (en) | Virtual partitions in virtual databases | |
| US11176110B2 (en) | Data updating method and device for a distributed database system | |
| US11385830B2 (en) | Data storage method, apparatus and system, and server, control node and medium | |
| US9400767B2 (en) | Subgraph-based distributed graph processing | |
| CN104516966A (en) | High-availability solving method and device of database cluster | |
| CN103761165A (en) | Log backup method and log backup device | |
| CN103744745A (en) | Data storage detection method and device and distributed storage system | |
| CN104881466A (en) | Method and device for processing data fragments and deleting garbage files | |
| CN112654978A (en) | Method, equipment and system for checking data consistency in distributed heterogeneous storage system in real time | |
| CN104978336A (en) | Unstructured data storage system based on Hadoop distributed computing platform | |
| CN111309245B (en) | Hierarchical storage writing method and device, reading method and device and system | |
| CN106776720A (en) | A kind of document handling method and device | |
| CN103716384A (en) | Method and device for realizing cloud storage data synchronization in cross-data-center manner | |
| CN111177257A (en) | Data storage and access method, device and equipment of block chain | |
| CN111125065A (en) | Visual data synchronization method, system, terminal and computer readable storage medium | |
| CN110213326A (en) | Expansion method, system, equipment and the storage medium of metadata node cluster | |
| CN104462342B (en) | database snapshot synchronization processing method and device | |
| US20200192888A1 (en) | Distributed trust data storage system | |
| CN111240892A (en) | Data backup method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220725 Address after: Room 801, 8th floor, No. 104, floors 1-19, building 2, yard 6, Jiuxianqiao Road, Chaoyang District, Beijing 100015 Patentee after: BEIJING QIHOO TECHNOLOGY Co.,Ltd. Address before: 100088 room 112, block D, 28 new street, new street, Xicheng District, Beijing (Desheng Park) Patentee before: BEIJING QIHOO TECHNOLOGY Co.,Ltd. Patentee before: Qizhi software (Beijing) Co.,Ltd. |
|
| TR01 | Transfer of patent right |