CN105677249A - Data block partitioning method, device and system - Google Patents
Data block partitioning method, device and system Download PDFInfo
- Publication number
- CN105677249A CN105677249A CN201610003072.0A CN201610003072A CN105677249A CN 105677249 A CN105677249 A CN 105677249A CN 201610003072 A CN201610003072 A CN 201610003072A CN 105677249 A CN105677249 A CN 105677249A
- Authority
- CN
- China
- Prior art keywords
- data block
- memory node
- group
- disk
- integer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data block partitioning method, device and system. The method comprises the specific implementation steps that M storage nodes are built, wherein M is a positive integer not larger than N+1, N is any a prime number, and each storage node comprises N physical disks; each physical disk contained by the corresponding storage node is partitioned into multiple data blocks, and the number of the data blocks contained by each physical disk is N; the data blocks are grouped, the same group of data blocks belong to a same erasure group, the data blocks in each group are made to be distributed on the different storage nodes, and the other data blocks in the group corresponding to the data blocks contained by any physical disks in any storage nodes are all distributed on the different physical disks. According to the method, the storage efficiency and the storage performance of a storage system are improved, and particularly, improvement of the storage efficiency and performance on a storage system applying an erasure code mechanism is more obvious in the data reconstruction process.
Description
Technical field
The application relates to technical field of memory, particularly relates to the division methods of a kind of data block, Apparatus and system.
Background technology
In video monitoring with safety-security area, it is necessary to constantly video stream data or image data are carried out storage backup, it is contemplated that the safety of data, current storage system has been generally adopted redundant memory technology, such as correcting and eleting codes technology. After introducing correcting and eleting codes technology within the storage system, performance issue when data reconstruction or multichannel read-write will be related to. When carrying out data reconstruction or multichannel read-write, if relating to multiple data block concurrent reading and concurrent writings of same disk, owing to the speed of disk I/O is far smaller than the CPU speed calculated, therefore, performance consumption is huge, can produce to significantly affect to normal business, thus have impact on the memory property of whole storage system.
In prior art, generally can adopt crush algorithm, solve the problems referred to above. Crush algorithm is a kind of pseudo-random data Distribution Algorithm, and it can effectively be distributed the copy of object in the storage cluster of hierarchical structure, it is ensured that load balancing, and adapts to the increase and decrease of storage device. But this algorithm is based on hierarchy, more complicated, algorithm and the too much hierarchic design of its complexity can cause extra performance consumption, thus reducing storage efficiency and the memory property of storage system.
Summary of the invention
The application provides the division methods of a kind of data block, Apparatus and system, the problem low with memory property to solve the storage efficiency storing system in prior art.
First aspect according to the embodiment of the present application, it is provided that the division methods of a kind of data block, described method is applied in storage system, applies correcting and eleting codes mechanism in described storage system, and described method includes:
Building M memory node, wherein, M is the positive integer being not more than N+1, and N is any prime number, and the physical disk number that each described memory node comprises is N;
Each physical disk that each described memory node comprises is marked off multiple data block, and wherein, the number of the data block that each described physical disk comprises is N;
Described data block is grouped, the data block of same group belongs to same entangling and deletes group, the data block in every group is made to be distributed on different memory nodes, and, for any physical disk in any described memory node, other data block in the group that the data block that comprised is corresponding, is distributed on different physical disks.
Second aspect according to the embodiment of the present application, it is provided that the division device of a kind of data block, described device is applied in storage system, applies correcting and eleting codes mechanism in described storage system, and described device includes:
Construction unit, is used for building M memory node, and wherein, M is the positive integer being not more than N+1, and N is any prime number, and the physical disk number that each described memory node comprises is N;
Division unit, for each physical disk that each described memory node comprises is marked off multiple data block, wherein, the number of the data block that each described physical disk comprises is N;
Grouped element, for described data block is grouped, the data block of same group belongs to same entangling and deletes group, the data block in every group is made to be distributed on different memory nodes, and, for any physical disk in any described memory node, other data block in the group that the data block that comprises is corresponding, it is distributed on different physical disks.
The third aspect according to the embodiment of the present application, it is provided that a kind of storage system, applies correcting and eleting codes mechanism in described storage system, described storage system includes M memory node, and wherein, M is the positive integer being not more than N+1, N is any prime number, and each described memory node comprises the physical disk that number is N; Each physical disk is divided into multiple data block, and wherein, the number of the data block that each described physical disk comprises is N; Described data block is divided into many groups, the data block of same group belongs to same entangling and deletes group, often the data block in group is distributed on different memory nodes, and, for any physical disk in any described memory node, other data block in the group that the data block that comprised is corresponding, is distributed on different physical disks.
The division methods of the data block that above-described embodiment of the application provides, Apparatus and system, by constructing M memory node, and meeting M is the positive integer being not more than N+1, N is any prime number, the each physical disk comprised by each memory node marks off multiple data block, above-mentioned data block is grouped, the data block of same group belongs to same entangling and deletes group, the data block in every group is made to be distributed on different memory nodes, and, for any physical disk in any memory node, other data block in the group that the data block that comprised is corresponding, it is distributed on different physical disks. thus taking full advantage of the IO performance of many physical disks in storage cluster, ensure in data access process, for a certain piece of physical disk, access its corresponding entangling of data block comprised and delete group, arbitrarily other node is the parallel work-flow of different disk, at utmost ensure the concurrency of data access, thus improve storage efficiency and the memory property of storage system, particularly with the storage system applying correcting and eleting codes mechanism, in data reconstruction processes, the lifting of storage efficiency and performance becomes apparent from.
Accompanying drawing explanation
Accompanying drawing herein is merged in description and constitutes the part of this specification, it is shown that meet embodiments herein, and for explaining the principle of the application together with description.
The flow chart of the division methods of a kind of data block that Fig. 1 provides for the embodiment of the present application;
The flow chart of the division methods of the another kind of data block that Fig. 2 provides for the embodiment of the present application;
Fig. 3 is the division schematic diagram of a kind of data block storing system that the embodiment of the present application provides;
Fig. 4 is the division schematic diagram of the data block of the another kind storage system that the embodiment of the present application provides;
Fig. 5 is the hardware environment Organization Chart dividing plant running of the application data block;
Fig. 6 is the block diagram dividing device of a kind of data block that the embodiment of the present application provides;
Fig. 7 is the block diagram dividing device of the another kind of data block that the embodiment of the present application provides.
Detailed description of the invention
Here in detail exemplary embodiment being illustrated, its example representation is in the accompanying drawings. When as explained below relates to accompanying drawing, unless otherwise indicated, the same numbers in different accompanying drawings represents same or analogous key element. Embodiment described in following exemplary embodiment does not represent all embodiments consistent with the application. On the contrary, they only with in appended claims describe in detail, the application some in the example of consistent apparatus and method.
It is only merely for the purpose describing specific embodiment at term used in this application, and is not intended to be limiting the application. " one ", " described " and " being somebody's turn to do " of the singulative used in the application and appended claims is also intended to include most form, unless context clearly shows that other implications. It is also understood that term "and/or" used herein refers to and comprises any or all of one or more project of listing being associated and be likely to combination.
Although should be appreciated that and be likely to adopt term first, second, third, etc. to describe various information in the application, but these information should not necessarily be limited by these terms. These terms are only used for being distinguished from each other out same type of information. Such as, when without departing from the application scope, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depend on linguistic context, word as used in this " if " can be construed to " ... time " or " when ... " or " in response to determining ".
For problems of the prior art, proposing the division methods of a kind of data block in the embodiment of the present application, the method is applied in storage system, applies correcting and eleting codes mechanism, as it is shown in figure 1, the method specifically may comprise steps of in this storage system:
In a step 101, building M memory node, wherein, M is the positive integer being not more than N+1, and N is any prime number, and the physical disk number that each memory node comprises is N.
In general, memory node can be an independent server, is made up of in conjunction with disk array (disk group) controller. In the present embodiment, for the correcting and eleting codes of n+m pattern, memory node refers in particular to a stored position of data in the correcting and eleting codes of this n+m pattern, namely needs n+m memory node altogether under the pattern of n+m correcting and eleting codes.
In the present embodiment, it is possible, firstly, to choose multiple physical disk to build M memory node, the physical disk number that each memory node comprises is N, and wherein, M is the positive integer being not more than N+1, and N is any prime number, for instance, N can be 3, or 5, or 7 etc. The concrete value of N depends on the storage needs of reality, and the concrete value aspect of N is not limited by the application.
In a step 102, each physical disk that each memory node comprises is marked off multiple data block.
In the present embodiment, after constructing M memory node, each memory node comprises multiple physical disk, and each physical disk carries out dividing the process of data block. The each physical disk comprised by each memory node marks off multiple data block, and, the number of the data block that each physical disk comprises is N.
Such as, if choosing N=3, then M is maximum can be 4. That is, if the physical disk number that each memory node comprises is 3, then the number of the memory node that can create can be 4, and now, entangling the pattern deleting group can be 2+2 pattern, it is also possible to be 3+1 pattern etc. The number of memory node can also be 3, and now, entangling the pattern deleting group can be 2+1 pattern etc. The number of the data block that each physical disk comprises can be 3.
Again such as, if choosing N=5, then M is maximum can be 6. That is, if the physical disk number that each memory node comprises is 5, then the number of the memory node that can create can be 6, and now, entangling the pattern deleting group can be 3+3 pattern, it is also possible to be 4+2 pattern etc. The number of memory node can also be 5, and now, entangling the pattern deleting group can be 3+2 pattern, it is also possible to be 4+1 pattern etc. The number of the data block that each physical disk comprises can be 5.
In step 103, above-mentioned data block being grouped, the data block of same group belongs to same entangling and deletes group.
In general, correcting and eleting codes (ErasureCode) technology is a kind of input tolerant for Chinese technology, it is the problem solving the loss in the transmission of part data in the communications industry the earliest, its ultimate principle is the signal subsection transmission, add the contact that certain verification allows each intersegmental generation certain again, even if losing part signal in transmitting procedure, receiving terminal still can be calculated complete information by algorithm. From the grown form of correcting and eleting codes, it is that n data add m verification (n+m data block) and constitute, and wherein, the value of n and m can both according to certain rule settings, it is possible to this group n+m data block is entangled as one and deletes group.
In the present embodiment, one is entangled the data block (n+m data block) deleting group and is distributed in n+m different memory node, when entangling the loss of data deleting some in group (number is less than or equal to m) data block for one, it is possible to entangle according to this and delete the data that in group, the data convert of other data block has been lost. Further, data reconstruction and be written and read operate time, it is necessary to this is entangled delete in group data block simultaneously be written and read operation.
In the present embodiment, it is necessary to the data block marked off is grouped, the data block of same group belongs to same entangling and deletes group. Often the data block in group is distributed on different memory nodes, and, for any physical disk in any memory node, other data block in the group that the data block that comprises is corresponding, it is distributed on different physical disks. Such as, for physical disk 1, comprise tri-data blocks of a1, b1, c1, be belonging respectively to A, B, C tri-and entangle and delete group, A, B, C tri-entangle and delete group and respectively further comprise a2, a3, a4, b2, b3, b4, c2, c3, c4 data block, wherein, a2, a3, a4, b2, b3, b4, these nine data blocks of c2, c3, c4 are distributed on different physical disks, are absent from any number of (two or more) data block being distributed on Same Physical disk.
The division methods of the data block that above-described embodiment of the application provides, construct M memory node, and meeting M is the positive integer being not more than N+1, N is any prime number, the each physical disk comprised by each memory node marks off multiple data block, above-mentioned data block is grouped, the data block of same group belongs to same entangling and deletes group, the data block in every group is made to be distributed on different memory nodes, and, for any physical disk in any memory node, other data block in the group that the data block that comprised is corresponding, it is distributed on different physical disks. thus taking full advantage of the IO performance of many physical disks in storage cluster, ensure in data access process, for a certain piece of physical disk, access its corresponding entangling of data block comprised and delete group, arbitrarily other node is the parallel work-flow of different disk, at utmost ensure the concurrency of data access, thus improve storage efficiency and the memory property of storage system, particularly with the storage system applying correcting and eleting codes mechanism, in data reconstruction processes, the lifting of storage efficiency and performance becomes apparent from.
Referring to Fig. 2, being the flow chart of the division methods of the another kind of data block that the embodiment of the present application provides, the method specifically may comprise steps of:
In step 201, building N+1 memory node, wherein, N is any prime number, and the physical disk number that each memory node comprises is N.
In step 202., each physical disk that each memory node comprises is marked off multiple data block.
In step 203, to each memory node serial number.
In step 204, each memory node is divided into N number of virtual disk, makes each virtual disk include N number of data block, and the data block that each virtual disk includes is distributed on different physical disks.
In step 205, to each virtual disk serial number in each memory node.
In step 206, to each physical disk serial number in each memory node.
In step 207, above-mentioned data block is grouped, uses Mk[i, j] represents data block so that the arbitrarily data block M of k ≠ 1kThe data block M of [i, j] and k=11[j-(k-2) (i-1)+nN, i] belongs to one group, and belongs to same entangling and delete group, wherein, k represents the numbering of the memory node belonging to data block, k ∈ [1, N] and be integer, i represents the numbering of the virtual disk belonging to data block, i ∈ [1, N] and be integer, j represents the numbering of the physical disk belonging to data block, j ∈ [1, N] and be integer, n is for making the integer of j-(k-2) (i-1)+nN ∈ [1, N].
In the present embodiment, first, each memory node is carried out serial number, such as, assume there are 3 memory nodes, then these 3 memory nodes can be numbered node 1, node 2, node 3, it is possible to understand that, can also having other numbering, the concrete numbering aspect of memory node is not limited by the application. Further, above-mentioned numbering is only a kind of representation of memory node, and the concrete numeral of numbering is unrelated with the content of the character of memory node and storage.
Then, each memory node is divided into N number of virtual disk, makes each virtual disk include N number of data block, and the data block that each virtual disk includes is distributed on different physical disks. Wherein, virtual disk is not an in esse disk, and the data block on a virtual disk can be respectively stored on different physical disks. Then, each virtual disk is carried out serial number, for instance, assume there are 3 virtual disks, then these 3 virtual disks can be numbered 1,2,3 respectively, it is possible to understand that, can also having other numbering, the concrete numbering aspect of virtual disk is not limited by the application.Further, above-mentioned numbering is only a kind of representation of virtual disk, and the concrete numeral of numbering is unrelated with the data content of the character of virtual disk and storage.
Then, each physical disk is carried out serial number, for instance, assume there are 3 physical disks, then these 3 physical disks can be numbered 1,2,3 respectively, it is possible to understand that, can also having other numbering, the concrete numbering aspect of physical disk is not limited by the application. Further, above-mentioned numbering is only a kind of representation of physical disk, and the concrete numeral of numbering is unrelated with the data content of the character of physical disk and storage.
Finally, M is usedk[i, j] represents data block, and data block is grouped so that the arbitrarily data block M of k ≠ 1kThe data block M of [i, j] and k=11[j-(k-2) (i-1)+nN, i] belongs to one group, and belongs to same entangling and delete group, and wherein, n is for making the integer of j-(k-2) (i-1)+nN ∈ [1, N].
Such as, as N=3, it is possible to build 4 memory nodes, as it is shown on figure 3, these 4 memory nodes are numbered, respectively node 1, node 2, node 3 and node 4. Each memory node is divided into 3 virtual disks, and a virtual disk includes 3 data blocks. Each virtual disk and physical disk are numbered, can by 1,2,3 numberings representing correspondence. In figure, a line of each memory node represents a virtual disk, and string represents a physical disk. Each sub-box in figure represents a data block, and the numeral in sub-box can represent that entangling belonging to this data block deletes the group number of group, and in corresponding grid, the identical data block of numeral is divided into one group of data block. After data block is grouped so that the arbitrary data block M of any one node these 3 nodes from node 2 to node 4k[i, j], all with the data block M of node 11[j-(k-2) (i-1)+nN, i] belongs to one group, and belongs to same entangling and delete group. As, M3[2,1] and M1[3,2] belong to one to entangle and delete group. M4[2,1] and M1[2,2] belong to one to entangle and delete group. M2[3,3]、M3[3,2]、M4[3,1] and M1[3,3] belong to one and entangle and delete group, etc. It can be seen that the data block in the often group of the present embodiment is all distributed on different memory nodes, and, for any physical disk in any memory node, other data block in the group that the data block that comprises is corresponding, it is distributed on different physical disks.
Again such as, as N=5, it is possible to build 6 memory nodes, as shown in Figure 4, these 6 memory nodes are numbered, respectively node 1, node 2, node 3, node 4, node 5 and node 6. Each memory node is divided into 5 virtual disks, and a virtual disk includes 5 data blocks. Each virtual disk and physical disk are numbered, can by 1,2,3,4,5 numberings representing correspondence. In figure, a line of each memory node represents a virtual disk, and string represents a physical disk. Each sub-box in figure represents a data block, and the numeral in sub-box can represent that entangling belonging to this data block deletes the group number of group, and in corresponding grid, the identical data block of numeral is divided into one group of data block. After data block is grouped so that the arbitrary data block M of any one node these 5 nodes from node 2 to node 6k[i, j], all with the data block M of node 11[j-(k-2) (i-1)+nN, i] belongs to one group, and belongs to same entangling and delete group. As, M2[5,3] and M1[3,5] belong to one to entangle and delete group.M6[2,3] and M1[4,2] belong to one to entangle and delete group. M2[2,1]、M3[2,2]、M4[2,2]、M5[2,4]、M6[2,5] and M1[1,2] belongs to one and entangles and delete group, etc. It can be seen that the data block in the often group of the present embodiment is all distributed on different memory nodes, and, for any physical disk in any memory node, other data block in the group that the data block that comprises is corresponding, it is distributed on different physical disks.
The division methods of the data block that above-described embodiment of the application provides, by to each memory node serial number, each memory node is divided into N number of virtual disk, each virtual disk is made to include N number of data block, and the data block that each virtual disk includes is distributed on different physical disks, to each virtual disk serial number, to each described physical disk serial number, use Mk[i, j] represents data block so that the arbitrarily data block M of k ≠ 1kThe data block M of [i, j] and k=11[j-(k-2) (i-1)+nN, i] belongs to one and entangles and delete group. Thus taking full advantage of the IO performance of many physical disks in storage cluster, ensure in data access process, for a certain piece of physical disk, access its corresponding entangling of data block comprised and delete group, arbitrarily other node is the parallel work-flow of different disk, at utmost ensure the concurrency of data access, thus improve storage efficiency and the memory property of storage system, particularly with the storage system applying correcting and eleting codes back mechanism, in data reconstruction processes, the lifting of storage efficiency and performance becomes apparent from.
In some optional embodiments, it is also possible to building less than N+1 memory node, wherein, N is any prime number, for instance, when N is equal to 3, it is also possible to build 3 memory nodes. Again such as, when N is equal to 5, it is also possible to build 5 memory nodes, or 4 memory nodes etc. can also be built. When the memory node built is less than N+1, still can adopt the model split data block described in above-described embodiment and data block is grouped. Can also mode described in above-described embodiment is virtual constructs N+1 memory node, then optional M node from N+1 memory node of this virtual structure, go out this M memory node according to corresponding building mode actual implementation.
Although it should be noted that, describe the operation of the application method in the accompanying drawings with particular order, but, this does not require that or implies and must operate to perform these according to this particular order, or having to carry out all shown operation could realize desired result. On the contrary, the step described in flow chart can change execution sequence. Additionally or alternatively, it is convenient to omit some step, multiple steps are merged into a step and performs, and/or a step is decomposed into the execution of multiple step.
Referring to Fig. 5, for the hardware environment Organization Chart dividing plant running of data block in a kind of embodiment of the application:
The embodiment dividing device of the application data block can be realized by software, it is also possible to is realized by the mode of hardware or software and hardware combining. Implemented in software for example, as the device on a logical meaning, it is that computer program instructions corresponding in nonvolatile memory is read to run in internal memory and formed by the CPU by its place equipment. Say from hardware view; as shown in Figure 5; a kind of hardware structure diagram dividing device place equipment for the application data block; except the CPU shown in Fig. 5, internal memory, network interface and nonvolatile memory; in embodiment, the equipment at device place generally can also include other hardware, and this Fig. 5 is no longer illustrated one by one.
Corresponding with the division methods embodiment of aforementioned data block, present invention also provides the division device of data block and the embodiment of the storage system applied thereof.
As shown in Figure 6, Fig. 6 is the division device block diagram of the application a kind of data block according to an exemplary embodiment, and this device includes: construction unit 601, division unit 602, grouped element 603.
Wherein, construction unit 601, it is used for building M memory node, wherein, M is the positive integer being not more than N+1, and N is any prime number, and the physical disk number that each memory node comprises is N.
Division unit 602, marks off multiple data block for each physical disk comprised by each memory node, and wherein, the number of the data block that each physical disk comprises is N.
Grouped element 603, for above-mentioned data block is grouped, the data block of same group belongs to same entangling and deletes group, the data block in every group is made to be distributed on different memory nodes, and, for any physical disk in any memory node, other data block in the group that the data block that comprises is corresponding, it is distributed on different physical disks.
As shown in Figure 7, Fig. 7 is the division device block diagram of the application another kind of data block according to an exemplary embodiment, this embodiment is on the basis of aforementioned embodiment illustrated in fig. 6, when the memory node number built is N+1, grouped element 603 may include that the first numbering subelement 701, and virtual disk divides subelement 702, the second numbering subelement 703,3rd numbering subelement 704, data block packet subelement 705.
Wherein, the first numbering subelement 701, for each memory node serial number.
Virtual disk divides subelement 702, for each memory node is divided into N number of virtual disk, makes each virtual disk include N number of data block, and the data block that each virtual disk includes is distributed on different physical disks.
Second numbering subelement 703, for each virtual disk serial number in each memory node.
3rd numbering subelement 704, for each physical disk serial number in each memory node.
Data block packet subelement 705, is used for Mk[i, j] represents data block, and above-mentioned data block is grouped so that the arbitrarily data block M of k ≠ 1kThe data block M of [i, j] and k=11[j-(k-2) (i-1)+nN, i] belong to one group, and belong to same entangling and delete group, wherein, k represents the numbering of the memory node belonging to above-mentioned data block, k ∈ [1, N] and be integer, i represents the numbering of the virtual disk belonging to above-mentioned data block, i ∈ [1, N] and be integer, j represents the numbering of the physical disk belonging to above-mentioned data block, j ∈ [1, N] and be integer, n is for making the integer of j-(k-2) (i-1)+nN ∈ [1, N].
For device embodiment, owing to it corresponds essentially to embodiment of the method, so relevant part illustrates referring to the part of embodiment of the method. Device embodiment described above is merely schematic, the wherein said modular unit illustrated as separating component can be or may not be physically separate, the parts shown as modular unit can be or may not be physical module unit, namely may be located at a place, or can also be distributed on multiple NE. Some or all of module therein can be selected according to the actual needs to realize the purpose of the application scheme. Those of ordinary skill in the art, when not paying creative work, are namely appreciated that and implement.
Accordingly, the application also provides for a kind of storage system, applies the back mechanism of correcting and eleting codes in this storage system, this storage system includes M memory node, and wherein, M is the positive integer being not more than N+1, N is any prime number, and each memory node comprises the physical disk that number is N.Each physical disk is divided into multiple data block, and wherein, the number of the data block that each physical disk comprises is N. Above-mentioned data block is divided into many groups, the data block of same group belongs to same entangling and deletes group, often the data block in group is distributed on different memory nodes, and, for any physical disk in any memory node, other data block in the group that the data block that comprised is corresponding, is distributed on different physical disks.
In further optional embodiment, storage system includes N+1 memory node, each memory node is divided into N number of virtual disk, and each virtual disk includes N number of data block, and the data block that each virtual disk includes is distributed on different physical disks. Above-mentioned data block is divided into many groups, uses Mk[i, j] represents data block, the data block M of any k ≠ 1kThe data block M of [i, j] and k=11[j-(k-2) (i-1)+nN, i] belong to one group, and belong to same entangling and delete group, n is for making j-(k-2) (i-1)+nN ∈ [1, N] integer, wherein, k represents the numbering of the memory node belonging to above-mentioned data block, k ∈ [1, N] and be integer, i represents the numbering of the virtual disk belonging to above-mentioned data block, i ∈ [1, N] and be integer, j represents the numbering of the physical disk belonging to above-mentioned data block, j ∈ [1, N] and be integer.
Those skilled in the art, after considering description and putting into practice invention disclosed herein, will readily occur to other embodiment of the application. The application is intended to any modification of the application, purposes or adaptations, and these modification, purposes or adaptations are followed the general principle of the application and include the undocumented known general knowledge in the art of the application or conventional techniques means. Description and embodiments is considered only as exemplary, and the true scope of the application and spirit are pointed out by claim below.
It should be appreciated that the application is not limited to precision architecture described above and illustrated in the accompanying drawings, and various amendment and change can carried out without departing from the scope. Scope of the present application is only limited by appended claim.
Claims (6)
1. the division methods of a data block, it is characterised in that described method is applied in storage system, applies correcting and eleting codes mechanism in described storage system, described method includes:
Building M memory node, wherein, M is the positive integer being not more than N+1, and N is any prime number, and the physical disk number that each described memory node comprises is N;
Each physical disk that each described memory node comprises is marked off multiple data block, and wherein, the number of the data block that each described physical disk comprises is N;
Described data block is grouped, the data block of same group belongs to same entangling and deletes group, the data block in every group is made to be distributed on different memory nodes, and, for any physical disk in any described memory node, other data block in the group that the data block that comprised is corresponding, is distributed on different physical disks.
2. method according to claim 1, it is characterised in that when the memory node number built is N+1, described described data block is grouped, including:
To each described memory node serial number;
Each described memory node is divided into N number of virtual disk, makes each described virtual disk include N number of data block, and the data block that each described virtual disk includes is distributed on different physical disks;
To each described virtual disk serial number in each described memory node;
To each described physical disk serial number in each described memory node;
Described data block is grouped, uses Mk[i, j] represents data block so that the arbitrarily data block M of k ≠ 1kThe data block M of [i, j] and k=11[j-(k-2) (i-1)+nN, i] belongs to one and entangles and delete group, wherein, k represents the numbering of the memory node belonging to described data block, k ∈ [1, N] and be integer, i represents the numbering of the virtual disk belonging to described data block, i ∈ [1, N] and be integer, j represents the numbering of the physical disk belonging to described data block, j ∈ [1, N] and be integer, n is for making the integer of j-(k-2) (i-1)+nN ∈ [1, N].
3. the division device of a data block, it is characterised in that described device is applied in storage system, applies correcting and eleting codes mechanism in described storage system, described device includes:
Construction unit, is used for building M memory node, and wherein, M is the positive integer being not more than N+1, and N is any prime number, and the physical disk number that each described memory node comprises is N;
Division unit, for each physical disk that each described memory node comprises is marked off multiple data block, wherein, the number of the data block that each described physical disk comprises is N;
Grouped element, for described data block is grouped, the data block of same group belongs to same entangling and deletes group, the data block in every group is made to be distributed on different memory nodes, and, for any physical disk in any described memory node, other data block in the group that the data block that comprises is corresponding, it is distributed on different physical disks.
4. device according to claim 3, it is characterised in that when the memory node number built is N+1, described grouped element includes:
First numbering subelement, for each described memory node serial number;
Virtual disk divides subelement, for each described memory node is divided into N number of virtual disk, makes each described virtual disk include N number of data block, and the data block that each described virtual disk includes is distributed on different physical disks;
Second numbering subelement, for each described virtual disk serial number in each described memory node;
3rd numbering subelement, for each described physical disk serial number in each described memory node;
Data block packet subelement, for described data block is grouped, uses Mk[i, j] represents data block so that the arbitrarily data block M of k ≠ 1kThe data block M of [i, j] and k=11[j-(k-2) (i-1)+nN, i] belongs to one and entangles and delete group, wherein, k represents the numbering of the memory node belonging to described data block, k ∈ [1, N] and be integer, i represents the numbering of the virtual disk belonging to described data block, i ∈ [1, N] and be integer, j represents the numbering of the physical disk belonging to described data block, j ∈ [1, N] and be integer, n is for making the integer of j-(k-2) (i-1)+nN ∈ [1, N].
5. a storage system, it is characterised in that apply correcting and eleting codes mechanism in described storage system,
Described storage system includes M memory node, and wherein, M is the positive integer being not more than N+1, and N is any prime number, and each described memory node comprises the physical disk that number is N; Each physical disk is divided into multiple data block, and wherein, the number of the data block that each described physical disk comprises is N; Described data block is divided into many groups, the data block of same group belongs to same entangling and deletes group, often the data block in group is distributed on different memory nodes, and, for any physical disk in any described memory node, other data block in the group that the data block that comprised is corresponding, is distributed on different physical disks.
6. storage system according to claim 5, it is characterised in that
Described storage system includes N+1 memory node, and each described memory node is divided into N number of virtual disk, and each described virtual disk includes N number of data block, and the data block that each described virtual disk includes is distributed on different physical disks;Described data block is divided into many groups, uses Mk[i, j] represents data block, the data block M of any k ≠ 1kThe data block M of [i, j] and k=11[j-(k-2) (i-1)+nN, i] belong to one group, and belong to same entangling and delete group, n is for making j-(k-2) (i-1)+nN ∈ [1, N] integer, wherein, k represents the numbering of the memory node belonging to described data block, k ∈ [1, N] and be integer, i represents the numbering of the virtual disk belonging to described data block, i ∈ [1, N] and be integer, j represents the numbering of the physical disk belonging to described data block, j ∈ [1, N] and be integer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610003072.0A CN105677249B (en) | 2016-01-04 | 2016-01-04 | The division methods of data block, apparatus and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610003072.0A CN105677249B (en) | 2016-01-04 | 2016-01-04 | The division methods of data block, apparatus and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105677249A true CN105677249A (en) | 2016-06-15 |
| CN105677249B CN105677249B (en) | 2019-01-15 |
Family
ID=56298824
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610003072.0A Active CN105677249B (en) | 2016-01-04 | 2016-01-04 | The division methods of data block, apparatus and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105677249B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106527986A (en) * | 2016-11-03 | 2017-03-22 | 北京百度网讯科技有限公司 | Method and device for storing data |
| CN113176858A (en) * | 2021-05-07 | 2021-07-27 | 锐捷网络股份有限公司 | Data processing method, storage system and storage device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080168225A1 (en) * | 2007-01-10 | 2008-07-10 | International Business Machines Corporation | Providing enhanced tolerance of data loss in a disk array system |
| CN101840377A (en) * | 2010-05-13 | 2010-09-22 | 上海交通大学 | Data storage method based on RS (Reed-Solomon) erasure codes |
| CN103152395A (en) * | 2013-02-05 | 2013-06-12 | 北京奇虎科技有限公司 | Storage method and device of distributed file system |
| CN103984607A (en) * | 2013-02-08 | 2014-08-13 | 华为技术有限公司 | Distributed storage method, device and system |
-
2016
- 2016-01-04 CN CN201610003072.0A patent/CN105677249B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080168225A1 (en) * | 2007-01-10 | 2008-07-10 | International Business Machines Corporation | Providing enhanced tolerance of data loss in a disk array system |
| CN101840377A (en) * | 2010-05-13 | 2010-09-22 | 上海交通大学 | Data storage method based on RS (Reed-Solomon) erasure codes |
| CN103152395A (en) * | 2013-02-05 | 2013-06-12 | 北京奇虎科技有限公司 | Storage method and device of distributed file system |
| CN103984607A (en) * | 2013-02-08 | 2014-08-13 | 华为技术有限公司 | Distributed storage method, device and system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106527986A (en) * | 2016-11-03 | 2017-03-22 | 北京百度网讯科技有限公司 | Method and device for storing data |
| CN113176858A (en) * | 2021-05-07 | 2021-07-27 | 锐捷网络股份有限公司 | Data processing method, storage system and storage device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105677249B (en) | 2019-01-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6985995B2 (en) | Data file migration from a mirrored RAID to a non-mirrored XOR-based RAID without rewriting the data | |
| CN110334075B (en) | Data migration method based on message middleware and related equipment | |
| CN104978151B (en) | Data reconstruction method in the data de-duplication storage system perceived based on application | |
| CN103136243B (en) | File system duplicate removal method based on cloud storage and device | |
| CN105468473A (en) | Data migration method and data migration apparatus | |
| CN106021016A (en) | Virtual point in time access between snapshots | |
| US20080201335A1 (en) | Method and Apparatus for Storing Data in a Peer to Peer Network | |
| CN101853190B (en) | Data integrity verification method suitable for embedded processor | |
| CN106777029A (en) | A kind of distributed rule automotive engine system and its construction method | |
| US20180136869A1 (en) | Reducing storage requirements for storing memory addresses and values | |
| CN107798130A (en) | A kind of Snapshot Method of distributed storage | |
| CN104239443B (en) | A kind of storage method of serialized data operation log | |
| CN107851102A (en) | System and method for key assignments storage | |
| CN1983266A (en) | File system storing transaction records in flash-like media | |
| CN103914522A (en) | Data block merging method applied to deleting duplicated data in cloud storage | |
| CN103064765A (en) | Method and device for data recovery and cluster storage system | |
| CN110383251A (en) | Storage system, computer readable recording medium, the control method of system | |
| CN104809178A (en) | Write-in method of key/value database memory log | |
| CN103049539A (en) | Method and device for storing file data in file system | |
| Nodine et al. | Greed sort: Optimal deterministic sorting on parallel disks | |
| CN106775481B (en) | Method for reading data and equipment | |
| CN107111534A (en) | A kind of method and apparatus of data processing | |
| CN111506552B (en) | Dynamic database design method and system of tree structure | |
| Kirsch et al. | Simple summaries for hashing with choices | |
| CN105677249A (en) | Data block partitioning method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |