CN112118316B - Distributed storage data migration method and device - Google Patents
Distributed storage data migration method and device Download PDFInfo
- Publication number
- CN112118316B CN112118316B CN202010988120.2A CN202010988120A CN112118316B CN 112118316 B CN112118316 B CN 112118316B CN 202010988120 A CN202010988120 A CN 202010988120A CN 112118316 B CN112118316 B CN 112118316B
- Authority
- CN
- China
- Prior art keywords
- node
- data
- migration
- migrated
- coding
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005012 migration Effects 0.000 title claims abstract description 146
- 238000013508 migration Methods 0.000 title claims abstract description 146
- 238000000034 method Methods 0.000 title claims abstract description 44
- 230000005540 biological transmission Effects 0.000 claims abstract description 33
- 238000012544 monitoring process Methods 0.000 claims description 19
- 230000009467 reduction Effects 0.000 claims description 19
- 230000006854 communication Effects 0.000 claims description 13
- 238000004891 communication Methods 0.000 claims description 13
- 230000001960 triggered effect Effects 0.000 claims description 12
- 230000004044 response Effects 0.000 claims description 7
- 230000000737 periodic effect Effects 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 4
- 230000008569 process Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 238000013144 data compression Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000000844 transformation Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000007175 bidirectional communication Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000001771 impaired effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000003245 working effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
- H04L1/0056—Systems characterized by the type of code used
- H04L1/0057—Block codes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a distributed storage data migration method and a device, wherein the method comprises the following steps: determining data to be migrated, an immigration node and an immigration node; further determining a plurality of intermediate nodes forming a migration path at a location in the network topology of the distributed storage; sequentially executing source coding comprising Huffman coding and channel coding comprising fountain codes by the data to be migrated by the migrated node, and sending the source coding and the channel coding to a first intermediate node in a plurality of intermediate nodes in a data packet form; performing, by the head intermediate node, network coding on the data packet and transmitting to a tail intermediate node of the plurality of intermediate nodes through the migration path to perform network decoding to restore the data packet; and the tail intermediate node sends the restored data packet to the migration node and sequentially executes channel decoding comprising fountain codes and source decoding comprising Huffman codes to restore the data to be migrated. The invention can avoid feedback retransmission, is compatible with low-quality channels, is suitable for point-to-multipoint transmission and further improves the transmission efficiency and reliability.
Description
Technical Field
The present invention relates to the field of data migration, and in particular, to a method and an apparatus for migrating distributed storage data.
Background
The conventional data migration usually depends on the bidirectional communication between the migration end and the simple storage and forwarding of information by the intermediate node: the file of the migrating end is transmitted in the form of a plurality of small data packets, one file is simply and roughly cut into a plurality of data packet-sized segments, the migrating end codes the data packets and then passes the packets with information through an intermediate node, and the intermediate node directly forwards the information to be transmitted to the receiving end without any processing.
However, none of these conventional methods can reach the theoretical maximum capacity of network multicast transmission, and the success or failure depends on the evaluation of each packet received by the receiving end, and if the packet can be decoded, an acknowledgement is sent to the migrating end; otherwise, the damaged packet is discarded, and a request is transmitted to let the migrating end send the packet again. For example, commonly used TCP/IP uses a retransmission mechanism to ensure the reliability of the transmission. However, in many cases the TCP/IP protocol is not suitable, such as point-to-multipoint transmission, or transmission over severely impaired channels (poor quality wireless or satellite links). Feedback retransmission has poor transmission efficiency and reliability (packet loss) when the transmission distance is too long or a "feedback storm" occurs in the case of network multicast.
Aiming at the problems that the distributed storage data migration in the prior art has strict requirements on channels, is not friendly to point-to-multipoint transmission and has low transmission efficiency and reliability, no effective solution is available at present.
Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a method and an apparatus for migrating distributed storage data, which can avoid feedback retransmission, are compatible with low quality channels, and are suitable for point-to-multipoint transmission, thereby improving transmission efficiency and reliability.
Based on the above object, a first aspect of the embodiments of the present invention provides a distributed storage data migration method, including the following steps:
determining data to be migrated, a migration-in node and a migration-out node in response to the fact that a distributed storage cluster migration event is triggered through monitoring;
further determining a plurality of intermediate nodes forming migration paths based on the positions of the immigration nodes and the immigration nodes in the network topology of the distributed storage;
sequentially executing source coding comprising Huffman coding and channel coding comprising fountain codes by the data to be migrated by the migrated node, and sending the source coding and the channel coding to a first intermediate node in a plurality of intermediate nodes in a data packet form;
performing, by the head intermediate node, network coding on the data packet and transmitting to a tail intermediate node of the plurality of intermediate nodes through the migration path to perform network decoding to restore the data packet;
and the tail intermediate node sends the restored data packet to the migration node and sequentially executes channel decoding comprising fountain codes and source decoding comprising Huffman codes to restore the data to be migrated.
In some embodiments, the distributed storage cluster migration event comprises: the method comprises the following steps that any node fails, expands or contracts, and/or migration scanning events triggered periodically during the period that all nodes do not fail, expand or contract exist.
In some embodiments, determining the data to be migrated, the migrating node and the migrating node comprises: determining a node with a fault event as a migration node, and determining all data in the migration node as data to be migrated; or determining the node with the capacity reduction event as an emigration node, and determining the data matched with the capacity reduction in the emigration node as the data to be migrated; or determining the node with the capacity expansion event as the migration node.
In some embodiments, the periodic trigger comprises: the periodic triggering is performed using a period determined based on the total amount of traffic of the distributed storage or is performed based on a fixed threshold period set in advance.
In some embodiments, sequentially performing source coding including huffman coding on the data to be migrated comprises: performing source coding including Huffman coding on the data to be migrated to change the data to be migrated from a first data size to a second data size smaller than the first data size.
In some embodiments, performing channel coding including fountain codes on data to be migrated includes:
determining an expected packet loss rate based on a signal fidelity test or inference;
and generating a plurality of coding packets containing redundant information based on the expected packet loss rate, wherein the product of the information amount of the data to be migrated and the expected packet loss rate in the plurality of coding packets is larger than the total information amount of the data to be migrated.
In some embodiments, the migration path from the head intermediate node to the tail intermediate node uses a different communication channel than from the migrating node to the head intermediate node and from the tail intermediate node to the migrating node;
performing network coding on the data packet includes: network coding is performed on the data packets to cause the data packets to traverse the migration path in a manner suitable for transmission over the communication channel of the migration path.
In view of the above object, a second aspect of the embodiments of the present invention provides a distributed storage data migration apparatus, including:
a processor; and
a memory storing program code executable by the processor, the program code when executed performing the steps of:
determining data to be migrated, an immigration node and an immigration node in response to the fact that a distributed storage cluster migration event is triggered through monitoring;
further determining a plurality of intermediate nodes forming migration paths based on the positions of the immigration nodes and the immigration nodes in the network topology of the distributed storage;
sequentially executing source coding comprising Huffman coding and channel coding comprising fountain codes by the data to be migrated by the migrated node, and sending the source coding and the channel coding to a first intermediate node in a plurality of intermediate nodes in a data packet form;
performing, by the head intermediate node, network coding on the data packet and transmitting to a tail intermediate node of the plurality of intermediate nodes through the migration path to perform network decoding to restore the data packet;
and the tail intermediate node sends the restored data packet to the migration node and sequentially executes channel decoding comprising fountain codes and source decoding comprising Huffman codes to restore the data to be migrated.
In some embodiments, the distributed storage cluster migration event comprises: the method comprises the following steps that a fault, capacity expansion or capacity reduction event occurs to any node, and/or a migration scanning event triggered periodically during the period that no fault, capacity expansion or capacity reduction occurs to all nodes exists;
determining the data to be migrated, the migration node and the migration node comprises: determining a node with a fault event as a migration node, and determining all data in the migration node as data to be migrated; or determining the node with the capacity reduction event as a migration node, and determining the data matched with the capacity reduction in the migration node as the data to be migrated; or determining the node with the capacity expansion event as the migration node.
In some embodiments, performing channel coding including fountain codes on data to be migrated includes: determining an expected packet loss rate based on a signal fidelity test or inference; generating a plurality of coding packets containing redundant information based on the expected packet loss rate, wherein the product of the information amount of the data to be migrated and the expected packet loss rate in the plurality of coding packets is larger than the total information amount of the data to be migrated;
the migration path from the head intermediate node to the tail intermediate node uses a different communication channel than the communication channel from the migrated node to the head intermediate node and from the tail intermediate node to the migrated node;
performing network coding on the data packet includes: network coding is performed on the data packets to cause the data packets to traverse the migration path in a manner suitable for transmission over the communication channel of the migration path.
The invention has the following beneficial technical effects: according to the distributed storage data migration method and device provided by the embodiment of the invention, the data to be migrated, the migration-in node and the migration-out node are determined by responding to the monitoring to trigger the migration event of the distributed storage cluster; further determining a plurality of intermediate nodes forming migration paths based on the positions of the immigration nodes and the immigration nodes in the network topology of the distributed storage; sequentially executing source coding comprising Huffman coding and channel coding comprising fountain codes by the data to be migrated by the migrated node, and sending the source coding and the channel coding to a first intermediate node in a plurality of intermediate nodes in a data packet form; performing, by the head intermediate node, network coding on the data packet and transmitting to a tail intermediate node of the plurality of intermediate nodes through the migration path to perform network decoding to restore the data packet; the technical scheme that the tail intermediate node sends the restored data packet to the immigration node and sequentially executes channel decoding comprising fountain codes and information source decoding comprising Huffman codes to restore the data to be migrated can avoid feedback retransmission, is compatible with low-quality channels, is suitable for point-to-multipoint transmission and further improves transmission efficiency and reliability.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a distributed storage data migration method according to the present invention;
fig. 2 is a detailed flowchart of the distributed storage data migration method provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
In view of the above, a first aspect of the embodiments of the present invention provides an embodiment of a distributed storage data migration method for avoiding feedback retransmission, being compatible with low quality channels, and being suitable for point-to-multipoint transmission. Fig. 1 is a schematic flow chart of a distributed storage data migration method provided by the present invention.
The distributed storage data migration method, as shown in fig. 1, includes the following steps:
step S101: determining data to be migrated, an immigration node and an immigration node in response to the fact that a distributed storage cluster migration event is triggered through monitoring;
step S103: further determining a plurality of intermediate nodes forming migration paths based on the positions of the immigration nodes and the immigration nodes in the network topology of the distributed storage;
step S105: sequentially executing source coding comprising Huffman coding and channel coding comprising fountain codes by the data to be migrated by the migrated node, and sending the source coding and the channel coding to a first intermediate node in a plurality of intermediate nodes in a data packet form;
step S107: performing, by the head intermediate node, network coding on the data packet and transmitting to a tail intermediate node of the plurality of intermediate nodes through the migration path to perform network decoding to restore the data packet;
step S109: and the tail intermediate node sends the restored data packet to the migration node and sequentially executes channel decoding comprising fountain codes and source decoding comprising Huffman codes to restore the data to be migrated.
Huffman coding, fountain coding, and network coding provide convenience for data migration in distributed storage systems. Huffman coding is a very effective data compression technology, and can be compressed by 20% -90% according to the characteristics of data to be compressed, but Huffman codes have no error protection function, and when decoding, a code string can be correctly decoded if no error exists; if the code string has errors, the coding is increased, and the reliability is improved. The fountain code is a new channel coding method aiming at large-scale network data distribution and reliable transmission, and different from the traditional block code, the fountain code can independently generate any number of code words according to a certain probability distribution. The receiving end does not need to care about specific code packets and packet sequences, and can realize correct coding only by receiving enough code packets, thereby providing the reliability of the system. The intermediate node of the network only has the function of storing and forwarding, can use network coding to perform linear or nonlinear coding operation on the received related information, directly transmits the processed coded data, and has the function of coding information in the middle, thereby reducing the transmission and duplication process in the middle, saving the consumption of network bandwidth resources and greatly improving the transmission efficiency.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program to instruct relevant hardware to perform the processes, and the processes can be stored in a computer readable storage medium, and when executed, the processes can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a Random Access Memory (RAM), or the like. Embodiments of the computer program may achieve the same or similar effects as any of the preceding method embodiments to which it corresponds.
In some embodiments, the distributed storage cluster migration event comprises: the method comprises the following steps that any node fails, expands or contracts, and/or migration scanning events triggered periodically during the period that all nodes do not fail, expand or contract exist. The combination of active monitoring and passive triggering can better discover the migration scanning event in time.
In some embodiments, determining the data to be migrated, the migrating node, and the migrating node includes: determining a node with a fault event as a migration node, and determining all data in the migration node as data to be migrated; or determining the node with the capacity reduction event as a migration node, and determining the data matched with the capacity reduction in the migration node as the data to be migrated; or determining the node with the capacity expansion event as the migration node. Further, determining event requirements based on discovered migration scan events facilitates fast processing of the migration, improving efficiency.
In some embodiments, the periodic trigger comprises: the periodic triggering is performed using a period determined based on the total amount of traffic of the distributed storage or is performed based on a fixed threshold period set in advance. The period can be selectively set according to the actual environment requirement so as to further improve the working effect.
In some embodiments, sequentially performing source coding including huffman coding on the data to be migrated comprises: performing source coding including Huffman coding on the data to be migrated to change the data to be migrated from a first data size to a second data size smaller than the first data size. The data compression of source coding can reduce the transmission amount and further improve the efficiency.
In some embodiments, performing channel coding including fountain codes on data to be migrated includes:
determining an expected packet loss rate based on a signal fidelity test or inference;
and generating a plurality of coding packets containing redundant information based on the expected packet loss rate, wherein the product of the information amount of the data to be migrated and the expected packet loss rate in the plurality of coding packets is larger than the total information amount of the data to be migrated.
The fountain code can independently generate any number of code words according to specific probability distribution, a receiving end does not need to care about specific code packets and packet sequences, and correct coding can be realized only by receiving enough code packets, so that the reliability of a system is improved. It is therefore necessary to ensure that enough code packets are generated to achieve the desired amount of information to transmit.
In some embodiments, the migration path from the head intermediate node to the tail intermediate node uses a different communication channel than from the migrating node to the head intermediate node and from the tail intermediate node to the migrating node. Performing network coding on the data packet includes: network coding is performed on the data packets to cause the data packets to traverse the migration path in a manner suitable for transmission over the communication channel of the migration path. Selectively adjusting the channel coding according to the change of the path may further improve transmission efficiency.
In order to more clearly illustrate the implementation of the present invention, the following embodiments are further provided to illustrate the present invention.
Fig. 2 shows a detailed implementation flow of the present invention. The data migration method comprises the steps that a data migration unit 100 and a data migration unit 200 are used, the data migration unit 100 is used for sequentially monitoring data of distributed storage cluster nodes of data to be migrated, compressing the data through Huffman coding, performing fountain code coding and network coding on the migrated data again, packaging the migrated data into a migrated data packet and sending the migrated data packet to a network; the data migration unit 200 is configured to perform data monitoring, network decoding, fountain code decoding, and huffman decoding on the migration data packet in sequence if the migration data packet migrates into the distributed storage cluster node, and restore the migration data packet to the original data.
The data emigration unit 100 performs the following steps:
s1: monitoring data and nodes to be migrated by a data monitoring module, wherein the nodes comprise migrated nodes and migrated nodes;
wherein s 1-0: when the data monitoring module monitors that the distributed storage cluster has node faults, node expansion or capacity reduction, the data to be migrated and the nodes of the current distributed storage cluster are immediately acquired; otherwise, perform s 1-1: when the nodes of the distributed storage cluster are normal, and the nodes are not expanded and reduced, monitoring the data to be migrated and the nodes of the current distributed storage cluster according to a certain period.
S2: performing data compression based on Huffman coding on the data to be migrated according to the data to be migrated acquired by the data monitoring module by using a Huffman coding module;
s3: encoding the data which has been compressed by the Huffman-based coding with fountain codes again;
at an intermediate node between data migration unit 100 and data migration unit 200:
s4: the network coding module is used for carrying out network coding on the data which is compressed based on the Huffman coding and encoded based on the fountain code again so as to adapt to channel transmission of the intermediate node;
data migration unit 200 performs the following steps:
s5: the data immigration monitoring module is used for monitoring data to be immigrated;
s6: network decoding is carried out on the migrated data by a network decoding module;
s7: performing fountain code decoding on the data which is migrated into the network and decoded by a fountain code decoding module;
s8: and the Huffman decoding module is used for further carrying out Huffman decoding on the data decoded by the fountain code to recover the original data.
The invention combines Hoffman codes, fountain codes and network codes, realizes the data migration and migration of the cluster nodes in the virtual distributed storage system, reduces the data migration amount through the compression number of the Hoffman codes, improves the reliability of the migrated data through the fountain codes, and greatly improves the throughput of the whole network through the network codes.
It can be seen from the foregoing embodiment that, in the distributed storage data migration method provided in the embodiment of the present invention, the data to be migrated, the migration-in node, and the migration-out node are determined in response to the event that triggers the migration of the distributed storage cluster through monitoring; further determining a plurality of intermediate nodes forming migration paths based on the positions of the immigration nodes and the immigration nodes in the network topology of the distributed storage; sequentially executing source coding comprising Huffman coding and channel coding comprising fountain codes by the migration node to-be-migrated data, and sending the source coding and the channel coding to a first intermediate node in a plurality of intermediate nodes in a data packet form; performing, by the head intermediate node, network coding on the data packet and transmitting to a tail intermediate node of the plurality of intermediate nodes through the migration path to perform network decoding to restore the data packet; the technical scheme that the data packet is sent to the emigration node after being restored by the tail intermediate node and channel decoding including fountain codes and information source decoding including Huffman coding are sequentially executed to restore the data to be migrated can avoid feedback retransmission, is compatible with low-quality channels, is suitable for point-to-multipoint transmission and further improves transmission efficiency and reliability.
It should be particularly noted that, the steps in the embodiments of the distributed storage data migration method described above may be mutually intersected, replaced, added, or deleted, and therefore, these reasonable permutation and combination transformations should also belong to the scope of the present invention, and should not limit the scope of the present invention to the described embodiments.
In view of the above-mentioned objects, a second aspect of the embodiments of the present invention provides an embodiment of a distributed storage data migration apparatus suitable for point-to-multipoint transmission, which avoids feedback retransmission, is compatible with low quality channels, and is suitable for point-to-multipoint transmission. The distributed storage data migration device comprises:
a processor; and
a memory storing program code executable by the processor, the program code when executed performing the steps of:
determining data to be migrated, a migration-in node and a migration-out node in response to the fact that a distributed storage cluster migration event is triggered through monitoring;
further determining a plurality of intermediate nodes forming migration paths based on the positions of the immigration nodes and the immigration nodes in the network topology of the distributed storage;
sequentially executing source coding comprising Huffman coding and channel coding comprising fountain codes by the data to be migrated by the migrated node, and sending the source coding and the channel coding to a first intermediate node in a plurality of intermediate nodes in a data packet form;
performing, by the head intermediate node, network coding on the data packet and transmitting to a tail intermediate node of the plurality of intermediate nodes through the migration path to perform network decoding to restore the data packet;
and the tail intermediate node sends the restored data packet to the migration node and sequentially executes channel decoding comprising fountain codes and source decoding comprising Huffman codes to restore the data to be migrated.
In some embodiments, the distributed storage cluster migration event comprises: the method comprises the following steps that a fault, capacity expansion or capacity reduction event occurs to any node, and/or a migration scanning event triggered periodically during the period that no fault, capacity expansion or capacity reduction occurs to all nodes exists;
determining the data to be migrated, the migration node and the migration node comprises: determining a node with a fault event as a migration node, and determining all data in the migration node as data to be migrated; or determining the node with the capacity reduction event as a migration node, and determining the data matched with the capacity reduction in the migration node as the data to be migrated; or determining the node with the capacity expansion event as the migration node.
In some embodiments, performing channel coding including fountain codes on data to be migrated includes: determining an expected packet loss rate based on a signal fidelity test or inference; generating a plurality of coding packets containing redundant information based on the expected packet loss rate, wherein the product of the information amount of the data to be migrated and the expected packet loss rate in the plurality of coding packets is larger than the total information amount of the data to be migrated;
the migration path from the head intermediate node to the tail intermediate node uses a different communication channel than from the migrant node to the head intermediate node and from the tail intermediate node to the migrant node. Performing network coding on the data packet includes: network coding is performed on the data packets to cause the data packets to traverse the migration path in a manner suitable for transmission over the communication channel of the migration path.
As can be seen from the foregoing embodiments, the distributed storage data migration apparatus provided in the embodiments of the present invention determines data to be migrated, an migration node, and a migration node by responding to a monitoring event that triggers a migration event of a distributed storage cluster; further determining a plurality of intermediate nodes forming migration paths based on the positions of the immigration nodes and the immigration nodes in the network topology of the distributed storage; sequentially executing source coding comprising Huffman coding and channel coding comprising fountain codes by the data to be migrated by the migrated node, and sending the source coding and the channel coding to a first intermediate node in a plurality of intermediate nodes in a data packet form; performing, by the head intermediate node, network coding on the data packet and transmitting to a tail intermediate node of the plurality of intermediate nodes through the migration path to perform network decoding to restore the data packet; the technical scheme that the tail intermediate node sends the restored data packet to the immigration node and sequentially executes channel decoding comprising fountain codes and information source decoding comprising Huffman codes to restore the data to be migrated can avoid feedback retransmission, is compatible with low-quality channels, is suitable for point-to-multipoint transmission and further improves transmission efficiency and reliability.
It should be particularly noted that, the above-mentioned embodiment of the distributed storage data migration apparatus employs the embodiment of the distributed storage data migration method to specifically describe the working process of each module, and those skilled in the art can easily think that these modules are applied to other embodiments of the distributed storage data migration method. Of course, since the steps in the embodiment of the distributed storage data migration method may be mutually intersected, replaced, added, or deleted, these reasonable permutation and combination transformations of the distributed storage data migration apparatus should also belong to the scope of the present invention, and should not limit the scope of the present invention to the embodiment.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of an embodiment of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A distributed storage data migration method is characterized by comprising the following steps:
determining data to be migrated, an immigration node and an immigration node in response to the fact that a distributed storage cluster migration event is triggered through monitoring;
further determining a plurality of intermediate nodes forming a migration path based on the locations of the migrating nodes and the migrating nodes in the network topology of the distributed storage;
sequentially performing source coding including Huffman coding and channel coding including fountain codes on the data to be migrated by the migration node, and sending the data to a first intermediate node in a plurality of intermediate nodes in a data packet form;
performing, by the head intermediate node, network coding on the data packet and performing network decoding to restore the data packet by sending to a tail intermediate node of the plurality of intermediate nodes through the migration path;
and the tail intermediate node sends the restored data packet to the migration node and sequentially executes channel decoding comprising fountain codes and source decoding comprising Huffman coding so as to restore the data to be migrated.
2. The method of claim 1, wherein the distributed storage cluster migration event comprises: the method comprises the following steps that any node fails, expands or contracts, and/or migration scanning events triggered periodically during the period that all nodes do not fail, expand or contract exist.
3. The method of claim 2, wherein determining the data to be migrated, the migrating node, and the migrating node comprises:
determining a node with a fault event as the migrated node, and determining all data in the migrated node as the data to be migrated; or determining a node with a capacity reduction event as the migration node, and determining data matched with the capacity reduction in the migration node as the data to be migrated; or determining the node with the capacity expansion event as the migration node.
4. The method of claim 2, wherein the periodic triggering comprises: the periodic triggering is performed using a period determined based on the total amount of traffic of the distributed storage or is performed based on a fixed threshold period set in advance.
5. The method of claim 1, wherein sequentially performing source coding including huffman coding on the data to be migrated comprises:
performing source coding including Huffman coding on the data to be migrated to change the data to be migrated from a first data size to a second data size smaller than the first data size.
6. The method of claim 1, wherein performing channel coding comprising fountain codes on the data to be migrated comprises:
determining an expected packet loss rate based on a signal fidelity test or inference;
generating a plurality of coding packets containing redundant information based on the expected packet loss rate, wherein the product of the information amount of the data to be migrated and the expected packet loss rate in the plurality of coding packets is greater than the total information amount of the data to be migrated.
7. The method of claim 1 wherein the migration path from the head intermediate node to the tail intermediate node uses a different communication channel than from the migrant node to the head intermediate node and from the tail intermediate node to the migrant node;
performing network coding on the data packet comprises: performing network coding on the data packets to cause the data packets to traverse the migration path in a manner suitable for transmission over a communication channel of the migration path.
8. A distributed storage data migration apparatus, comprising:
a processor; and
a memory storing program code executable by the processor, the program code when executed performing the steps of:
determining data to be migrated, an immigration node and an immigration node in response to the fact that a distributed storage cluster migration event is triggered through monitoring;
further determining a plurality of intermediate nodes forming a migration path based on the locations of the migrating nodes and the migrating nodes in the network topology of the distributed storage;
sequentially performing source coding including Huffman coding and channel coding including fountain codes on the data to be migrated by the migration node, and sending the data to a first intermediate node in a plurality of intermediate nodes in a data packet form;
performing, by the head intermediate node, network coding on the data packet and performing network decoding to restore the data packet by sending to a tail intermediate node of the plurality of intermediate nodes through the migration path;
and the tail intermediate node sends the restored data packet to the migration node and sequentially executes channel decoding including fountain codes and source decoding including Huffman coding to restore the data to be migrated.
9. The apparatus of claim 8, wherein the distributed storage cluster migration event comprises: the method comprises the following steps that a fault, capacity expansion or capacity reduction event occurs to any node, and/or a migration scanning event triggered periodically during the period that no fault, capacity expansion or capacity reduction occurs to all nodes exists;
determining the data to be migrated, the migration node and the migration node comprises: determining a node with a fault event as the migrated node, and determining all data in the migrated node as the data to be migrated; or determining a node with a capacity reduction event as the migration node, and determining data matched with the capacity reduction in the migration node as the data to be migrated; or determining the node with the capacity expansion event as the migration node.
10. The apparatus of claim 8, wherein performing channel coding comprising fountain codes on the data to be migrated comprises: determining an expected packet loss rate based on a signal fidelity test or inference; generating a plurality of coding packets containing redundant information based on the expected packet loss rate, wherein the product of the information amount of the data to be migrated and the expected packet loss rate in the plurality of coding packets is larger than the total information amount of the data to be migrated;
the migration path from the head intermediate node to the tail intermediate node uses a different communication channel than from the migrant node to the head intermediate node and from the tail intermediate node to the migrant node;
performing network coding on the data packet comprises: performing network coding on the data packets to cause the data packets to traverse the migration path in a manner suitable for transmission over a communication channel of the migration path.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010988120.2A CN112118316B (en) | 2020-09-18 | 2020-09-18 | Distributed storage data migration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010988120.2A CN112118316B (en) | 2020-09-18 | 2020-09-18 | Distributed storage data migration method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112118316A CN112118316A (en) | 2020-12-22 |
| CN112118316B true CN112118316B (en) | 2022-06-10 |
Family
ID=73800726
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010988120.2A Active CN112118316B (en) | 2020-09-18 | 2020-09-18 | Distributed storage data migration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112118316B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115103020B (en) * | 2022-08-25 | 2022-11-15 | 建信金融科技有限责任公司 | Data migration processing method and device |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111273998A (en) * | 2020-01-19 | 2020-06-12 | 苏州浪潮智能科技有限公司 | Fountain code and network coding-based dynamic migration method and system for virtual machine |
-
2020
- 2020-09-18 CN CN202010988120.2A patent/CN112118316B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111273998A (en) * | 2020-01-19 | 2020-06-12 | 苏州浪潮智能科技有限公司 | Fountain code and network coding-based dynamic migration method and system for virtual machine |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112118316A (en) | 2020-12-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Karzand et al. | Design of FEC for low delay in 5G | |
| US6675346B2 (en) | Code transmission scheme for communication system using error correcting codes | |
| EP1421765B1 (en) | Extension header compression | |
| Du et al. | RLT Code Based Handshake‐Free Reliable MAC Protocol for Underwater Sensor Networks | |
| KR102383892B1 (en) | Coding and decoding method, apparatus, system and medium of self-adapting system code FEC based on media content | |
| CN112600647A (en) | Multi-hop wireless network transmission method based on network coding endurance | |
| CN112118316B (en) | Distributed storage data migration method and device | |
| KR102002939B1 (en) | On-demand file recovery methods and systems | |
| CN113364508B (en) | Voice data transmission control method, system and equipment | |
| CN111273998B (en) | Fountain code and network coding-based dynamic migration method and system for virtual machine | |
| WO2021208694A1 (en) | Data transmission method and network device | |
| Tsimbalo et al. | Reliability of relay networks under random linear network coding | |
| CN113098661A (en) | Fountain code-based file transmission method under satellite channel | |
| Paris et al. | Addressing reliability needs of industrial applications in 5G NR with network coding | |
| Singhvi et al. | Rate-optimal streaming codes over the three-node decode-and-forward relay network | |
| Hansen et al. | Bridging inter-flow and intra-flow network coding for video applications: Testbed description and performance evaluation | |
| Sagduyu et al. | On broadcast stability region in random access through network coding | |
| Malak et al. | ARQ with cumulative feedback to compensate for burst errors | |
| US20170339045A1 (en) | System and method for transmitting data | |
| US10880731B2 (en) | System and method for enhanced diversity and network coding (eDC-NC) | |
| Malak et al. | Throughput and delay analysis for coded ARQ | |
| EP3035540B1 (en) | Maximum likelihood erasure decoding of sparse graph codes | |
| Hashemi et al. | Delete-and-conquer: Rateless coding with constrained feedback | |
| US20240235731A1 (en) | Forward Error Correction Decode Processing Power Reduction | |
| RU2789978C1 (en) | Method for parallel data transmission in self-organizing radio networks of groups of robotics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |