WO2016107220A1 - 基于重复数据删除的远程复制方法及装置 - Google Patents
基于重复数据删除的远程复制方法及装置 Download PDFInfo
- Publication number
- WO2016107220A1 WO2016107220A1 PCT/CN2015/089977 CN2015089977W WO2016107220A1 WO 2016107220 A1 WO2016107220 A1 WO 2016107220A1 CN 2015089977 W CN2015089977 W CN 2015089977W WO 2016107220 A1 WO2016107220 A1 WO 2016107220A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data block
- fingerprint
- snapshot
- added
- primary
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G06F11/1453—Management of the data involved in backup or backup restore using de-duplication of the data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1456—Hardware arrangements for backup
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/128—Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
- G06F16/1752—De-duplication implemented within the file system, e.g. based on file segments based on file chunks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/184—Distributed file systems implemented as replicated file system
- G06F16/1844—Management specifically adapted to replicated file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0613—Improving I/O performance in relation to throughput
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
- G06F3/0641—De-duplication techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Definitions
- the embodiments of the present invention relate to the field of computer technologies, and in particular, to a remote replication method and apparatus based on deduplication.
- Deduplication is a data reduction technique designed to reduce the amount of storage used in storage systems. Find duplicate variable-size data blocks at different locations in different data, and replace these duplicate data blocks with indicators to reduce the amount of data stored. Highly redundant data sets (such as backup data) benefit greatly from data deduplication technology, and users can achieve a 10 to 1 to 50 to 1 reduction ratio.
- Remote replication is a disaster recovery technology. The data of the primary device is copied to the remote disaster recovery device. When the data of the primary device is damaged, the data can be recovered through the disaster recovery device.
- the remote replication scheme based on the deduplication in the prior art is to send the fingerprint of the difference data of the data to be copied of the primary device to the disaster recovery device for query, and the fingerprint of the disaster recovery device is loaded. Whether the fingerprint of the difference data of the primary device exists on the disaster-tolerant device, and sends the fingerprint of the device at the disaster-tolerant device to the primary device. If the fingerprint exists in the disaster-tolerant device, the difference data is considered. If the fingerprint is not in the disaster recovery device, the difference data does not exist on the DR device. The master device needs to send the difference data to the DR device.
- the problem in the prior art is that the fingerprint query operation is too cumbersome, and the fingerprint query process increases network interaction and occupies more network bandwidth.
- the embodiment of the invention provides a remote copying method and device based on deduplication, which overcomes the problem that the fingerprint query operation in the prior art is too cumbersome, the fingerprint query process increases network interaction, and occupies more network bandwidth.
- the embodiment of the present invention provides a remote replication method based on deduplication, which is applied to a remote replication system, where the system includes: a primary device and a disaster recovery device, and the primary device and the device The disaster mitigation device stores the first snapshot, and the method includes:
- the master device acquires a second snapshot of the master device
- the first data block, the fingerprint of the first data block, and the added The metadata of the data block is sent to the disaster tolerant device.
- the primary end device determines whether a fingerprint of the data block added in the second snapshot relative to the first snapshot is related to the first snapshot
- the fingerprints in the data block are the same, including:
- the fingerprint of the first data block is The fingerprint of the data block in the first snapshot is different, and the fingerprint of the data block except the first data block in the added data block is the same as the fingerprint of the data block in the first snapshot.
- the primary end device determines whether a fingerprint of the data block added in the second snapshot relative to the first snapshot is related to the first snapshot
- the fingerprints in the data block are the same, including:
- the primary device acquires a difference record; the difference record is used to record the added data block;
- the processing module is specifically configured to:
- the method further includes:
- the first data block in the added data block is the same as the fingerprint of the data block in the first snapshot, the first data block is not copied to the disaster tolerant device.
- the embodiment of the present invention provides a remote replication method based on deduplication, which is applied to a remote replication system, where the system includes: a primary device and a disaster recovery device, and the primary device and the device The disaster mitigation device stores the first snapshot, and the method includes:
- the master device acquires a second snapshot of the master device
- the master device acquires a difference record; the difference record is used to record a data block added to the first snapshot in the second snapshot;
- the master device acquires a fingerprint of the added data block
- the master device sends the fingerprint of the added data block to the disaster recovery end device, so that the disaster end device determines whether the fingerprint of the added data block already exists in the disaster recovery end. device;
- the primary end device sends the metadata of the first data block and the added data block indicated by the check information to the disaster recovery end device according to the check information.
- the acquiring, by the master device, the fingerprint of the added data block includes:
- the master device acquires a fingerprint of the added data block by using a hardware acceleration card.
- an embodiment of the present invention provides a master device, including:
- a storage module configured to store the first snapshot; the first snapshot is also stored in the disaster tolerant device;
- An acquiring module configured to acquire a second snapshot of the primary device
- a processing module configured to determine whether a fingerprint of the data block added in the second snapshot relative to the first snapshot is the same as a fingerprint of the data block in the first snapshot
- a sending module configured to: when the fingerprint of the first data block in the added data block is different from the fingerprint of the data block in the first snapshot, the fingerprint of the first data block and the first data block The metadata of the added data block is sent to the disaster tolerant device.
- the processing module is specifically configured to:
- the fingerprint of the first data block is The fingerprint of the data block in the first snapshot is different, and the fingerprint of the data block except the first data block in the added data block is the same as the fingerprint of the data block in the first snapshot.
- the acquiring module is specifically configured to:
- the difference record is used to record the added data block
- the processing module is specifically configured to:
- the processing module is further configured to:
- the first data block in the added data block is the same as the fingerprint of the data block in the first snapshot, the first data block is not copied to the disaster tolerant device.
- an embodiment of the present invention provides a master device, including:
- a storage module configured to store the first snapshot; the first snapshot is also stored in the disaster tolerant device;
- An acquiring module configured to acquire a second snapshot of the primary device
- the obtaining module is further configured to acquire a difference record, where the difference record is used to record a data block added to the first snapshot in the second snapshot;
- the obtaining module is further configured to acquire a fingerprint of the added data block
- a sending module configured to send the fingerprint of the added data block to the disaster recovery end device, where the disaster recovery end device determines whether the fingerprint of the added data block is already present in the disaster recovery end device;
- a receiving module configured to receive the check information sent by the disaster tolerant device, where the check information includes a first data block in the increased data block different from a fingerprint of the data block in the first snapshot Fingerprint
- the sending module is further configured to: send, according to the check information, the metadata of the first data block and the added data block indicated by the check information to the disaster recovery end device.
- the acquiring module is specifically configured to:
- the fingerprint of the added data block is obtained by a hardware acceleration card.
- the embodiment of the present invention provides a primary end device, which is applied to a remote replication system, where the system includes: the primary end device and the disaster recovery end device, the primary end device and the disaster recovery end device. Storing a first snapshot, the primary device comprising a processor and the memory, the processor and the memory being in communication over a bus, the processor executing computer instructions in the memory for use in any of the first aspects Said method.
- the embodiment of the present invention provides a primary end device, which is applied to a remote replication system, where the system includes: the primary end device and the disaster recovery end device, the primary end device and the disaster recovery end device. Storing a first snapshot, the primary device comprising a processor and the memory, the processor and the memory being in communication over a bus, the processor executing computer instructions in the memory for use in any of the second aspects Said method.
- the remote copying method and device based on deduplication acquires a second snapshot of the primary device by the primary device; the primary device determines that the second snapshot is increased relative to the first snapshot. Whether the fingerprint of the data block is the same as the fingerprint of the data block in the first snapshot; when the fingerprint of the first data block in the added data block is different from the fingerprint of the data block in the first snapshot, The first data block, the fingerprint of the first data block, and the metadata of the added data block are sent to the disaster recovery end device, and the primary end device and the disaster recovery end device both store the first snapshot.
- the remote replication based on deduplication is implemented, and the fingerprint is not sent to the disaster recovery device for query, the fingerprint query operation is reduced, and the interaction of the network is reduced, that is, the network bandwidth is reduced, and the remote replication disaster recovery is improved.
- the performance of the backup solves the problem that the fingerprint query operation in the prior art is too cumbersome, the fingerprint query process increases the network interaction, and occupies more network bandwidth.
- Embodiment 1 is a flowchart of Embodiment 1 of a remote copying method based on deduplication according to the present invention
- FIG. 2 is a schematic diagram of remote copying data according to an embodiment of a method of the present invention.
- Embodiment 3 is a flowchart of Embodiment 2 of a remote copying method based on deduplication according to the present invention
- FIG. 4 is a flowchart 1 of Embodiment 3 of a remote copying method based on deduplication according to the present invention
- Embodiment 5 is a second flowchart of Embodiment 3 of a remote replication method based on deduplication according to the present invention.
- Embodiment 1 of a primary end device is a schematic structural diagram of Embodiment 1 of a primary end device according to the present invention.
- Embodiment 7 is a schematic structural diagram of Embodiment 2 of a primary end device according to the present invention.
- FIG. 8 is a schematic structural diagram of Embodiment 3 of a primary end device according to the present invention.
- FIG. 1 is a flowchart of Embodiment 1 of a remote copying method based on deduplication according to the present invention.
- 2 is a schematic diagram of remotely replicating data according to an embodiment of a method of the present invention.
- the method in this embodiment may be applied to a remote replication system, where the system includes: a primary device and a disaster recovery device.
- the execution entity of this embodiment may be a primary device.
- the primary end device and the disaster recovery end device both store a first snapshot, and the method includes:
- Step 101 The primary device acquires a second snapshot of the primary device.
- Step 102 The master device determines whether the fingerprint of the data block added in the second snapshot relative to the first snapshot is the same as the fingerprint of the data block in the first snapshot.
- Step 103 When the fingerprint of the first data block in the added data block is different from the fingerprint of the data block in the first snapshot, the first data block, the fingerprint of the first data block, and the The metadata of the added data block is sent to the disaster tolerant device.
- the primary device needs to create a second snapshot of the data to be copied, and then obtain a first snapshot of the copied data, where the primary device and the disaster recovery device store the first snapshot, which is copied.
- the data is the previously copied data, and further determining that the second snapshot is relatively fast relative to the first Whether the fingerprint of the added data block is the same as the fingerprint of the data block in the first snapshot, where the first snapshot and the second snapshot may respectively have multiple fingerprints, if the second snapshot is opposite to the first
- the fingerprint of the data block added by the snapshot is different from the fingerprint of the data block in the first snapshot, and the data block is recorded as the first data block, and the fingerprint of the first data block and the first data block is used.
- the metadata of the added data block is sent to the disaster tolerant device, and the metadata includes a fingerprint index of the data block, that is, a location in the data block, a storage location of the data block, a size of the data block, and an identifier, and the like. .
- the first snapshot is copied to the disaster tolerant device, and if the fingerprint of the added data block in the second snapshot is different from the fingerprint of the data block in the first snapshot, that is, the second snapshot is The fingerprint 4, the data block 4 corresponding to the fingerprint 4, the metadata 23 and the fingerprint 4 of the data block 4, and the metadata 21 of the data 2 corresponding to the fingerprint 2 in the second snapshot, and the second snapshot
- the metadata 22 of the data 3 corresponding to the fingerprint 3, that is, the data in the dotted line frame in FIG. 2 is sent to the disaster recovery end device.
- the disaster tolerance end device After receiving the data, the disaster tolerance end device writes the data, and creates a snapshot after the copying is completed.
- the method in this embodiment further includes:
- the first data block in the added data block is the same as the fingerprint of the data block in the first snapshot, the first data block is not copied to the disaster tolerant device.
- the first data block is not copied to the disaster tolerant device.
- the master device obtains a second snapshot of the primary device, and the primary device determines whether the fingerprint of the data block added in the second snapshot relative to the first snapshot is the first snapshot.
- the fingerprint of the data block in the same is the same; when the fingerprint of the first data block in the added data block is different from the fingerprint of the data block in the first snapshot, the first data block, the first data is
- the fingerprint of the block and the metadata of the added data block are sent to the disaster recovery end device, and the primary end device and the disaster recovery end device both store the first snapshot, and implement remote replication based on deduplication.
- the fingerprint is not sent to the disaster recovery device for query, the fingerprint query operation is reduced, and the interaction of the network is reduced, that is, the network bandwidth is reduced, the performance of the remote replication disaster recovery backup is improved, and the fingerprint in the prior art is solved.
- the query operation is too cumbersome, and the fingerprint query process increases network interaction and occupies more network bandwidth.
- FIG. 3 is a flowchart of Embodiment 2 of a remote copying method based on deduplication according to the present invention.
- the primary end device determines whether a fingerprint of a data block added in the second snapshot relative to the first snapshot is related to a data block in the first snapshot.
- the fingerprints are the same, including:
- the fingerprint of the first data block is The fingerprint of the data block in the first snapshot is different, and the fingerprint of the data block except the first data block in the added data block is the same as the fingerprint of the data block in the first snapshot.
- the master device determines whether the fingerprint of the data block that is added to the first snapshot in the second snapshot is the same as the fingerprint of the data block in the first snapshot, and includes:
- the primary device acquires a difference record; the difference record is used to record the added data block;
- the master device determines whether the fingerprint of the added data block recorded in the difference record is the same as the fingerprint of the data block in the first snapshot.
- the primary device needs to create a snapshot to copy the data of the snapshot of the primary device to the DR device.
- the disaster recovery device will also create a snapshot when replication is complete.
- the snapshot data of the DR device is the same as the snapshot data of the master device.
- the data that is copied each time is the difference data between the snapshot that was copied and the snapshot that was previously copied.
- the fingerprint of the data to be copied exists in the last snapshot, the fingerprint of the data must exist on the DR device.
- a second snapshot of the data to be copied is first created, and then a first snapshot of the copied data is obtained, and the copied data is the previously copied data, and the second snapshot and the first a difference record between snapshots; the difference record is used to record the added data block; as shown in FIG. 2, the difference record is metadata 21, metadata 22, metadata 23, and the difference record is obtained a fingerprint of the recorded data block; the fingerprint is a fingerprint 2, a fingerprint 3, and a fingerprint 4; determining whether the fingerprint of the data block recorded by the difference record is the same as the fingerprint of the data block in the first snapshot, that is, determining the fingerprint 2 Whether the fingerprint 3 and the fingerprint 4 are the same as the fingerprint of the data block in the first snapshot. In this embodiment, it is determined whether the fingerprint of the data block recorded by the difference record is the same as the fingerprint of the data block in the first snapshot. Judging by the transaction number, the specific process is as follows:
- the second transaction number of the fingerprint corresponds to a first transaction number
- the fingerprint of each data block in the first snapshot also corresponds to a transaction number
- the transaction number corresponding to the fingerprint may be equal to the first transaction number
- the fingerprint of each data block in the added data block in the second snapshot corresponds to a second transaction number, and the corresponding first transaction number of the first snapshot is 1, the fingerprint of the first snapshot, and the fingerprint 2
- the transaction number corresponding to the fingerprint 3 is also 1; the second transaction number corresponding to the fingerprint 2 of the second snapshot is 1, the second transaction number corresponding to the fingerprint 3 is 1, and the second transaction number corresponding to the fingerprint 4 is 2, then the second The fingerprint of the snapshot 2, the second transaction number of the fingerprint 3 is equal to the first transaction number of the first snapshot, the fingerprint 2 of the second snapshot, the fingerprint 3 of the second snapshot, the fingerprint 2 of the first snapshot, the fingerprint 3, and the fingerprint 2 of the second snapshot.
- the data block 2 corresponding to the fingerprint 3 and the data block 3 need not be transmitted to the disaster recovery end device.
- the second transaction number of the fingerprint 4 of the second snapshot is greater than the first transaction number corresponding to the first snapshot, and the fingerprint of the second snapshot is 4 The fingerprint is different from the first snapshot, so the fingerprint 4 is required.
- the first data block, the metadata 23 of the first data block, and the fingerprint 4 are transmitted to the disaster recovery end device, and the metadata 21 and the metadata 22 corresponding to the data block 2 and the data block 3 are also transmitted to the disaster recovery end device. .
- a transaction is a program execution unit that accesses and possibly updates data. Transactions are usually caused by the execution of a user program written in a high-level programming language (such as SQL, C++, or Java) and are defined by shapes such as begin transaction and end transaction statements (or function calls). A transaction consists of the entire operation performed between a begin transaction and an end transaction. Everything corresponds to a number, the transaction number. A transaction has four attributes: atomicity, consistency, isolation, and persistence.
- FIG. 4 is a first flowchart of Embodiment 3 of a remote replication method based on deduplication according to the present invention.
- FIG. 5 is a second flowchart of Embodiment 3 of a remote replication method based on deduplication.
- the method in this embodiment is applied to a remote replication system, where the system includes: a primary device and a disaster recovery device, and the execution entity may be a primary device.
- the primary end device and the disaster recovery end device both store a first snapshot, and the method includes:
- Step 401 The primary device acquires a second snapshot of the primary device.
- Step 402 The master device acquires a difference record, where the difference record is used to record a data block that is added to the first snapshot in the second snapshot.
- Step 403 The primary device acquires a fingerprint of the added data block.
- Step 404 The primary device sends the fingerprint of the added data block to the disaster recovery end device, so that the disaster recovery end device determines whether the fingerprint of the added data block already exists in the Disaster recovery equipment;
- Step 405 The primary end device receives the check information sent by the disaster tolerant device, where the check information includes the first of the added data blocks different from the fingerprint of the data block in the first snapshot. a fingerprint of a data block;
- Step 406 The primary device sends the metadata of the first data block and the added data block indicated by the check information to the disaster recovery end device according to the check information.
- the primary device needs to create a second snapshot of the data to be copied, and then obtain a first snapshot of the copied data, where the primary device and the disaster recovery device store the first snapshot, which is copied.
- the data is the previously copied data, and the difference record between the second snapshot and the first snapshot is obtained; the difference record is used to record the data block added to the first snapshot in the second snapshot;
- the difference is recorded as metadata 21, metadata 22, and metadata 23, and a fingerprint of the data block of the difference record is acquired; the fingerprint is a fingerprint 2, a fingerprint 3, and a fingerprint 4;
- the fingerprint is sent to the disaster recovery device, and the disaster recovery device checks the fingerprint sent by the primary device.
- the data block of the difference record is considered to exist in the disaster recovery device. If the fingerprint does not exist in the disaster-tolerant device, the data block of the difference record does not exist in the disaster-tolerant device, and the master device needs to send the data block of the difference record to the disaster-tolerant device.
- the disaster recovery end device sends the check information to the primary device, where the check information includes a fingerprint of the first data block in the increased data block different from the fingerprint of the data block in the first snapshot.
- the primary end device receives the check information, and sends the first data block indicated by the check information and the metadata of the added data block to the disaster recovery end device, that is, The data block 4 corresponding to the fingerprint 4 is sent to the disaster recovery end device, and the transmitted data further includes the metadata 23 of the data block 4, the fingerprint 4, and the metadata 21 and the metadata 22 corresponding to the other fingerprints.
- the disaster recovery device After receiving the above data, the disaster recovery device writes the data and creates a snapshot after the replication is completed.
- the master device acquires the fingerprint of the added data block, including:
- the master device acquires a fingerprint of the added data block by using a hardware acceleration card.
- the solution of the embodiment can implement remote copying based on deduplication by the method of calculating the fingerprint on the primary end by the hardware acceleration card.
- the second snapshot of the primary device is obtained by the primary device; the primary device obtains a difference record; and the difference record is used to record an increase in the second snapshot relative to the first snapshot.
- the device determines whether the fingerprint of the added data block is already present in the disaster tolerant device; the primary device receives the check information sent by the disaster tolerant device, and the check information includes the first
- the fingerprint of the data block in the snapshot is different from the fingerprint of the first data block in the increased data block; the primary end device, according to the check information, the first data block indicated by the check information Transmitting the metadata of the added data block to the disaster tolerant device, implementing remote replication based on deduplication when the primary end does not have a data deduplication function, and the primary end calculates the fingerprint by using the hardware acceleration card.
- FIG. 6 is a schematic structural diagram of Embodiment 1 of a primary device according to the present invention.
- the primary device of the present embodiment includes: an obtaining module 601, a processing module 602, a sending module 603, and a storage module 604;
- the storage module 604 is configured to store the first snapshot; the first snapshot is also stored in the disaster tolerant device;
- the obtaining module 601 is configured to acquire a second snapshot of the primary device.
- the processing module 602 is configured to determine whether a fingerprint of the data block added in the second snapshot relative to the first snapshot is the same as a fingerprint of the data block in the first snapshot.
- the sending module 603 is configured to: when the fingerprint of the first data block in the added data block is different from the fingerprint of the data block in the first snapshot, the first data block, the first data block The fingerprint and the metadata of the added data block are sent to the disaster tolerant device.
- processing module 602 is specifically configured to:
- the fingerprint of the first data block is The fingerprint of the data block in the first snapshot is different, and the fingerprint of the data block except the first data block in the added data block is the same as the fingerprint of the data block in the first snapshot.
- the obtaining module 601 is specifically configured to:
- the difference record is used to record the added data block
- the processing module 602 is specifically configured to:
- processing module 602 is further configured to:
- the first data block in the added data block is the same as the fingerprint of the data block in the first snapshot, the first data block is not copied to the disaster tolerant device.
- the main device of this embodiment can be used to implement the technical solution of the method embodiment shown in FIG. 1 , and the implementation principle and technical effects are similar, and details are not described herein again.
- FIG. 7 is a schematic structural diagram of Embodiment 2 of a primary device according to the present invention.
- the primary device of the present embodiment includes: an obtaining module 701, a sending module 702, a receiving module 703, and a storage module 704;
- the storage module 704 is configured to store the first snapshot; the first snapshot is also stored in the disaster recovery end device; the obtaining module 701 is configured to acquire the second snapshot of the primary end device;
- the obtaining module 701 is further configured to acquire a difference record, where the difference record is used to record a data block that is added to the first snapshot in the second snapshot, and the acquiring module 701 is further configured to acquire the The fingerprint of the data block;
- the sending module 702 is configured to send the fingerprint of the added data block to the disaster-tolerant device, and configured to enable the disaster-tolerant device to determine whether the fingerprint of the added data block is already present in the disaster-tolerant device. ;
- the receiving module 703 is configured to receive the check information sent by the disaster tolerant device, where the check information includes the first data in the added data block that is different from the fingerprint of the data block in the first snapshot.
- Block fingerprint
- the sending module 702 is further configured to send, according to the check information, the first data block indicated by the check information and the metadata of the added data block to the disaster recovery end device.
- the obtaining module 701 is specifically configured to:
- the fingerprint of the added data block is obtained by a hardware acceleration card.
- the main device of this embodiment can be used to implement the technical solution of the method embodiment shown in FIG. 4, and the implementation principle and technical effects are similar, and details are not described herein again.
- FIG. 8 is a schematic structural diagram of Embodiment 3 of a primary end device according to the present invention.
- the primary device in this embodiment is applied to a remote replication system, and the system includes: the primary device and the disaster recovery device, and the primary device and the disaster recovery device are both A first snapshot is stored, the primary device comprising a processor 801 and a memory 802, the processor 801 and the memory 802 being in communication over a bus 803, the processor 801 executing computer instructions in the memory 802 for The method of any of the method embodiments shown in Figure 1 is performed.
- the primary device in this embodiment may be applied to the remote replication system according to the structural diagram shown in FIG. 8.
- the system includes: the primary device and the disaster tolerance.
- An end device wherein the primary end device and the disaster tolerant end device each store a first snapshot, the primary end device includes a processor and a memory, and the processor and the memory communicate by using a bus,
- the processor executes the computer instructions in the memory to perform the method as described in the method embodiment shown in FIG.
- the disclosed apparatus and method may be implemented in other manners.
- the device embodiments described above are merely illustrative.
- the division of the unit or module is only a logical function division.
- there may be another division manner for example, multiple units or modules may be used. Combinations can be integrated into another system, or some features can be ignored or not executed.
- the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or module, and may be electrical, mechanical or otherwise.
- the modules described as separate components may or may not be physically separated.
- the components displayed as modules may or may not be physical modules, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
- the aforementioned program can be stored in a computer readable storage medium.
- the program when executed, performs the steps including the foregoing method embodiments; and the foregoing storage medium includes various media that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
一种基于重复数据删除的远程复制方法及装置。基于重复数据删除的远程复制方法,应用于远程复制***中,所述***包括:主端设备和容灾端设备,所述主端设备和所述容灾端设备均存储第一快照,所述方法包括:主端设备获取所述主端设备的第二快照(101);所述主端设备判断所述第二快照中相对第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同(102);当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹不同,将所述第一数据块、所述第一数据块的指纹和所述增加的数据块的元数据发送给所述容灾端设备(103)。该方法及装置提高了远程复制容灾备份的性能。
Description
本发明实施例涉及计算机技术领域,尤其涉及一种基于重复数据删除的远程复制方法及装置。
重复数据删除是一种数据缩减技术,旨在减少存储***中使用的存储容量。查找不同数据中不同位置的重复可变大小数据块,将这些重复的数据块用指示符取代,从而缩减存储的数据量。高度冗余的数据集(例如备份数据)从数据重复删除技术的获益极大,用户可以实现10比1至50比1的缩减比。远程复制是一种容灾技术,指将主端设备的数据复制到远程的容灾端设备,当主端设备的数据发生损坏时,可以通过容灾端设备将数据恢复出来。
如果将重复数据删除技术运用于远程复制当中,就可以减少数据的传输量,降低带宽的使用和减小复制的窗口。现有技术中基于重复数据删除的远程复制方案,是将主端设备的待复制数据与前一次已复制数据的差异数据的指纹发送到容灾端设备进行查询,容灾端设备加载指纹,查询主端设备的所述差异数据的指纹是否存在于容灾端设备,并将存在于容灾端设备的指纹发送给主端设备,如果指纹在容灾端设备中存在,那么认为所述差异数据存在于容灾端设备,如果指纹在容灾端设备不存在,那么所述差异数据不存在于容灾端设备,需要主端设备将所述差异数据发送到容灾端设备。
然而,现有技术中存在的问题是,指纹查询操作太繁琐,指纹查询过程增加网络交互,占用较多网络带宽。
发明内容
本发明实施例提供一种基于重复数据删除的远程复制方法及装置,以克服现有技术中指纹查询操作太繁琐,指纹查询过程增加网络交互,占用较多网络带宽的问题。
第一方面,本发明实施例提供一种基于重复数据删除的远程复制方法,应用于远程复制***中,所述***包括:主端设备和容灾端设备,所述主端设备和所述容灾端设备均存储第一快照,所述方法包括:
所述主端设备获取所述主端设备的第二快照;
所述主端设备判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同;
当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹不同,将所述第一数据块、所述第一数据块的指纹和所述增加的数据块的元数据发送给所述容灾端设备。
结合第一方面,在第一方面的第一种可能的实现方式中,所述主端设备判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同,包括:
所述主端设备获取所述第一快照的第一事务号以及所述增加的数据块中的每个数据块的指纹的第二事务号;
判断所述第二事务号是否大于所述第一事务号,若所述第一数据块的指纹的第二事务号大于所述第一事务号,则所述第一数据块的指纹与所述第一快照中的数据块的指纹不同,所述增加的数据块中除所述第一数据块外的数据块的指纹与所述第一快照中的数据块的指纹相同。
结合第一方面,在第一方面的第二种可能的实现方式中,所述主端设备判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同,包括:
所述主端设备获取差异记录;所述差异记录用于记录所述增加的数据块;
所述处理模块,具体用于:
判断所述差异记录中记录的所述增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同。
结合第一方面、或第一方面的第一、第二种可能的实现方式,在第一方面的第三种可能的实现方式中,还包括:
当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹相同,则不复制所述第一数据块至所述容灾端设备。
第二方面,本发明实施例提供一种基于重复数据删除的远程复制方法,应用于远程复制***中,所述***包括:主端设备和容灾端设备,所述主端设备和所述容灾端设备均存储第一快照,所述方法包括:
所述主端设备获取所述主端设备的第二快照;
所述主端设备获取差异记录;所述差异记录用于记录所述第二快照中相对所述第一快照增加的数据块;
所述主端设备获取所述增加的数据块的指纹;
所述主端设备将所述增加的数据块的指纹发送给所述容灾端设备,用于使所述容灾端设备判断所述增加的数据块的指纹是否已经存在于所述容灾端设备;
所述主端设备接收所述容灾端设备发送的查重信息,所述查重信息包含与所述第一快照中的数据块的指纹不同的所述增加的数据块中的第一数据块的指纹;
所述主端设备根据所述查重信息,将所述查重信息指示的所述第一数据块和所述增加的数据块的元数据发送给所述容灾端设备。
结合第二方面,在第二方面的第一种可能的实现方式中,所述主端设备获取所述增加的数据块的指纹,包括:
所述主端设备通过硬件加速卡获取所述增加的数据块的指纹。
第三方面,本发明实施例提供一种主端设备,包括:
存储模块,用于存储第一快照;所述第一快照也存储于容灾端设备;
获取模块,用于获取所述主端设备的第二快照;
处理模块,用于判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同;
发送模块,用于当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹不同,将所述第一数据块、所述第一数据块的指纹和所述增加的数据块的元数据发送给所述容灾端设备。
结合第三方面,在第三方面的第一种可能的实现方式中,所述处理模块,具体用于:
获取所述第一快照的第一事务号以及所述增加的数据块中的每个数据块
的指纹的第二事务号;
判断所述第二事务号是否大于所述第一事务号,若所述第一数据块的指纹的第二事务号大于所述第一事务号,则所述第一数据块的指纹与所述第一快照中的数据块的指纹不同,所述增加的数据块中除所述第一数据块外的数据块的指纹与所述第一快照中的数据块的指纹相同。
结合第三方面,在第三方面的第二种可能的实现方式中,所述获取模块,具体用于:
获取差异记录;所述差异记录用于记录所述增加的数据块;
所述处理模块,具体用于:
判断所述差异记录中记录的所述增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同。
结合第三方面、或第三方面的第一、第二种可能的实现方式,在第一方面的第二种可能的实现方式中,所述处理模块,还用于:
当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹相同,则不复制所述第一数据块至所述容灾端设备。
第四方面,本发明实施例提供一种主端设备,包括:
存储模块,用于存储第一快照;所述第一快照也存储于容灾端设备;
获取模块,用于获取所述主端设备的第二快照;
所述获取模块,还用于获取差异记录;所述差异记录用于记录所述第二快照中相对所述第一快照增加的数据块;
所述获取模块,还用于获取所述增加的数据块的指纹;
发送模块,用于将所述增加的数据块的指纹发送给所述容灾端设备,用于使所述容灾端设备判断所述增加的数据块的指纹是否已经存在于所述容灾端设备;
接收模块,用于接收所述容灾端设备发送的查重信息,所述查重信息包含与所述第一快照中的数据块的指纹不同的所述增加的数据块中的第一数据块的指纹;
所述发送模块,还用于根据所述查重信息,将所述查重信息指示的所述第一数据块和所述增加的数据块的元数据发送给所述容灾端设备。
结合第四方面,在第四方面的第一种可能的实现方式中,所述获取模块,具体用于:
通过硬件加速卡获取所述增加的数据块的指纹。
第五方面,本发明实施例提供一种主端设备,应用于远程复制***中,所述***包括:所述主端设备和容灾端设备,所述主端设备和所述容灾端设备均存储第一快照,所述主端设备包括处理器和存储器,所述处理器和所述存储器通过总线通信,所述处理器执行所述存储器中的计算机指令以用于第一方面中任一所述的方法。
第五方面,本发明实施例提供一种主端设备,应用于远程复制***中,所述***包括:所述主端设备和容灾端设备,所述主端设备和所述容灾端设备均存储第一快照,所述主端设备包括处理器和存储器,所述处理器和所述存储器通过总线通信,所述处理器执行所述存储器中的计算机指令以用于第二方面中任一所述的方法。
本发明实施例基于重复数据删除的远程复制方法及装置,通过所述主端设备获取所述主端设备的第二快照;所述主端设备判断所述第二快照中相对第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同;当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹不同,将所述第一数据块、所述第一数据块的指纹和所述增加的数据块的元数据发送给所述容灾端设备,所述主端设备和所述容灾端设备均存储第一快照,实现了基于重复数据删除的远程复制,且复制中不需要将指纹发送给容灾端设备进行查询,减少指纹查询操作,以及网络的交互,即减少网络带宽的占用,提高了远程复制容灾备份的性能,解决了现有技术中指纹查询操作太繁琐,指纹查询过程增加网络交互,占用较多网络带宽的问题。
为了更清楚地说明本发明实施例中的技术方案,下面将对实施例或现有技术描述中所需要使用的附图作一简单地介绍,显而易见地,下面描述中的附图是本发明的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动性的前提下,还可以根据这些附图获得其他的附图。
图1为本发明基于重复数据删除的远程复制方法实施例一的流程图;
图2为本发明方法实施例的远程复制数据的示意图;
图3为本发明基于重复数据删除的远程复制方法实施例二的流程图;
图4为本发明基于重复数据删除的远程复制方法实施例三的流程图一;
图5为本发明基于重复数据删除的远程复制方法实施例三的流程图二;
图6为本发明主端设备实施例一的结构示意图;
图7为本发明主端设备实施例二的结构示意图;
图8为本发明主端设备实施例三的结构示意图。
为使本发明实施例的目的、技术方案和优点更加清楚,下面将结合本发明实施例中的附图,对本发明实施例中的技术方案进行清楚地描述,显然,所描述的实施例是本发明一部分实施例,而不是全部的实施例。基于本发明中的实施例,本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例,都属于本发明保护的范围。
图1为本发明基于重复数据删除的远程复制方法实施例一的流程图。图2为本发明方法实施例的远程复制数据的示意图。本实施例的方法可应用于远程复制***中,所述***包括:主端设备和容灾端设备,本实施例的执行主体可以为主端设备。如图1所示,本实施例的方法中,所述主端设备和所述容灾端设备均存储第一快照,所述方法包括:
步骤101、所述主端设备获取所述主端设备的第二快照。
步骤102、所述主端设备判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同。
步骤103、当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹不同,将所述第一数据块、所述第一数据块的指纹和所述增加的数据块的元数据发送给所述容灾端设备。
具体来说,首先所述主端设备需创建待复制数据的第二快照,然后获取已复制数据的第一快照,所述主端设备和所述容灾端设备均存储第一快照,已复制数据为前一次复制的数据,进一步还需判断所述第二快照中相对所述第一快
照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同,这里第一快照和第二快照可以分别都有多个指纹,若所述第二快照中相对所述第一快照增加的数据块的指纹与所述第一快照中的数据块的指纹不同,将所述数据块记为第一数据块,则将所述第一数据块、所述第一数据块的指纹和所述增加的数据块的元数据发送给所述容灾端设备,元数据包含数据块的指纹索引即在数据块中的位置、数据块的存储位置、数据块的大小、标识等描述信息。
如图2所示,第一快照已复制到容灾端设备中,若所述第二快照中增加的数据块的指纹与所述第一快照中的数据块的指纹不同,即第二快照的指纹4,则将指纹4对应的数据块4、所述数据块4的元数据23和指纹4、以及所述第二快照中指纹2对应的数据2的元数据21、所述第二快照中指纹3对应的数据3的元数据22,即图2中虚线框内的数据发送给容灾端设备,容灾端设备接收到上述数据后,写入数据,并在复制完成后创建一个快照。
可选地,本实施例的方法,还包括:
当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹相同,则不复制所述第一数据块至所述容灾端设备。
具体来说,若所述第二快照中所有增加的数据块的指纹都与所述第一快照中的数据块的指纹相同,则不复制所述第一数据块至所述容灾端设备。
本实施例,通过所述主端设备获取所述主端设备的第二快照;所述主端设备判断所述第二快照中相对第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同;当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹不同,将所述第一数据块、所述第一数据块的指纹和所述增加的数据块的元数据发送给所述容灾端设备,所述主端设备和所述容灾端设备均存储第一快照,实现了基于重复数据删除的远程复制,且复制中不需要将指纹发送给容灾端设备进行查询,减少指纹查询操作,以及网络的交互,即减少网络带宽的占用,提高了远程复制容灾备份的性能,解决了现有技术中指纹查询操作太繁琐,指纹查询过程增加网络交互,占用较多网络带宽的问题。
图3为本发明基于重复数据删除的远程复制方法实施例二的流程图。如图3所示,在本实施例的方法中,所述主端设备判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同,包括:
所述主端设备获取所述第一快照的第一事务号以及所述增加的数据块中的每个数据块的指纹的第二事务号;
判断所述第二事务号是否大于所述第一事务号,若所述第一数据块的指纹的第二事务号大于所述第一事务号,则所述第一数据块的指纹与所述第一快照中的数据块的指纹不同,所述增加的数据块中除所述第一数据块外的数据块的指纹与所述第一快照中的数据块的指纹相同。
可选地,所述主端设备判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同,包括:
所述主端设备获取差异记录;所述差异记录用于记录所述增加的数据块;
所述主端设备判断所述差异记录中记录的所述增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同。
具体来说,初始复制时,主端设备需要创建一个快照,将主端设备的快照的数据复制到容灾端设备。复制完成时容灾端设备也将创建一个快照。复制完成后,容灾端设备的快照数据和主端设备的快照数据一致,以后每一次复制的数据都为本次复制的快照与前一次复制的快照之间的差异数据。
因此如果当前复制的数据,已存在于上一次复制的快照中,那么当前复制的数据一定存在于容灾端设备。
当主端设备和容灾端设备开启重复数据删除后,如果需要复制数据的指纹存在于上一次复制的快照中,那么该数据的指纹必然存在于容灾端设备。
如图2、图3所示,首先需创建待复制数据的第二快照,然后获取已复制数据的第一快照,已复制数据为前一次复制的数据,获取所述第二快照和所述第一快照之间的差异记录;所述差异记录用于记录所述增加的数据块;如图2所示,所述差异记录为元数据21、元数据22、元数据23,获取所述差异记录记录的数据块的指纹;所述指纹为指纹2、指纹3、指纹4;判断所述差异记录记录的数据块的指纹是否与所述第一快照中的数据块的指纹相同,即判断指纹2、指纹3、指纹4是否与所述第一快照中的数据块的指纹相同,在本实施例中判断差异记录记录的数据块的指纹是否与所述第一快照中的数据块的指纹相同可以通过事务号的方式判断,具体过程如下:
获取所述第一快照的第一事务号以及所述增加的数据块中的每个数据块
的指纹的第二事务号;第一快照对应一个第一事务号,第一快照中每个数据块的指纹也对应一个事务号,所述指纹对应的事务号可能等于所述第一事务号,第二快照中所述增加的数据块中的每个数据块的指纹都对应一个第二事务号,假设第一快照的对应的第一事务号为1,第一快照的指纹1、指纹2、指纹3对应的事务号也为1;第二快照的指纹2对应的第二事务号为1,指纹3对应的第二事务号为1,指纹4对应的第二事务号为2,那么第二快照的指纹2、指纹3的第二事务号等于第一快照的第一事务号,则第二快照的指纹2、指纹3与第一快照的指纹2、指纹3相同,第二快照的指纹2、指纹3对应的数据块2、数据块3不需要传输到容灾端设备,第二快照的指纹4的第二事务号大于第一快照对应的第一事务号,则第二快照的指纹4与第一快照的指纹都不相同,因此需将指纹4对应的第一数据块、该第一数据块的元数据23、指纹4传输到容灾端设备,数据块2、数据块3对应的元数据21、元数据22也需传输到容灾端设备。
事务(Transaction)是访问并可能更新数据的一个程序执行单元(unit)。事务通常由高级数编程语言(如SQL,C++或Java)书写的用户程序的执行所引起,并用形如begin transaction和end transaction语句(或函数调用)来界定。事务由事务开始(begin transaction)和事务结束(end transaction)之间执行的全体操作组成。每个事物都对应一个编号,即事务号。事务具有4个属性:原子性、一致性、隔离性、持久性。
图4为本发明基于重复数据删除的远程复制方法实施例三的流程图一,图5为本发明基于重复数据删除的远程复制方法实施例三的流程图二。本实施例的方法应用于远程复制***中,所述***包括:主端设备和容灾端设备,执行主体可以为主端设备。如图4、图5所示,本实施例的方法中,所述主端设备和所述容灾端设备均存储第一快照,所述方法包括:
步骤401、所述主端设备获取所述主端设备的第二快照;
步骤402、所述主端设备获取差异记录;所述差异记录用于记录所述第二快照中相对所述第一快照增加的数据块;
步骤403、所述主端设备获取所述增加的数据块的指纹;
步骤404、所述主端设备将所述增加的数据块的指纹发送给所述容灾端设备,用于使所述容灾端设备判断所述增加的数据块的指纹是否已经存在于所述
容灾端设备;
步骤405、所述主端设备接收所述容灾端设备发送的查重信息,所述查重信息包含与所述第一快照中的数据块的指纹不同的所述增加的数据块中的第一数据块的指纹;
步骤406、所述主端设备根据所述查重信息,将所述查重信息指示的所述第一数据块和所述增加的数据块的元数据发送给所述容灾端设备。
具体来说,首先所述主端设备需创建待复制数据的第二快照,然后获取已复制数据的第一快照,所述主端设备和所述容灾端设备均存储第一快照,已复制数据为前一次复制的数据,获取所述第二快照和所述第一快照之间的差异记录;所述差异记录用于记录所述第二快照中相对所述第一快照增加的数据块;如图2所示,所述差异记录为元数据21、元数据22、元数据23,获取所述差异记录记录的数据块的指纹;所述指纹为指纹2、指纹3、指纹4;将所述指纹发送到容灾端设备,容灾端设备对主端设备发送来的指纹进行查重,如果指纹在容灾端设备中存在,那么认为这些差异记录记录的数据块存在于容灾端设备中,如果指纹在容灾端设备中不存在,那么这些差异记录记录的数据块不存在于容灾端设备,需要主端设备将所述差异记录记录的数据块发送到容灾端设备;容灾端设备将查重信息发送给主端设备,所述查重信息包含与所述第一快照中的数据块的指纹不同的所述增加的数据块中的第一数据块的指纹,即指纹4,主端设备接收到所述查重信息,将所述查重信息指示的所述第一数据块和所述增加的数据块的元数据发送给所述容灾端设备,即将指纹4对应的数据块4发送到容灾端设备,发送的数据还包括数据块4的元数据23、指纹4,以及其他指纹对应的元数据21、元数据22。容灾端设备接收到上述数据后,写入数据,并在复制完成后创建一个快照。
可选地,所述主端设备获取所述增加的数据块的指纹,包括:
所述主端设备通过硬件加速卡获取所述增加的数据块的指纹。
具体来说,在主端设备的主存中,由于主业务对中央处理器CPU时延的要求,主业务很有可能没有重删功能,或者手动/自动地关闭了重删功能。因此,本实施例的方案可通过硬件加速卡在主端计算指纹的方法实现基于重复数据删除的远程复制。
本实施例,通过所述主端设备获取所述主端设备的第二快照;所述主端设备获取差异记录;所述差异记录用于记录所述第二快照中相对所述第一快照增加的数据块;所述主端设备获取所述增加的数据块的指纹;所述主端设备将所述增加的数据块的指纹发送给所述容灾端设备,用于使所述容灾端设备判断所述增加的数据块的指纹是否已经存在于所述容灾端设备;所述主端设备接收所述容灾端设备发送的查重信息,所述查重信息包含与所述第一快照中的数据块的指纹不同的所述增加的数据块中的第一数据块的指纹;所述主端设备根据所述查重信息,将所述查重信息指示的所述第一数据块和所述增加的数据块的元数据发送给所述容灾端设备,实现了当主端没有重复数据删除功能时的基于重复数据删除的远程复制,主端通过在硬件加速卡中计算指纹,将指纹发送给容灾端设备进行查重,根据查重信息中指示的新增数据块的指纹,将所述新增数据块发送给容灾端设备,提高了远程复制容灾备份的性能。
图6为本发明主端设备实施例一的结构示意图。如图6所示,本实施例的主端设备,包括:获取模块601、处理模块602、发送模块603和存储模块604;
其中,存储模块604,用于存储第一快照;所述第一快照也存储于容灾端设备;
获取模块601,用于获取所述主端设备的第二快照;
处理模块602,用于判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同;
发送模块603,用于当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹不同,将所述第一数据块、所述第一数据块的指纹和所述增加的数据块的元数据发送给所述容灾端设备。
可选地,所述处理模块602,具体用于:
获取所述第一快照的第一事务号以及所述增加的数据块中的每个数据块的指纹的第二事务号;
判断所述第二事务号是否大于所述第一事务号,若所述第一数据块的指纹的第二事务号大于所述第一事务号,则所述第一数据块的指纹与所述第一快照中的数据块的指纹不同,所述增加的数据块中除所述第一数据块外的数据块的指纹与所述第一快照中的数据块的指纹相同。
可选地,所述获取模块601,具体用于:
获取差异记录;所述差异记录用于记录所述增加的数据块;
所述处理模块602,具体用于:
判断所述差异记录中记录的所述增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同。
可选地,所述处理模块602,还用于:
当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹相同,则不复制所述第一数据块至所述容灾端设备。
本实施例的主端设备,可以用于执行图1所示方法实施例的技术方案,其实现原理和技术效果类似,此处不再赘述。
图7为本发明主端设备实施例二的结构示意图。如图7所示,本实施例的主端设备,包括:获取模块701、发送模块702、接收模块703和存储模块704;
其中,存储模块704,用于存储第一快照;所述第一快照也存储于容灾端设备;获取模块701,用于获取所述主端设备的第二快照;
所述获取模块701,还用于获取差异记录;所述差异记录用于记录所述第二快照中相对所述第一快照增加的数据块;所述获取模块701,还用于获取所述增加的数据块的指纹;
发送模块702,将所述增加的数据块的指纹发送给所述容灾端设备,用于使所述容灾端设备判断所述增加的数据块的指纹是否已经存在于所述容灾端设备;
接收模块703,用于接收所述容灾端设备发送的查重信息,所述查重信息包含与所述第一快照中的数据块的指纹不同的所述增加的数据块中的第一数据块的指纹;
所述发送模块702,还用于根据所述查重信息,将所述查重信息指示的所述第一数据块和所述增加的数据块的元数据发送给所述容灾端设备。
可选地,所述获取模块701,具体用于:
通过硬件加速卡获取所述增加的数据块的指纹。
本实施例的主端设备,可以用于执行图4所示方法实施例的技术方案,其实现原理和技术效果类似,此处不再赘述。
图8为本发明主端设备实施例三的结构示意图。如图8所示,本实施例的主端设备,应用于远程复制***中,所述***包括:所述主端设备和容灾端设备,所述主端设备和所述容灾端设备均存储第一快照,所述主端设备包括处理器801和存储器802,所述处理器801和所述存储器802通过总线803通信,所述处理器801执行所述存储器802中的计算机指令以用于执行如图1所示方法实施例中任一所述的方法。
在本发明主端设备实施例四中,本实施例的主端设备,可以采用如图8所示的结构示意图,应用于远程复制***中,所述***包括:所述主端设备和容灾端设备,其特征在于,所述主端设备和所述容灾端设备均存储第一快照,所述主端设备包括处理器和存储器,所述处理器和所述存储器通过总线通信,所述处理器执行所述存储器中的计算机指令以执行如图4所示方法实施例中所述的方法。
在本申请所提供的几个实施例中,应该理解到,所揭露的设备和方法,可以通过其它的方式实现。例如,以上所描述的设备实施例仅仅是示意性的,例如,所述单元或模块的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,例如多个单元或模块可以结合或者可以集成到另一个***,或一些特征可以忽略,或不执行。另一点,所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口,设备或模块的间接耦合或通信连接,可以是电性,机械或其它的形式。
所述作为分离部件说明的模块可以是或者也可以不是物理上分开的,作为模块显示的部件可以是或者也可以不是物理模块,即可以位于一个地方,或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
本领域普通技术人员可以理解:实现上述各方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成。前述的程序可以存储于一计算机可读取存储介质中。该程序在执行时,执行包括上述各方法实施例的步骤;而前述的存储介质包括:ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是:以上各实施例仅用以说明本发明的技术方案,而非对其限制;尽管参照前述各实施例对本发明进行了详细的说明,本领域的普通技术
人员应当理解:其依然可以对前述各实施例所记载的技术方案进行修改,或者对其中部分或者全部技术特征进行等同替换;而这些修改或者替换,并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。
Claims (14)
- 一种基于重复数据删除的远程复制方法,应用于远程复制***中,所述***包括:主端设备和容灾端设备,其特征在于,所述主端设备和所述容灾端设备均存储第一快照,所述方法包括:所述主端设备获取所述主端设备的第二快照;所述主端设备判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同;当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹不同,将所述第一数据块、所述第一数据块的指纹和所述增加的数据块的元数据发送给所述容灾端设备。
- 根据权利要求1所述的方法,其特征在于,所述主端设备判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同,包括:所述主端设备获取所述第一快照的第一事务号以及所述增加的数据块中的每个数据块的指纹的第二事务号;判断所述第二事务号是否大于所述第一事务号,若所述第一数据块的指纹的第二事务号大于所述第一事务号,则所述第一数据块的指纹与所述第一快照中的数据块的指纹不同,所述增加的数据块中除所述第一数据块外的数据块的指纹与所述第一快照中的数据块的指纹相同。
- 根据权利要求1所述的方法,其特征在于,所述主端设备判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同,包括:所述主端设备获取差异记录;所述差异记录用于记录所述增加的数据块;所述主端设备判断所述差异记录中记录的所述增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同。
- 根据权利要求1至3任一项所述的方法,其特征在于,还包括:当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹相同,则不复制所述第一数据块至所述容灾端设备。
- 一种基于重复数据删除的远程复制方法,应用于远程复制***中,所述***包括:主端设备和容灾端设备,其特征在于,所述主端设备和所述容灾 端设备均存储第一快照,所述方法包括:所述主端设备获取所述主端设备的第二快照;所述主端设备获取差异记录;所述差异记录用于记录所述第二快照中相对所述第一快照增加的数据块;所述主端设备获取所述增加的数据块的指纹;所述主端设备将所述增加的数据块的指纹发送给所述容灾端设备,用于使所述容灾端设备判断所述增加的数据块的指纹是否已经存在于所述容灾端设备;所述主端设备接收所述容灾端设备发送的查重信息,所述查重信息包含与所述第一快照中的数据块的指纹不同的所述增加的数据块中的第一数据块的指纹;所述主端设备根据所述查重信息,将所述查重信息指示的所述第一数据块和所述增加的数据块的元数据发送给所述容灾端设备。
- 根据权利要求5所述的方法,其特征在于,所述主端设备获取所述增加的数据块的指纹,包括:所述主端设备通过硬件加速卡获取所述增加的数据块的指纹。
- 一种主端设备,其特征在于,包括:存储模块,用于存储第一快照;所述第一快照也存储于容灾端设备;获取模块,用于获取所述主端设备的第二快照;处理模块,用于判断所述第二快照中相对所述第一快照增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同;发送模块,用于当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹不同,将所述第一数据块、所述第一数据块的指纹和所述增加的数据块的元数据发送给所述容灾端设备。
- 根据权利要求7所述的主端设备,其特征在于,所述处理模块,具体用于:获取所述第一快照的第一事务号以及所述增加的数据块中的每个数据块的指纹的第二事务号;判断所述第二事务号是否大于所述第一事务号,若所述第一数据块的指纹 的第二事务号大于所述第一事务号,则所述第一数据块的指纹与所述第一快照中的数据块的指纹不同,所述增加的数据块中除所述第一数据块外的数据块的指纹与所述第一快照中的数据块的指纹相同。
- 根据权利要求7所述的主端设备,其特征在于,所述获取模块,具体用于:获取差异记录;所述差异记录用于记录所述增加的数据块;所述处理模块,具体用于:判断所述差异记录中记录的所述增加的数据块的指纹是否与所述第一快照中的数据块的指纹相同。
- 根据权利要求7至9任一项所述的主端设备,其特征在于,所述处理模块,还用于:当所述增加的数据块中的第一数据块的指纹与所述第一快照中的数据块的指纹相同,则不复制所述第一数据块至所述容灾端设备。
- 一种主端设备,其特征在于,包括:存储模块,用于存储第一快照;所述第一快照也存储于容灾端设备;获取模块,用于获取所述主端设备的第二快照;所述获取模块,还用于获取差异记录;所述差异记录用于记录所述第二快照中相对所述第一快照增加的数据块;所述获取模块,还用于获取所述增加的数据块的指纹;发送模块,用于将所述增加的数据块的指纹发送给所述容灾端设备,用于使所述容灾端设备判断所述增加的数据块的指纹是否已经存在于所述容灾端设备;接收模块,用于接收所述容灾端设备发送的查重信息,所述查重信息包含与所述第一快照中的数据块的指纹不同的所述增加的数据块中的第一数据块的指纹;所述发送模块,还用于根据所述查重信息,将所述查重信息指示的所述第一数据块和所述增加的数据块的元数据发送给所述容灾端设备。
- 根据权利要求11所述的主端设备,其特征在于,所述获取模块,具体用于:通过硬件加速卡获取所述增加的数据块的指纹。
- 一种主端设备,其特征在于,应用于远程复制***中,所述***包括:所述主端设备和容灾端设备,所述主端设备和所述容灾端设备均存储第一快照,所述主端设备包括处理器和存储器,所述处理器和所述存储器通过总线通信,所述处理器执行所述存储器中的计算机指令以用于权利要求1-4任一所述的方法。
- 一种主端设备,其特征在于,应用于远程复制***中,所述***包括:所述主端设备和容灾端设备,所述主端设备和所述容灾端设备均存储第一快照,所述主端设备包括处理器和存储器,所述处理器和所述存储器通过总线通信,所述处理器执行所述存储器中的计算机指令以用于权利要求5或6所述的方法。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP15874905.1A EP3200096A4 (en) | 2014-12-31 | 2015-09-18 | Remote replication method and apparatus based on duplicated data deletion |
US15/486,536 US10860539B2 (en) | 2014-12-31 | 2017-04-13 | De-duplication-based remote replication method, and apparatus |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410854706.4 | 2014-12-31 | ||
CN201410854706.4A CN104484480B (zh) | 2014-12-31 | 2014-12-31 | 基于重复数据删除的远程复制方法及装置 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/486,536 Continuation US10860539B2 (en) | 2014-12-31 | 2017-04-13 | De-duplication-based remote replication method, and apparatus |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2016107220A1 true WO2016107220A1 (zh) | 2016-07-07 |
Family
ID=52759021
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2015/089977 WO2016107220A1 (zh) | 2014-12-31 | 2015-09-18 | 基于重复数据删除的远程复制方法及装置 |
Country Status (4)
Country | Link |
---|---|
US (1) | US10860539B2 (zh) |
EP (1) | EP3200096A4 (zh) |
CN (1) | CN104484480B (zh) |
WO (1) | WO2016107220A1 (zh) |
Families Citing this family (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104484480B (zh) * | 2014-12-31 | 2018-06-05 | 华为技术有限公司 | 基于重复数据删除的远程复制方法及装置 |
CN106250265A (zh) * | 2016-07-18 | 2016-12-21 | 乐视控股(北京)有限公司 | 用于对象存储的数据备份方法和*** |
CN107066355A (zh) * | 2017-05-15 | 2017-08-18 | 郑州云海信息技术有限公司 | 一种高可用的备份服务器***及其搭建方法 |
CN109697021A (zh) * | 2017-10-23 | 2019-04-30 | 阿里巴巴集团控股有限公司 | 一种磁盘快照的数据处理方法及装置 |
CN110018986B (zh) * | 2017-11-08 | 2022-10-28 | 阿里巴巴集团控股有限公司 | 异常快照识别方法及装置 |
CN109753228B (zh) * | 2017-11-08 | 2022-08-02 | 阿里巴巴集团控股有限公司 | 快照删除方法、装置及*** |
US10303555B1 (en) * | 2017-12-23 | 2019-05-28 | Rubrik, Inc. | Tagging data for automatic transfer during backups |
CN110399333B (zh) * | 2018-04-20 | 2023-07-18 | 伊姆西Ip控股有限责任公司 | 删除快照的方法、设备和计算机程序产品 |
CN109491832A (zh) * | 2018-09-26 | 2019-03-19 | 华为技术有限公司 | 数据容灾方法与站点 |
CN109684130B (zh) * | 2018-12-12 | 2020-12-22 | 网宿科技股份有限公司 | 一种机房间数据备份的方法及装置 |
CN110417718B (zh) * | 2018-12-27 | 2022-04-05 | 腾讯科技(深圳)有限公司 | 处理网站中的风险数据的方法、装置、设备及存储介质 |
US11579892B2 (en) * | 2019-05-29 | 2023-02-14 | Lg Electronics Inc. | Digital device for performing booting process and control method therefor |
US20210397599A1 (en) * | 2020-06-18 | 2021-12-23 | Netflix, Inc. | Techniques for generating a consistent view of an eventually consistent database |
CN112286457B (zh) * | 2020-10-28 | 2022-08-26 | 杭州宏杉科技股份有限公司 | 对象重删方法、装置、电子设备及机器可读存储介质 |
US20240103984A1 (en) * | 2022-09-26 | 2024-03-28 | Dell Products L.P. | Leveraging backup process metadata for data recovery optimization |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102510340A (zh) * | 2011-10-11 | 2012-06-20 | 浪潮电子信息产业股份有限公司 | 利用普通Internet网络实现异地快速备份的方法 |
CN102624871A (zh) * | 2012-02-03 | 2012-08-01 | 杭州杰唐信息技术有限公司 | 一种基于分布式企业服务总线实现的远程文件同步方法 |
US20130212074A1 (en) * | 2010-08-31 | 2013-08-15 | Nec Corporation | Storage system |
CN103870362A (zh) * | 2014-03-21 | 2014-06-18 | 华为技术有限公司 | 一种数据恢复方法、装置及备份*** |
CN104484480A (zh) * | 2014-12-31 | 2015-04-01 | 华为技术有限公司 | 基于重复数据删除的远程复制方法及装置 |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7412462B2 (en) * | 2000-02-18 | 2008-08-12 | Burnside Acquisition, Llc | Data repository and method for promoting network storage of data |
AU2003214624A1 (en) * | 2002-04-25 | 2003-11-10 | Kashya Israel Ltd. | An apparatus for continuous compression of large volumes of data |
US7243110B2 (en) * | 2004-02-20 | 2007-07-10 | Sand Technology Inc. | Searchable archive |
US8577850B1 (en) * | 2010-11-15 | 2013-11-05 | Symantec Corporation | Techniques for global data deduplication |
US8392384B1 (en) * | 2010-12-10 | 2013-03-05 | Symantec Corporation | Method and system of deduplication-based fingerprint index caching |
US9495379B2 (en) * | 2012-10-08 | 2016-11-15 | Veritas Technologies Llc | Locality aware, two-level fingerprint caching |
-
2014
- 2014-12-31 CN CN201410854706.4A patent/CN104484480B/zh active Active
-
2015
- 2015-09-18 WO PCT/CN2015/089977 patent/WO2016107220A1/zh active Application Filing
- 2015-09-18 EP EP15874905.1A patent/EP3200096A4/en not_active Ceased
-
2017
- 2017-04-13 US US15/486,536 patent/US10860539B2/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130212074A1 (en) * | 2010-08-31 | 2013-08-15 | Nec Corporation | Storage system |
CN102510340A (zh) * | 2011-10-11 | 2012-06-20 | 浪潮电子信息产业股份有限公司 | 利用普通Internet网络实现异地快速备份的方法 |
CN102624871A (zh) * | 2012-02-03 | 2012-08-01 | 杭州杰唐信息技术有限公司 | 一种基于分布式企业服务总线实现的远程文件同步方法 |
CN103870362A (zh) * | 2014-03-21 | 2014-06-18 | 华为技术有限公司 | 一种数据恢复方法、装置及备份*** |
CN104484480A (zh) * | 2014-12-31 | 2015-04-01 | 华为技术有限公司 | 基于重复数据删除的远程复制方法及装置 |
Non-Patent Citations (1)
Title |
---|
See also references of EP3200096A4 * |
Also Published As
Publication number | Publication date |
---|---|
CN104484480B (zh) | 2018-06-05 |
EP3200096A4 (en) | 2017-11-15 |
EP3200096A1 (en) | 2017-08-02 |
CN104484480A (zh) | 2015-04-01 |
US20170235754A1 (en) | 2017-08-17 |
US10860539B2 (en) | 2020-12-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2016107220A1 (zh) | 基于重复数据删除的远程复制方法及装置 | |
EP3062226B1 (en) | Data replication method and storage system | |
US9535800B1 (en) | Concurrent data recovery and input/output processing | |
US9563517B1 (en) | Cloud snapshots | |
US9389800B1 (en) | Synthesizing virtual machine disk backups | |
US9235632B1 (en) | Synchronization of replication | |
US9003138B1 (en) | Read signature command | |
US8463746B2 (en) | Method and system for replicating data | |
US9251230B2 (en) | Exchanging locations of an out of synchronization indicator and a change recording indicator via pointers | |
US10838827B2 (en) | System and method for time parameter based database restoration | |
US11074224B2 (en) | Partitioned data replication | |
CN108351821A (zh) | 数据恢复方法及存储设备 | |
US10725967B1 (en) | Continuous data protection snapshots | |
US10372554B1 (en) | Verification and restore of replicated data using a cloud storing chunks of data and a plurality of hashes | |
US10884886B2 (en) | Copy-on-read process in disaster recovery | |
US9135128B2 (en) | Systems and methods for backing up storage volumes in a storage system | |
US10503426B2 (en) | Efficient space allocation in gathered-write backend change volumes | |
JP2020530155A (ja) | 冗長アレイデータ整列のためのシステムおよび方法 | |
US10007567B2 (en) | Method and device for recovering data replication between storage devices | |
WO2015074260A1 (zh) | 数据同步方法和数据同步*** | |
US20150169220A1 (en) | Storage control device and storage control method | |
US10831611B2 (en) | Method and system for efficient replication of block-based backups | |
US11514002B2 (en) | Indexing splitter for any pit replication | |
US9547651B1 (en) | Establishing file relationships based on file operations | |
US10216596B1 (en) | Fast consistent write in a distributed system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 15874905 Country of ref document: EP Kind code of ref document: A1 |
|
REEP | Request for entry into the european phase |
Ref document number: 2015874905 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2015874905 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |