CN116450739A - Multi-station data synchronization method, system, device and medium - Google Patents
Multi-station data synchronization method, system, device and medium Download PDFInfo
- Publication number
- CN116450739A CN116450739A CN202310416704.6A CN202310416704A CN116450739A CN 116450739 A CN116450739 A CN 116450739A CN 202310416704 A CN202310416704 A CN 202310416704A CN 116450739 A CN116450739 A CN 116450739A
- Authority
- CN
- China
- Prior art keywords
- data
- site
- station
- barrel
- missing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000003860 storage Methods 0.000 claims description 28
- 239000012634 fragment Substances 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims description 12
- 238000005192 partition Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 abstract description 8
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000011084 recovery Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 6
- 230000001360 synchronised effect Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 239000002360 explosive Substances 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a multi-station data synchronization method, a system, a device and a medium, which are applied to the field of data processing. According to the multi-station data synchronization method, whether the stations are missing data is judged by comparing the data information of the stations, when the missing data of the stations is confirmed, the missing data is obtained from other stations and written into the stations missing the data, the purpose of redundancy of the multi-station data is achieved, the data can be recovered through other stations after the data of the stations are lost, the data synchronization of the fault stations and the other stations is guaranteed, and therefore the multi-station synchronization data function is more powerful, and the market competitiveness of distributed object products is improved. The multi-station data synchronization system, the multi-station data synchronization device and the multi-station data synchronization medium have the same beneficial effects.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a method, system, device, and medium for synchronizing multiple site data.
Background
With the explosive growth of massive unstructured data, distributed object storage has become a storage foundation for users to build data center architecture, and more key services are accessed into the distributed object storage. In the scene, the distributed object storage product not only needs to ensure the storage requirement of customers on mass data, but also can meet the disaster recovery backup of key data.
Based on the above requirements, the object storage multi-site is all capable of realizing data synchronization among the multi-sites, and the current multi-site is to synchronize data to one site to achieve a backup effect after writing data in the other site, and when one site fails, the data access is switched to the other site to ensure normal access.
However, the current data synchronization method can only synchronize the data of the main site at the standby site, plays a role in backup, accesses the standby site to acquire the data after the main site fails, and cannot recover the data of the main site with failure.
Disclosure of Invention
The purpose of the application is to provide a multi-station data synchronization method, a system, a device and a medium, which are used for solving the problem that data recovery cannot be performed after a certain or some stations fail under the condition of multiple stations, so that the data synchronization of the failed stations and other stations is ensured, and the market competitiveness of distributed object products is improved.
In order to solve the above technical problems, the present application provides a multi-station data synchronization method, including:
acquiring data information of each site;
judging whether each site lacks data or not by comparing the data information of each site;
if yes, acquiring the missing data through other sites and writing the missing data into the sites;
if not, returning to the step of acquiring the data information of each site.
Preferably, the stations comprise a first station and a second station;
correspondingly, obtaining the data information of each site comprises the following steps:
first data information of a first site and second data information of a second site are acquired.
Preferably, the first data information includes a first bucket list, a first bucket quota, and a first object number, and the second data information includes a second bucket list, a second bucket quota, and a second object number.
Preferably, determining whether each site lacks data by comparing data information of each site includes:
judging whether the first barrel list is consistent with the second barrel list;
if the first barrel list is inconsistent with the second barrel list, determining that the first station or the second station lacks data;
if the first barrel list is consistent with the second barrel list, judging whether the first barrel quota is consistent with the second barrel quota;
if the first barrel quota is inconsistent with the second barrel quota, determining that the first station or the second station lacks data;
and if the first barrel quota is consistent with the second barrel quota, determining that the first station or the second station does not miss data.
Preferably, when the first bucket list is inconsistent with the second bucket list, the site that acquires the missing data and writes the missing data through the other site includes:
creating a first barrel fragment consistent with another site in the site with missing data according to the first barrel list and the second barrel list and synchronizing barrel information;
acquiring an object list on another site barrel fragment;
and writing the object on the other site into the first barrel slice according to the object list.
Preferably, when the first barrel quota is inconsistent with the second barrel quota, the steps of obtaining the missing data by the other stations and writing the missing data to the stations include:
comparing whether the first object number on the barrel slice of the first site is consistent with the second object number on the barrel slice of the second site;
if so, continuously comparing whether the first object number on the barrel slice of the first site is consistent with the second object number on the barrel slice of the second site;
and if the objects are inconsistent, acquiring the objects missing by the sites missing the data according to the first object list of the first site and the second object list of the second site, and writing the objects into the sites missing the data.
Preferably, when the first object number on the bucket fragment of the first site is inconsistent with the second object number on the bucket fragment of the second site, acquiring the missing object of the site missing the data according to the first object list of the first site and the second object list of the second site, and writing the object into the site missing the data further comprises:
and returning to the step of acquiring the first data information of the first site and the second data information of the second site.
In order to solve the above technical problem, the present application further provides a multi-station data synchronization system, including:
the first acquisition module is used for acquiring data information of each site;
the judging module is used for judging whether the stations lack data or not by comparing the data information of the stations;
the second acquisition module is used for acquiring the missing data through other stations and writing the missing data into the stations when the stations are missing data;
and the return module is used for returning to the step of acquiring the data information of each site when the data of each site is not deleted.
In order to solve the technical problem, the application also provides a multi-station data synchronization device, which comprises a memory for storing a computer program;
and a processor for implementing the steps of the multi-site data synchronization method when executing the computer program.
In order to solve the above technical problem, the present application further provides a computer readable storage medium, on which a computer program is stored, where the computer program when executed by a processor implements the steps of the multi-station data synchronization method.
According to the multi-station data synchronization method, whether the stations are missing data is judged by comparing the data information of the stations, when the missing data of the stations is confirmed, the missing data is obtained from other stations and written into the stations missing the data, the purpose of redundancy of the multi-station data is achieved, the data can be recovered through other stations after the data of the stations are lost, the data synchronization of the fault stations and the other stations is guaranteed, and therefore the multi-station synchronization data function is more powerful, and the market competitiveness of distributed object products is improved.
The multi-station data synchronization system, the multi-station data synchronization device and the multi-station data synchronization medium have the same beneficial effects.
Drawings
For a clearer description of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described, it being apparent that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a multi-station data synchronization method according to an embodiment of the present application;
FIG. 2 is a mapping relationship diagram of bucket information, bucket fragments and objects according to an embodiment of the present disclosure;
FIG. 3 is a flowchart of an application of a multi-site data synchronization method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a multi-site data synchronization system according to an embodiment of the present application;
fig. 5 is a block diagram of a multi-site data synchronization apparatus according to another embodiment of the present application.
Detailed Description
The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments herein without making any inventive effort are intended to fall within the scope of the present application.
The core of the application is to provide a multi-site data synchronization method, a system, a device and a medium, which are used for solving the problem that data recovery cannot be carried out after a certain or some sites fail under the condition of multiple sites, thereby guaranteeing the data synchronization of the failed sites and other sites and improving the market competitiveness of distributed object products.
In order to provide a better understanding of the present application, those skilled in the art will now make further details of the present application with reference to the drawings and detailed description.
With the explosive growth of massive unstructured data, distributed object storage has become a storage foundation for users to build data center architecture, and more key services are accessed into the distributed object storage. In the scene, the distributed object storage product not only needs to ensure the storage requirement of customers on mass data, but also can meet the disaster recovery backup of key data.
Disaster recovery backup typically requires the establishment of two data centers: one is a data center and is responsible for daily production operation; the other is a disaster backup center, which is responsible for the operation of the application system after the disaster occurs. The disaster recovery backup comprises the disaster recovery backup in the same city or in the same city with the disaster backup center at a distance of not more than 200 km, and the disaster recovery backup in different places with the distance between the data center and the disaster backup center at a distance of more than 200 km. The disaster recovery backup is generally used for preventing disasters caused by fire disaster, building damage, power failure, computer systems and artificial damage. The backup of disaster recovery in different places can prevent the risks of fire, building damage and the like, such as large rain, as well as war, earthquake, flood and the like, and the backup of financing in the same city and the backup of disaster recovery in different places are long, so that the optimal disaster prevention effect is achieved, and a backup center is usually built in each of the same city and different places. The object storage multi-site aims at realizing data synchronization among the multi-sites, the existing multi-site is to synchronize data to the backup site to achieve a backup effect after the data is written in the master site, and when the master site fails, the data access is switched to the backup site to ensure normal access, so that the original data of the master site cannot be recovered to keep the data synchronization continuously, and only the backup effect is achieved.
Based on the above requirements, the embodiment of the present application provides a multi-station data synchronization method, as shown in fig. 1, and fig. 1 is a flowchart of the multi-station data synchronization method provided in the embodiment of the present application, as shown in fig. 1, including:
s10: and acquiring data information of each site.
Specifically, according to the multi-station data synchronization method provided by the embodiment of the application, a thread is started through a wireless access gateway (RGW), and data information of each station of the distributed object storage system is periodically acquired. The embodiments of the present application do not limit the period, and the default period is generally designed or adopted according to the requirement. It should be noted that, each site refers to a plurality of sites, and because the multi-site data synchronization method provided by the present application can enable the failed site data to be recovered and kept synchronous with other sites, as an optimization, two sites are usually used, but it should be noted that, in the embodiment of the present application, the specific number of sites is not limited, the number is often determined according to the requirement, in addition, the specific content of the data information is not limited, and the data information generally includes a bucket list, a bucket quota, an object number and the like.
S11: and judging whether the stations lack data or not by comparing the data information of the stations.
Specifically, after the RGW starting thread acquires data information, the data information of each site is compared one by one, and when the data information is different, the site with missing data is described. More specifically, when the data information included in other sites is not included in one site, it is indicated that the one site has data missing, when the data information included in one site is not included in other sites, it is indicated that the other sites have missing data, and all the data information is compared, so that the site with missing data and the missing data can be found. It should be noted that, when the data information includes a bucket list, a bucket quota, and the number of objects, the general information list is compared first, then the bucket quota is compared, and finally the number of objects is compared until the site where the data is missing and the missing data information are found.
S12: if yes, acquiring the missing data through other sites and writing the missing data into the sites.
Specifically, when it is confirmed that the data of each site is missing, the data missing is obtained by comparing the data information, which means that at least one site of the other sites includes the missing data of the site with the missing data, so that the missing data is obtained by the other sites, and then the missing data is written into the site with the missing data, so that the data among multiple sites can be kept synchronous.
S13: if not, returning to the step of acquiring the data information of each site.
Specifically, when confirming that no station data is missing in each station, the data information of each station is continuously acquired, so that whether the station data is missing in each station is continuously judged, and the data among the stations is ensured to be kept synchronous.
According to the multi-site data synchronization method provided by the embodiment of the invention, whether the site is missing data is judged by comparing the data information of each site, and when the missing data of the site is confirmed, the missing data is acquired from other sites and written into the site with missing data, so that the purpose of backing up the multi-site data can be achieved, the data can be recovered through other sites after the site data is lost, the data synchronization of the fault site and other sites is ensured, the multi-site synchronization data function is more powerful, and the market competitiveness of the distributed object products is improved.
Based on the above embodiments, the present embodiment is a preferred embodiment, where the stations include a first station and a second station;
correspondingly, obtaining the data information of each site comprises the following steps:
first data information of a first site and second data information of a second site are acquired.
Specifically, since the multi-station data synchronization method provided by the embodiment of the present application can write the missing data by acquiring the data of other stations after the data loss occurs in the stations, in order to reduce the cost of the distributed storage system, the two limited stations are preferably a first station and a second station, and are adapted to each other, and the first data information of the first station and the second data information of the second station are acquired when the data information of each station is acquired, so that whether the data loss exists in the first station and the second station is determined by comparing the first data information and the second data information, thereby ensuring that the data of the first station and the second station remains synchronized.
It should be noted that, the first site and the second site are only used for representing two sites, the first site and the second site are not separated from the main site and the standby site, and the two sites can be started according to the use condition.
According to the multi-station data synchronization method provided by the embodiment of the application, the limiting station comprises the first station and the second station, the cost required by establishing the station is reduced, the purpose of backing up the data can be achieved, the data can be recovered through other stations after the data of the station is lost, the data synchronization of the fault station and other stations is ensured, the multi-station synchronous data function is more powerful, and the market competitiveness of a distributed object product is improved.
Based on the foregoing embodiments, the embodiment of the present application is a preferred embodiment, where the first data information includes a first bucket list, a first bucket quota, and a first object number, and the second data information includes a second bucket list, a second bucket quota, and a second object number.
Specifically, the multi-station data synchronization method provided by the embodiment of the application further limits data information, and the limiting data signals comprise a bucket list, a bucket quota and an object number. Each storage cluster is composed of numerous barrels, each barrel is composed of 1024 full slices, a lot of objects are stored on each barrel slice, the number of the objects on the slice is recorded back on the barrel slice, the sum of the number of the objects on all slices of the barrel is recorded on barrel information, the sum of the number of the objects in the barrel is barrel quota, and as shown in fig. 2, fig. 2 is a mapping relation diagram of barrel information, barrel slices and objects provided by the embodiment of the application. Therefore, the limiting data information comprises a bucket list, a bucket quota and a number of objects, the first data information is a first bucket list, a first bucket quota and a first number of objects, the second data information is a second bucket list, a second bucket quota and a second number of objects, and whether data loss exists in the first site and the second site or not is judged by comparing the first bucket list with the second bucket list, the first bucket quota with the second bucket quota and the first number of objects with the second number of objects respectively.
The multi-station data synchronization method provided by the embodiment of the invention defines that the first data information comprises a first barrel list, a first barrel quota and a first object number, and the second data information comprises a second barrel list, a second barrel quota and a second object number, which are used as the basis for judging whether the first station and the second station have station missing data or not, and can recover data through another station after the station data is lost, so that the data synchronization of the fault station and other stations is ensured.
Based on the foregoing embodiments, the embodiment of the present application is a preferred embodiment, further defining that determining whether each site lacks data by comparing data information of each site includes:
judging whether the first barrel list is consistent with the second barrel list;
if the first barrel list is inconsistent with the second barrel list, determining that the first station or the second station lacks data;
if the first barrel list is consistent with the second barrel list, judging whether the first barrel quota is consistent with the second barrel quota;
if the first barrel quota is inconsistent with the second barrel quota, determining that the first station or the second station lacks data;
and if the first barrel quota is consistent with the second barrel quota, determining that the first station or the second station does not miss data.
Specifically, after the first bucket list and the second bucket list are obtained, judging whether the two bucket lists are consistent, and if the two bucket lists are inconsistent, that is, the bucket fragments have differences, indicating that at least one of the first site and the second site has data missing; if the two bucket lists are consistent, whether the first bucket quota is consistent with the second bucket quota is further compared; when the first barrel quota is inconsistent with the second barrel quota, indicating that at least one of the first site and the second site has data missing; when the first barrel quota is inconsistent with the second barrel quota, indicating that at least one of the first site and the second site has data missing; when the first barrel quota is consistent with the second barrel quota, the first station and the second station are indicated to have no data missing.
According to the multi-station data synchronization method provided by the embodiment of the application, whether the first station and the second station have the basis of station missing data is judged by gradually comparing the first barrel list with the second barrel list and the first barrel quota with the second barrel quota, and the data can be recovered through another station after the station data is lost, so that the data synchronization of the fault station and other stations is ensured.
Based on the above embodiments, the present embodiment is preferable, when the first bucket list is inconsistent with the second bucket list, the site that acquires missing data and writes the missing data through the other site includes:
creating a first barrel fragment consistent with another site in the site with missing data according to the first barrel list and the second barrel list and synchronizing barrel information;
acquiring an object list on another site barrel fragment;
and writing the object on the other site into the first barrel slice according to the object list.
Specifically, taking the case that the first site has data missing as an example, creating a first barrel partition in the first site for synchronizing barrel information with the second site, acquiring an object list on the barrel partition of the second site, and writing more objects on the second site than on the first site into the first barrel partition according to the acquired object list, thereby completing data synchronization of the first site and the second site.
It should be noted that, in the embodiment of the present application, only the case where the first station has data missing is taken as an example, and the method is not limited, and when the second station has data missing, the method is consistent with the processing method when the first station has data missing, and detailed description is omitted again.
According to the multi-station data synchronization method provided by the embodiment of the application, when the first bucket list is inconsistent with the second bucket list, new bucket fragments are established in the stations with missing data so as to write the missing data, missing objects are obtained through the object columns of the bucket fragments on the normal stations, the missing objects are written into the stations with missing data, the purpose of recovering the data of the stations with missing data is achieved, and therefore data synchronization of the faulty stations and other stations is guaranteed.
Based on the foregoing embodiments, the embodiment of the present application is a preferred embodiment, where when the first bucket quota is inconsistent with the second bucket quota, defining a site that acquires missing data and writes the missing data through other sites includes:
comparing whether the first object number on the barrel slice of the first site is consistent with the second object number on the barrel slice of the second site;
if so, continuously comparing whether the first object number on the barrel slice of the first site is consistent with the second object number on the barrel slice of the second site;
and if the objects are inconsistent, acquiring the objects missing by the sites missing the data according to the first object list of the first site and the second object list of the second site, and writing the objects into the sites missing the data.
Specifically, in the case where the first bucket list and the second bucket list are consistent, it is necessary to continuously compare whether the first bucket quota and the second bucket quota are consistent to determine whether there is a data loss in the first station and the second station. After confirming that the first barrel quota is inconsistent with the second barrel quota, comparing the first object number of each barrel partition in the first site with the second object number of each barrel partition in the second site in sequence, and if the object numbers are consistent, continuing to compare the next barrel partition until the object numbers are inconsistent. And when the first object number is inconsistent with the second object number, acquiring the missing object of the site missing the data through the first object list of the first site and the second object list of the second site so as to write the site missing the object. For example, when the first object number is smaller than the second object number, the missing data of the first site is described, the missing data is obtained through the comparison result of the first object list and the second object list, and the missing data is written into the first site.
According to the multi-station data synchronization method provided by the embodiment of the application, when the first barrel quota is inconsistent with the second barrel quota, whether the first object number is consistent with the second object number or not is judged to find the missing object, the missing object is written into the station with missing data, the purpose of recovering the data of the station with missing data is achieved, and therefore data synchronization of the fault station and other stations is guaranteed.
Based on the above embodiments, the present application preferably further includes, when the number of the first objects on the bucket fragments of the first site is inconsistent with the number of the second objects on the bucket fragments of the second site, acquiring, according to the first object list of the first site and the second object list of the second site, the object missing by the site missing the data, and writing the object to the site missing the data:
and returning to the step of acquiring the first data information of the first site and the second data information of the second site.
Specifically, when the number of the first objects on the barrel fragment of the first site is inconsistent with the number of the second objects on the barrel fragment of the second site, acquiring the objects missing from the site missing data according to the first object list of the first site and the second object list of the second site, and returning the objects to the step of acquiring the first data information of the first site and the second data information of the second site after writing the objects into the site missing data, thereby continuously comparing whether the data information of the first site and the second site is consistent or not, and ensuring the data synchronization between the two sites by writing the missing data when the data information of the two sites is inconsistent.
According to the multi-station data synchronization method provided by the embodiment of the application, after the missing data is written into the corresponding station with the missing data, the data information of the two stations is acquired again, so that the data synchronization between the two stations is ensured.
An embodiment of a multi-station data synchronization method is provided, where a station includes a first station and a second station, and fig. 3 is a flowchart of the multi-station data synchronization method provided in the embodiment of the present application, as shown in fig. 3, including:
s20: acquiring a two-station barrel list;
s21: comparing whether the two site bucket lists are consistent;
if the two-station bucket list is inconsistent, step S22 is entered: synchronizing the missing bucket information of the site missing the data and creating a bucket fragment, and proceeding to step S221;
s221: acquiring an object list on barrel fragments of a normal site;
s222: acquiring an object missing from a normal site and writing the object missing from the site missing data into the site missing data;
returning to the step of acquiring the two-station bucket list after executing S212;
if the two bucket lists are identical, step S23 is entered: comparing whether the barrel quota of the two sites is consistent;
if the two-station quota is consistent, returning to the step of comparing whether the two-station quota is consistent;
if the two-station quota is consistent, step S24 is executed: comparing whether the object numbers on the barrel fragments of the two stations are consistent;
if yes, returning to the step of comparing whether the quotas of the two sites are consistent;
if not, go to step S25: comparing object lists on the two-site barrel fragments;
s26: acquiring an object missing from a normal site and writing the object missing from the site missing data into the site missing data;
returning to the step of obtaining the two site bucket lists.
Specifically, a first bucket list of a first site and a second bucket list of a second site are obtained first, whether the first bucket list and the second bucket list are consistent or not is compared, when the first bucket list and the second bucket list are inconsistent, a site with missing data obtains missing information compared with a normal site, bucket information and bucket fragments are created at the site with missing data, an object list in the normal site is obtained, and data of the normal site is written into the site with missing data according to the object list. When the first barrel list is consistent with the second same list, the first barrel quota and the second barrel quota of the first site are sequentially obtained for comparison, when the first barrel quota and the second barrel quota are inconsistent, the first object number of the first site and the second object number of the second site are obtained, whether the first object number and the second object number of the second site are consistent or not is compared, when the object numbers are inconsistent, the object lists of the first site and the second site are obtained and compared to obtain missing data of the site with the corresponding object list of the missing data, and finally the missing object is obtained according to the object list and the site of the missing data is written into the object.
According to the multi-site data synchronization method provided by the embodiment of the invention, whether the site is missing data is judged by comparing the data information of the two sites, and when the missing data of the site is confirmed, the missing data is acquired from other sites and written into the site with missing data, so that the purpose of backing up the multi-site data can be achieved, the data can be recovered through other sites after the site data is lost, the data synchronization of the fault site and other sites is ensured, the multi-site synchronization data function is more powerful, and the market competitiveness of the distributed object products is improved.
In the above embodiments, the multi-site data synchronization method is described in detail, and the present application further provides a corresponding embodiment of the multi-site data synchronization apparatus. It should be noted that the present application describes an embodiment of the device portion from two angles, one based on the angle of the functional module and the other based on the angle of the hardware.
Based on the angle of the functional module, the present application provides a multi-station data synchronization system, and fig. 4 is a schematic diagram of the multi-station data synchronization system provided in the embodiment of the present application, as shown in fig. 4, including:
a first acquiring module 10, configured to acquire data information of each site;
a judging module 11, configured to judge whether each site lacks data by comparing data information of each site;
a second obtaining module 12, configured to obtain, when each site lacks data, the missing data through other sites and write the site lacking the data;
and the return module 13 is used for returning to the step of acquiring the data information of each site when the data of each site is not missing.
Since the embodiments of the system portion and the embodiments of the method portion correspond to each other, the embodiments of the system portion refer to the description of the embodiments of the method portion, which is not repeated herein.
Based on the hardware perspective, the embodiment of the present application provides a multi-station data synchronization device, and fig. 5 is a structural diagram of the multi-station data synchronization device provided in another embodiment of the present application, as shown in fig. 5, the multi-station data synchronization device includes: a memory 20 for storing a computer program;
a processor 21 for implementing the steps of the multi-site data synchronization method as mentioned in the above embodiments when executing a computer program.
The multi-site data synchronization device provided in this embodiment may include, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, or the like.
Processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in hardware in at least one of a digital signal processor (Digital Signal Processor, DSP), a Field programmable gate array (Field-Programmable Gate Array, FPGA), a programmable logic array (Programmable Logic Array, PLA). The processor 21 may also comprise a main processor, which is a processor for processing data in an awake state, also called central processor (Central Processing Unit, CPU), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with an image processor (Graphics Processing Unit, GPU) for taking care of rendering and rendering of the content that the display screen is required to display. In some embodiments, the processor 21 may also include an artificial intelligence (Artificial Intelligence, AI) processor for processing computing operations related to machine learning.
Memory 20 may include one or more computer-readable storage media, which may be non-transitory. Memory 20 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used for storing a computer program 201, where the computer program, when loaded and executed by the processor 21, is capable of implementing the relevant steps of the multi-site data synchronization method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may further include an operating system 202, data 203, and the like, where the storage manner may be transient storage or permanent storage. The operating system 202 may include Windows, unix, linux, among others. The data 203 may include, but is not limited to, data information, and the like.
In some embodiments, the multi-site data synchronization device may further include a display 22, an input/output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.
Those skilled in the art will appreciate that the structure shown in fig. 5 is not limiting of the multi-site data synchronization apparatus and may include more or fewer components than shown.
The multi-station data synchronization device provided by the embodiment of the application comprises a memory and a processor, wherein the processor can realize the following method when executing a program stored in the memory: a multi-station data synchronization method.
Finally, the present application also provides a corresponding embodiment of the computer readable storage medium. The computer-readable storage medium has stored thereon a computer program which, when executed by a processor, performs the steps as described in the method embodiments above.
It will be appreciated that the methods of the above embodiments, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored on a computer readable storage medium. With such understanding, the technical solution of the present application, or a part contributing to the prior art or all or part of the technical solution, may be embodied in the form of a software product stored in a storage medium, performing all or part of the steps of the method described in the various embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The method, the system, the device and the medium for synchronizing the multi-site data provided by the application are described in detail above. In the description, each embodiment is described in a progressive manner, and each embodiment is mainly described by the differences from other embodiments, so that the same similar parts among the embodiments are mutually referred. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section. It should be noted that it would be obvious to those skilled in the art that various improvements and modifications can be made to the present application without departing from the principles of the present application, and such improvements and modifications fall within the scope of the claims of the present application.
It should also be noted that in this specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A method for synchronizing data at multiple sites, comprising:
acquiring data information of each site;
judging whether each site lacks data or not by comparing the data information of each site;
if yes, acquiring the missing data through other sites and writing the missing data into the sites;
and if not, returning to the step of acquiring the data information of each site.
2. The multi-site data synchronization method of claim 1, wherein the sites comprise a first site and a second site;
correspondingly, the obtaining the data information of each site comprises the following steps:
and acquiring first data information of the first station and second data information of the second station.
3. The multi-site data synchronization method of claim 2, wherein the first data information comprises a first bucket list, a first bucket quota, a first number of objects, and the second data information comprises a second bucket list, a second bucket quota, a second number of objects.
4. The multi-site data synchronization method of claim 3, wherein said determining whether each of said sites lacks data by comparing data information of each of said sites comprises:
judging whether the first barrel list is consistent with the second barrel list;
if the first bucket list is inconsistent with the second bucket list, determining that the first station or the second station lacks data;
if the first barrel list is consistent with the second barrel list, judging whether the first barrel quota is consistent with the second barrel quota or not;
if the first barrel quota is inconsistent with the second barrel quota, determining that the first station or the second station lacks data;
and if the first barrel quota is consistent with the second barrel quota, determining that the first station or the second station does not miss data.
5. The multi-site data synchronization method of claim 4, wherein when the first bucket list is inconsistent with the second bucket list, the site that acquires missing data and writes missing data through the other sites comprises:
creating a first barrel fragment consistent with another site in the site where data is missing according to the first barrel list and the second barrel list, and synchronizing barrel information;
acquiring an object list on the barrel partition of the other station;
and writing the object on the other site into a first barrel slice according to the object list.
6. The method of claim 4, wherein when the first bucket quota is inconsistent with the second bucket quota, the station that obtains missing data and writes missing data through the other stations comprises:
comparing whether the first object number on the barrel slice of the first site is consistent with the second object number on the barrel slice of the second site;
if so, continuing to compare whether the first object number on the barrel slice of the first site is consistent with the second object number on the barrel slice of the second site;
and if the object is inconsistent, acquiring the object missing by the station with missing data according to the first object list of the first station and the second object list of the second station, and writing the object into the station with missing data.
7. The multi-site data synchronization method of claim 6, wherein when the first number of objects on the bucket fragments of the first site is inconsistent with the second number of objects on the bucket fragments of the second site, the step of obtaining the missing object of the site missing the data according to the first object list of the first site and the second object list of the second site, and the step of writing the object to the site missing the data further comprises:
and returning to the step of acquiring the first data information of the first station and the second data information of the second station.
8. A multi-site data synchronization system, comprising:
the first acquisition module is used for acquiring data information of each site;
the judging module is used for judging whether the stations lack data or not by comparing the data information of the stations;
the second acquisition module is used for acquiring the missing data through other stations and writing the missing data into the stations when the stations are missing data;
and the return module is used for returning the step of acquiring the data information of each site when the data of each site is not deleted.
9. A multi-site data synchronizing device comprising a memory for storing a computer program;
processor for implementing the steps of the multi-site data synchronization method according to any of claims 1 to 7 when executing said computer program.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the multi-site data synchronization method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310416704.6A CN116450739A (en) | 2023-04-13 | 2023-04-13 | Multi-station data synchronization method, system, device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310416704.6A CN116450739A (en) | 2023-04-13 | 2023-04-13 | Multi-station data synchronization method, system, device and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116450739A true CN116450739A (en) | 2023-07-18 |
Family
ID=87135290
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310416704.6A Pending CN116450739A (en) | 2023-04-13 | 2023-04-13 | Multi-station data synchronization method, system, device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116450739A (en) |
-
2023
- 2023-04-13 CN CN202310416704.6A patent/CN116450739A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3474516B1 (en) | Data processing method and device | |
| US20150213100A1 (en) | Data synchronization method and system | |
| CN111104069A (en) | Multi-region data processing method and device of distributed storage system and electronic equipment | |
| CN111666266A (en) | Data migration method and related equipment | |
| CN112190924A (en) | Data disaster tolerance method, device and computer readable medium | |
| CN104216802B (en) | Memory database recovery method and device | |
| CN109597718A (en) | A kind of disaster recovery platform and a kind of disaster recovery method | |
| CN116389233B (en) | Container cloud management platform active-standby switching system, method and device and computer equipment | |
| CN107526652B (en) | Data synchronization method and storage device | |
| CN112181724A (en) | Big data disaster tolerance method and device and electronic equipment | |
| CN112612851A (en) | Multi-center data synchronization method and device | |
| WO2023240995A1 (en) | Data recovery method and apparatus for dual-machine hot standby system, and medium | |
| CN116450739A (en) | Multi-station data synchronization method, system, device and medium | |
| CN111770139A (en) | Method, device and system for remotely synchronizing data of distributed storage system | |
| CN115174596A (en) | Equipment remote copying method, device and medium | |
| CN116383161A (en) | File synchronization method, device and medium | |
| CN114328007B (en) | Container backup and restoration method, device and medium thereof | |
| CN115509806A (en) | Real-time protection asynchronous remote copying method, device and equipment based on ROW | |
| CN112860376B (en) | Snapshot chain manufacturing method and device, electronic equipment and storage medium | |
| CN115129676A (en) | Data synchronization method, device, equipment and medium | |
| CN110569231B (en) | Data migration method, device, equipment and medium | |
| CN115705269A (en) | Data synchronization method, system, server and storage medium | |
| US9836515B1 (en) | Systems and methods for adding active volumes to existing replication configurations | |
| CN111597255A (en) | Data disaster recovery processing method and device, electronic equipment and storage medium | |
| CN111142795A (en) | Control method, control device and control equipment for write operation of distributed storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |