CN110674214A - Big data synchronization method and device, computer equipment and storage medium - Google Patents
Big data synchronization method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110674214A CN110674214A CN201910841880.8A CN201910841880A CN110674214A CN 110674214 A CN110674214 A CN 110674214A CN 201910841880 A CN201910841880 A CN 201910841880A CN 110674214 A CN110674214 A CN 110674214A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- historical
- synchronization
- reference period
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000001360 synchronised effect Effects 0.000 claims abstract description 46
- 238000012360 testing method Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 description 10
- 230000003203 everyday effect Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000010926 purge Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Development Economics (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Computing Systems (AREA)
- Economics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a big data synchronization method, a big data synchronization device, computer equipment and a storage medium, wherein the method comprises the following steps: responding to a synchronous data acquisition instruction sent by a local terminal, and acquiring the synchronous completion state of a local historical database; wherein the history database is used for performing batch synchronization operation on first history data generated by a remote system during a first reference period; according to the synchronous completion state of the historical database, selecting a local first database or a local second database as a target incremental database; wherein the first database is used for real-time synchronization of the first historical data generated by the remote system during the first reference period, and the second database is used for real-time synchronization of the second historical data generated by the remote system during the second reference period; taking all data in the historical database and the target incremental database as target synchronous data; and sending the target synchronous data to the local terminal.
Description
Technical Field
The present invention relates to the field of data synchronization technologies, and in particular, to a big data synchronization method and apparatus, a computer device, and a storage medium.
Background
From the perspective of a large data development platform, due to reasons such as component technology architecture or service security isolation, various operations or retrieval processing cannot be directly performed on data stored or generated by an online service system, so that the data needs to be collected into various storage and calculation components of the development platform for processing. In this process, data synchronization becomes a very important link.
In the data synchronization business, a large amount of data synchronization tasks are to synchronize data of the previous day near morning to wait for batch analysis. The following problems may often be encountered in such a scenario: the data may not be synchronized due to too large amount of information, e.g., the data synchronization task of the first day is not completed yet at the time the data synchronization task of the second day starts to execute. In this case, the data of the previous day cannot be queried in time for operation, for example, data of an insurance order generated in the insurance industry on the previous day can not be queried.
Therefore, how to solve the problem that the newly added data cannot be obtained in time due to the long time consumption of the synchronization process becomes a technical problem to be solved urgently by those skilled in the art.
Disclosure of Invention
The present invention provides a big data synchronization method, device, computer device and storage medium, so as to solve the above problems in the prior art.
In order to achieve the above object, the present invention provides a big data synchronization method, which comprises the following steps:
responding to a synchronous data acquisition instruction sent by a local terminal, and acquiring the synchronous completion state of a local historical database; wherein the history database is used for performing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion state is used for representing whether the historical database completes batch synchronization of the first historical data;
according to the synchronous completion state of the historical database, selecting a local first database or a local second database as a target incremental database; wherein the first database is used for real-time synchronization of the first historical data generated by the remote system during the first reference period, the second database is used for real-time synchronization of the second historical data generated by the remote system during a second reference period, and the second reference period is a next time period which is continuously arranged with the first reference period;
taking all data in the historical database and the target incremental database as target synchronous data;
and sending the target synchronous data to the local terminal.
According to the big data synchronization method provided by the invention, the step of selecting the first database or the second database as the target incremental database according to the synchronization completion state of the historical database comprises the following steps:
if the historical database completes batch synchronization of the first historical data, selecting the second database as a target incremental database;
and if the historical database does not synchronize the first historical data, selecting the first database as a target incremental database.
According to the big data synchronization method provided by the invention, after the step of selecting the second database as the target incremental database, the method further comprises the following steps:
the data in the first database is emptied and third history data generated by a remote system during a third reference period is synchronized to the first database in real time, wherein the third reference period is a next time period in a consecutive sequence with the second reference period.
According to the big data synchronization method provided by the invention, after the first database is selected as the target incremental database, the method further comprises the following steps:
and emptying the data in the second database, and synchronizing third history data generated by a remote system in a third reference period to the second database in real time, wherein the third reference period is the next time period which is continuously arranged with the second reference period.
According to the big data synchronization method provided by the invention, the step of acquiring the synchronization completion state of the local historical database comprises the following steps:
querying test data from the historical database, wherein the test data comprises a time field corresponding to the first reference period;
if the test data can be inquired from the historical database, the historical database completes batch synchronization of the historical data;
if the test data cannot be inquired from the historical database, the historical database does not complete batch synchronization of the historical data.
In order to achieve the above object, the present invention further provides a big data synchronization apparatus, including:
the state acquisition module is adapted to respond to a synchronous data acquisition instruction sent by a local terminal and acquire the synchronous completion state of a local historical database; wherein the history database is used for performing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion state is used for representing whether the historical database completes batch synchronization of the first historical data;
the increment selection module is suitable for selecting a local first database or a local second database as a target increment database according to the synchronous completion state of the historical database; wherein the first database is used for real-time synchronization of the first historical data generated by the remote system during the first reference period, the second database is used for real-time synchronization of the second historical data generated by the remote system during a second reference period, and the second reference period is a next time period which is continuously arranged with the first reference period;
a data determination module adapted to take all data in both the historical database and the target incremental database as target synchronization data;
and the data sending module is suitable for sending the target synchronous data to the local terminal.
According to the big data synchronization device provided by the invention, the target increment selection module comprises:
the first increment selection module is suitable for selecting the second database as a target increment database if the historical database completes batch synchronization of the first historical data;
and the second increment selection module is suitable for selecting the first database as a target increment database if the historical database is not synchronized with the first historical data.
The big data synchronization device provided by the invention is characterized by further comprising:
a first emptying module adapted to empty the data in the first database and synchronize third history data generated by a remote system during a third reference period to the first database in real time after selecting the second database as a target incremental database, wherein the third reference period is a next time period arranged in series with the second reference period;
and the second emptying module is suitable for emptying the data in the second database after the first database is selected as the target incremental database, and synchronizing third history data generated by the remote system in a third reference period to the second database in real time.
To achieve the above object, the present invention further provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the computer program.
To achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the above method.
The big data synchronization method, the big data synchronization device, the computer equipment and the storage medium provide a rapid and accurate data synchronization scheme for mass data. The present invention bulk-synchronizes first history data generated by a remote system during a first reference period through a history database, while synchronizing the first history data and second history data generated during the first reference period and a second reference period, which is a next time period consecutively arranged with the first reference period, in real time through the first database and the second database, respectively. In consideration of the time delay existing in the data synchronization process, the first data and the second data are correspondingly selected as the target incremental data according to the difference of the synchronization completion conditions of the historical data, so that the historical data and the target incremental data together form non-repeated full data, and the reliable execution of data synchronization is ensured.
Drawings
FIG. 1 is a flowchart of a big data synchronization method according to a first embodiment of the present invention;
FIG. 2 is a block diagram of a first embodiment of a big data synchronization apparatus according to the present invention;
fig. 3 is a schematic diagram of a hardware structure of a big data synchronization apparatus according to a first embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The big data synchronization method, the big data synchronization device, the computer equipment and the storage medium provide a quick and accurate data synchronization scheme for mass data in a remote system. The present invention bulk-synchronizes first history data generated by a remote system during a first reference period through a history database, while synchronizing the first history data and second history data generated during the first reference period and a second reference period, which is a next time period consecutively arranged with the first reference period, in real time through the first database and the second database, respectively. In consideration of the time delay existing in the data synchronization process, the first data and the second data are correspondingly selected as the target incremental data according to the difference of the synchronization completion conditions of the historical data, so that the historical data and the target incremental data together form non-repeated full data, and the reliable execution of data synchronization is ensured.
Example one
Referring to fig. 1, the present embodiment provides a big data synchronization method, which specifically includes the following steps:
s1, responding to the synchronous data acquisition command sent by the local terminal, and acquiring the synchronous completion state of the local historical database; wherein the history database is used for performing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion status is used to characterize whether the historical database has completed batch synchronization of the first historical data.
The invention is applicable to any situation where it is desirable to synchronize data in a remote system. For example, local public security departments at different levels synchronize citizen identity card information in the national public security system, local branch companies of insurance companies need to synchronize client policy data in the national insurance system, and the like. Taking an insurance company as an example, each local branch company can contain a plurality of terminal devices, and when any one of the terminal devices wants to view national customer insurance policy data, the national insurance system does not need to be directly connected, and corresponding national customer insurance policy data is obtained through the local server by sending a synchronous data obtaining instruction to the local server.
In response to a synchronous data acquisition instruction issued by the terminal device, the local server first needs to acquire the synchronization completion status of the local history database. Wherein the history database is used for performing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion status is used to characterize whether the historical database has completed batch synchronization of the first historical data.
The first reference period in the present invention is not a fixed specific time period, but a time period sequence with a regular arrangement characteristic, the first reference period may be set to be every day, every week, every month, etc., and generally the task start time of the first reference period is a fixed time point, for example, zero of every day, every week day is 20 nights; 00, evening of last day of each month 23: 00. The historical database in the invention follows a T-1 mode when the data is synchronized in batches, namely, the data synchronization task of the previous first reference period is executed at the starting time of each synchronization task, for example, the data batch synchronization task of the previous day is executed at every night. Taking the customer insurance policy data in the insurance company as an example, synchronizing the historical data in the remote system to the local historical database is equivalent to synchronizing all the customer insurance policy data generated in the national insurance system from the zero point every day.
It should be noted that, the historical database in the present invention performs a batch data synchronization operation, that is, for batch data generated in a certain time period, only after all data in the batch data is completely executed, operations such as querying, reading, and modifying the batch data in the historical database may be performed. For example, the historical database executes batch synchronization of data generated by the remote system in 1 month and 1 day from the zero point of 1 month and 2 days, the whole synchronization process takes 3 hours, and the synchronization work of the data in 1 month and 1 day is completed at 3 points of 1 month and 2 days, so that the user can query the data in 1 month and 1 day from the 3 points of 1 month and 2 days in the historical database, and the user cannot query the data in 1 month and 1 day before the 3 points of 1 month and 2 days. In summary, the first history data which is not completed in the batch data synchronization process can not be inquired in the history database; the first historical data may be queried in the historical database only after the batch synchronization is fully completed.
For the above reasons, the present invention needs to obtain whether the history database has completed a batch synchronization operation on the first history data in order to determine whether the first history data can be queried from the history database.
S2, selecting a local first database or a local second database as a target incremental database according to the synchronization completion state of the historical database; wherein the first database is used for real-time synchronization of the first historical data generated by the remote system during the first reference period, and the second database is used for real-time synchronization of the second historical data generated by the remote system during a second reference period, and the second reference period is a next time period which is continuously arranged with the first reference period.
Because the synchronized first historical data cannot be inquired before all batch data synchronization is completely executed, in order to ensure that all data in the remote system can be acquired in real time, the invention is provided with two local incremental databases, wherein the first database and the second database are respectively used for storing incremental data generated by the remote system from different moments. Specifically, the first database is used for real-time synchronization of first historical data generated by the remote system during the first reference period, and the second database is used for real-time synchronization of second historical data generated by the remote system during a second reference period, wherein the second reference period is a next time period which is continuously arranged with the first reference period. For example, if the first reference period is 1 month and 1 day, the second reference period is 1 month and 2 days; if the first reference period is 1 month first week, the second reference period is 1 month second week; if the first reference period is 2019, month 1, the second reference period is 2019, month 2, and so on. The first database and the second database in the invention adopt a real-time data synchronization mode, namely any data change in the remote database can be reflected in the first database or the second database in time, and the time for executing the data synchronization operation can be ignored. This is because the real-time data synchronization is performed at short intervals, for example, typically once every ten minutes or once every half hour. During such short intervals, the amount of new data generated by the remote system is small, and thus the synchronization time for performing the synchronization operation is small compared to batch synchronization, which is much shorter and negligible.
The technical personnel in the field can understand that the real-time synchronization needs to constantly detect the data change condition in the remote system at any time, so that more memory resources are occupied, but the real-time synchronization method has the advantages of good real-time performance and capability of timely acquiring the data change condition in the remote system. Trading data integrity by sacrificing some memory resources is more beneficial and less beneficial to the present invention.
At this time, the invention comprises three local databases, namely a history database, a first database and a second database. Assuming that the first reference period is calculated in units of days and the start time is set to zero every day, the historical database performing batch synchronization is set as database a, the first database performing real-time synchronization is set as database B, and the second database performing real-time synchronization is set as database C. Assuming that the database A needs to perform batch synchronization on the data of 1 month and 22 days, the invention synchronizes the data generated by the remote system in 1 month and 22 days in real time through the database B and synchronizes the data generated by the remote system in 1 month and 23 days through the database C.
Database a, database B and database C are also exemplified above. If the current time is 1 month and 23 days, the synchronization of the data of 1 month and 22 days should be started in the database A. If the database A at the current time has finished synchronization, all historical data of 1 month and 22 days and before are stored in the database A, and the data generated from 1 month and 23 are added to form complete full data. According to the arrangement of the present invention, the data generated from day 1 month 23 is recorded in the database C, so the database C becomes the target incremental database at this time.
Consider another case, if the database a at the current time has not completed the synchronization task, then the database a does not contain the data of day 22/month at the current time, but only the data of day 21/month and before. At this time, the data generated from 1 month and 22 days are added to form complete full data. According to the arrangement of the present invention, the data generated from day 1 month 22 is recorded in the database B, and thus the database B becomes the target incremental database at this time.
Regarding the completion of the batch synchronization of the first history data of the remote system by the history database, the present invention is determined by the steps of:
inquiring test data from the historical database, wherein the test data comprises a time field corresponding to the first reference time;
if the test data can be inquired from the historical database, the historical database completes the synchronization of the historical data; if the test data cannot be inquired from the historical database, the historical database does not finish synchronizing the historical data.
And S3, taking all data in the historical database and the target incremental database as target synchronous data.
On the basis that the database a has completed the current synchronization task, the data in the database a and the database C constitute the complete full amount of data, i.e. the target synchronization data. When the order information of a certain customer needs to be inquired, the correct result can be obtained only by inquiring from the database A and the database C.
On the basis that the database A does not complete the current synchronization task, the data in the database A and the database B form complete full data, namely target synchronization data. When the order information of a certain customer needs to be inquired, the correct result can be obtained only by inquiring from the database A and the database B.
Further, when it is determined that the data in the database a and the database C constitute target synchronization data, the data in the database B is emptied and third history data generated by a remote system during a third reference period, which is a next time period consecutively arranged with the second reference period, is synchronized to the database B in real time. For example, if the first reference period is 1 month and 22 days, the second reference period is 1 month and 23 days, and the third reference period is 1 month and 24 days.
When it is determined that the data in database a and database B constitute the target database, the database C is emptied of data and third history data generated by the remote system during a third reference period is synchronized to the database C in real time.
The invention empties the data in the database B or the database C, so as to be continuously used for storing the incremental data in the next data synchronization cycle, and ensures to obtain correct target synchronization data through the cyclic use of the database B and the database C. In summary, when the historical databases batch-synchronize first historical data in a first reference time, two incremental databases always exist as supplements, wherein one incremental database is used for real-time synchronization of the first historical data in the first reference time, and the other incremental database is used for real-time synchronization of the second historical data in a second reference time, so that whether the historical databases complete batch synchronization or not, corresponding target incremental data can be obtained, and the target incremental data and the data in the historical databases form complete and non-repeated full-volume data.
The above embodiments are still based on. And setting the current time as 1 month and 23 days, synchronizing all data generated by the remote system 22 days in the database A, synchronizing incremental data generated by the remote system at the time of 22 days in the database B, and synchronizing the incremental data generated by the remote system 23 days in the database C. It should be noted here that the mechanism for synchronizing data in database a is different from the mechanism for synchronizing data in databases B and C, and database a is all data generated in the previous day synchronized at a fixed time every day, and relatively, the time required for completing the synchronization of all data is relatively long. And the database B and the database C perform real-time synchronization on each new data in the remote system, so that the synchronization speed is high and almost no delay exists. After database a completes synchronizing all data generated by the remote system 22 day, database C is taken as the target incremental database and the data in database B is emptied.
Assuming that the current time reaches 1 month and 24 days, at this time, the database A needs to synchronize all data generated by the remote system on 23 days, the database C synchronizes incremental data generated by the remote system on 23-day-zero, and at this time, under the condition that the database B is idle, the database B is set to synchronize incremental data generated by the remote system on 24-day-zero. At this time, if the database a completes the synchronization of the 23-day data, the data in the database B and the database a constitute the complete full-scale data, and the database B becomes the target incremental database.
Consider another case, where database a needs to synchronize all data generated by the remote system on day 23, and database C is synchronized with incremental data generated in the remote system on day 23, assuming that the current time has reached day 1 and day 24, and where database B is idle, database B is set to synchronize incremental data generated in the remote system on day 24. At this time, if the database a is not synchronized with the 23-day data, the data in the database C and the database a constitute complete full-scale data, and the database C becomes the target incremental database.
In summary, in order to implement the synchronization of data from the remote system completely in real time, the invention is respectively provided with a history database and two incremental databases. Wherein the historical database is used to batch synchronize a first reference time with all data previously generated by the remote system, the first database is used to synchronize incremental data generated by the remote system at the first reference time in real time, and the second database is used to synchronize incremental data generated by the remote system at a second reference time, wherein the third reference time is the next time period in a consecutive sequence with the second reference time period.
Further, the invention selects one of the two incremental databases as a target incremental database according to whether the historical database completes the data synchronization task, and the data in the historical database and the data in the target incremental database form complete full data, namely target incremental data. Specifically, the method for selecting the target incremental database from the two incremental databases comprises the following steps: if the historical database completes the data synchronization task, selecting the second database as a target incremental database, and emptying the data in the first database; and if the historical database does not complete the data synchronization task, selecting the first database as a target incremental database, and emptying the data in the second database.
S4: and sending the target synchronous data to the local terminal.
On the basis of acquiring the target synchronous data with real-time performance and integrity, the local server sends the target synchronous data to the local terminal so that the local terminal can perform corresponding management operations such as inquiry, modification and the like.
Referring to fig. 2, a big data synchronization apparatus is shown, in the embodiment, the big data synchronization apparatus 10 may include or be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to implement the big data synchronization method. The program modules referred to herein are a series of computer program instruction segments that perform particular functions and are more suitable than the program itself for describing the execution of the big data synchronization apparatus 10 on a storage medium. The following description will specifically describe the functions of the program modules of the present embodiment:
the state acquisition module 11 is adapted to respond to a synchronous data acquisition instruction sent by a local terminal and acquire a synchronous completion state of a local historical database; wherein the history database is used for performing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion state is used for representing whether the historical database completes batch synchronization of the first historical data;
an increment selection module 12, adapted to select a local first database or a local second database as a target increment database according to the synchronization completion status of the historical database; wherein the first database is used for real-time synchronization of the first historical data generated by the remote system during the first reference period, the second database is used for real-time synchronization of the second historical data generated by the remote system during a second reference period, and the second reference period is a next time period which is continuously arranged with the first reference period;
a data determination module 13 adapted to take all data in both the historical database and the target incremental database as target synchronization data;
a data sending module 14, adapted to send the target synchronization data to the local terminal.
According to the big data synchronization device provided by the present invention, the increment selecting module 13 comprises:
a first incremental selecting module 131, adapted to select the second database as a target incremental database if the historical database completes batch synchronization of the first historical data;
a second incremental selecting module 132, adapted to select the first database as a target incremental database if the historical database is not synchronized with the first historical data.
The big data synchronization device provided by the invention further comprises:
a first emptying module 15 adapted to empty the data in the first database and synchronize third history data generated by a remote system during a third reference period to the first database in real time after selecting the second database as a target incremental database, wherein the third reference period is a next time period arranged in series with the second reference period;
a second purging module 16 adapted to purge the second database of data after selecting the first database as the target incremental database and to synchronize in real time to the second database third historical data generated by the remote system during a third reference.
The embodiment also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server or a rack server (including an independent server or a server cluster composed of a plurality of servers) capable of executing programs, and the like. The computer device 20 of the present embodiment includes at least, but is not limited to: a memory 21, a processor 22, which may be communicatively coupled to each other via a system bus, as shown in FIG. 3. It is noted that fig. 3 only shows the computer device 20 with components 21-22, but it is to be understood that not all shown components are required to be implemented, and that more or fewer components may be implemented instead.
In the present embodiment, the memory 21 (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the storage 21 may be an internal storage unit of the computer device 20, such as a hard disk or a memory of the computer device 20. In other embodiments, the memory 21 may also be an external storage device of the computer device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the computer device 20. Of course, the memory 21 may also include both internal and external storage devices of the computer device 20. In this embodiment, the memory 21 is generally used for storing an operating system and various application software installed on the computer device 20, such as the program code of the big data synchronization apparatus 10 in the first embodiment. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 22 is typically used to control the overall operation of the computer device 20. In this embodiment, the processor 22 is configured to execute the program code stored in the memory 21 or process data, for example, execute the big data synchronization apparatus 10, so as to implement the big data synchronization method according to the first embodiment.
The present embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, etc., on which a computer program is stored, which when executed by a processor implements corresponding functions. The computer-readable storage medium of the present embodiment is used for storing the big data synchronization apparatus 10, and when executed by a processor, the big data synchronization method of the first embodiment is implemented.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example" or "some examples" or the like are intended to mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A big data synchronization method is characterized by comprising the following steps:
responding to a synchronous data acquisition instruction sent by a local terminal, and acquiring the synchronous completion state of a local historical database; wherein the history database is used for performing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion state is used for representing whether the historical database completes batch synchronization of the first historical data;
according to the synchronous completion state of the historical database, selecting a local first database or a local second database as a target incremental database; wherein the first database is used for real-time synchronization of the first historical data generated by the remote system during the first reference period, the second database is used for real-time synchronization of the second historical data generated by the remote system during a second reference period, and the second reference period is a next time period which is continuously arranged with the first reference period;
taking all data in the historical database and the target incremental database as target synchronous data;
and sending the target synchronous data to the local terminal.
2. The big data synchronization method according to claim 1, wherein the step of selecting the first database or the second database as a target incremental database according to the synchronization completion status of the historical database comprises:
if the historical database completes batch synchronization of the first historical data, selecting the second database as a target incremental database;
and if the historical database does not synchronize the first historical data, selecting the first database as a target incremental database.
3. The big data synchronization method according to claim 2, further comprising, after the step of selecting the second database as a target incremental database:
the data in the first database is emptied and third history data generated by a remote system during a third reference period is synchronized to the first database in real time, wherein the third reference period is a next time period in a consecutive sequence with the second reference period.
4. The big data synchronization method according to claim 2, further comprising, after selecting the first database as a target incremental database:
and emptying the data in the second database, and synchronizing third history data generated by a remote system in a third reference period to the second database in real time, wherein the third reference period is the next time period which is continuously arranged with the second reference period.
5. The big data synchronization method according to claim 1, wherein the step of obtaining the synchronization completion status of the local history database comprises:
querying test data from the historical database, wherein the test data comprises a time field corresponding to the first reference period;
if the test data can be inquired from the historical database, the historical database completes batch synchronization of the historical data;
if the test data cannot be inquired from the historical database, the historical database does not complete batch synchronization of the historical data.
6. A big data synchronization apparatus, comprising:
the state acquisition module is adapted to respond to a synchronous data acquisition instruction sent by a local terminal and acquire the synchronous completion state of a local historical database; wherein the history database is used for performing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion state is used for representing whether the historical database completes batch synchronization of the first historical data;
the increment selection module is suitable for selecting a local first database or a local second database as a target increment database according to the synchronous completion state of the historical database; wherein the first database is used for real-time synchronization of the first historical data generated by the remote system during the first reference period, the second database is used for real-time synchronization of the second historical data generated by the remote system during a second reference period, and the second reference period is a next time period which is continuously arranged with the first reference period;
a data determination module adapted to take all data in both the historical database and the target incremental database as target synchronization data;
and the data sending module is suitable for sending the target synchronous data to the local terminal.
7. The big data synchronization apparatus according to claim 6, wherein the increment selection module comprises:
the first increment selection module is suitable for selecting the second database as a target increment database if the historical database completes batch synchronization of the first historical data;
and the second increment selection module is suitable for selecting the first database as a target increment database if the historical database is not synchronized with the first historical data.
8. The big data synchronization apparatus according to claim 7, further comprising:
a first emptying module adapted to empty the data in the first database and synchronize third history data generated by a remote system during a third reference period to the first database in real time after selecting the second database as a target incremental database, wherein the third reference period is a next time period arranged in series with the second reference period;
and the second emptying module is suitable for emptying the data in the second database after the first database is selected as the target incremental database, and synchronizing third history data generated by the remote system in a third reference period to the second database in real time.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 5 are implemented by the processor when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910841880.8A CN110674214B (en) | 2019-09-06 | 2019-09-06 | Big data synchronization method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910841880.8A CN110674214B (en) | 2019-09-06 | 2019-09-06 | Big data synchronization method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110674214A true CN110674214A (en) | 2020-01-10 |
| CN110674214B CN110674214B (en) | 2023-11-21 |
Family
ID=69076613
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910841880.8A Active CN110674214B (en) | 2019-09-06 | 2019-09-06 | Big data synchronization method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110674214B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377872A (en) * | 2021-06-25 | 2021-09-10 | 北京红山信息科技研究院有限公司 | Offline synchronization method, device and equipment of online system data in big data center |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030217152A1 (en) * | 2002-05-15 | 2003-11-20 | Adc Dsl Systems, Inc. | Resource sharing with database synchronization |
| CN104270424A (en) * | 2014-09-17 | 2015-01-07 | 深圳创维数字技术有限公司 | Database synchronization method, server and system |
| CN104809200A (en) * | 2015-04-24 | 2015-07-29 | 联动优势科技有限公司 | Database synchronization method and device |
| CN106919664A (en) * | 2017-02-16 | 2017-07-04 | 北京诺思维信科技有限公司 | Suitable for the inquiry of historical data methods of exhibiting and system of power station Full scope simulator |
| CN107729515A (en) * | 2017-10-26 | 2018-02-23 | 咪咕文化科技有限公司 | The method, apparatus and storage medium of a kind of data syn-chronization |
| US20180150540A1 (en) * | 2016-11-29 | 2018-05-31 | Sap Se | Synchronization Mechanism for Serialized Data Log Replay in Database Systems |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
-
2019
- 2019-09-06 CN CN201910841880.8A patent/CN110674214B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030217152A1 (en) * | 2002-05-15 | 2003-11-20 | Adc Dsl Systems, Inc. | Resource sharing with database synchronization |
| CN104270424A (en) * | 2014-09-17 | 2015-01-07 | 深圳创维数字技术有限公司 | Database synchronization method, server and system |
| CN104809200A (en) * | 2015-04-24 | 2015-07-29 | 联动优势科技有限公司 | Database synchronization method and device |
| US20180150540A1 (en) * | 2016-11-29 | 2018-05-31 | Sap Se | Synchronization Mechanism for Serialized Data Log Replay in Database Systems |
| CN106919664A (en) * | 2017-02-16 | 2017-07-04 | 北京诺思维信科技有限公司 | Suitable for the inquiry of historical data methods of exhibiting and system of power station Full scope simulator |
| CN107729515A (en) * | 2017-10-26 | 2018-02-23 | 咪咕文化科技有限公司 | The method, apparatus and storage medium of a kind of data syn-chronization |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377872A (en) * | 2021-06-25 | 2021-09-10 | 北京红山信息科技研究院有限公司 | Offline synchronization method, device and equipment of online system data in big data center |
| CN113377872B (en) * | 2021-06-25 | 2024-02-27 | 北京红山信息科技研究院有限公司 | Offline synchronization method, device and equipment of online system data in big data center |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110674214B (en) | 2023-11-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113111129B (en) | Data synchronization method, device, equipment and storage medium | |
| CN111078140B (en) | Nuclear power station file uploading management method and device, terminal equipment and medium | |
| CN113760476B (en) | Task dependency processing method and related device | |
| WO2019109519A1 (en) | Service rule management method, apparatus and device, and computer-readable storage medium | |
| CN111400288A (en) | Data quality inspection method and system | |
| CN111680104B (en) | Data synchronization method, device, computer equipment and readable storage medium | |
| CN111784318A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112488745A (en) | Intelligent charge control management method, device, equipment and storage medium | |
| CN106874343B (en) | Data deletion method and system for time sequence database | |
| CN111459629A (en) | Azkaban-based project operation method and device and terminal equipment | |
| CN112559525B (en) | Data checking system, method, device and server | |
| CN113468143A (en) | Data migration method, system, computing device and storage medium | |
| CN110674214B (en) | Big data synchronization method, device, computer equipment and storage medium | |
| US10409651B2 (en) | Incremental workflow execution | |
| CN115495527A (en) | Data synchronization management system and method | |
| WO2019062087A1 (en) | Attendance check data testing method, terminal and device, and computer readable storage medium | |
| CN115455106A (en) | Power distribution monitoring method, service platform, equipment and storage medium for power distribution operation and maintenance | |
| CN113971074A (en) | Transaction processing method and device, electronic equipment and computer readable storage medium | |
| CN116628042A (en) | Data processing method, device, equipment and medium | |
| CN116340051A (en) | Data processing method, related device and equipment | |
| CN117290113B (en) | Task processing method, device, system and storage medium | |
| CN108037981B (en) | Accessory management method and device | |
| CN117421337B (en) | Data acquisition method, device, equipment and computer readable medium | |
| CN115936402B (en) | Task replication method, device, electronic equipment and storage medium | |
| CN117493123A (en) | Monitoring task scheduling method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |