CN114385754A

CN114385754A - Data synchronization method, system and storage medium

Info

Publication number: CN114385754A
Application number: CN202111552168.XA
Authority: CN
Inventors: 陈洁娜
Original assignee: WeBank Co Ltd
Current assignee: WeBank Co Ltd
Priority date: 2021-12-17
Filing date: 2021-12-17
Publication date: 2022-04-22
Also published as: WO2023109238A1

Abstract

The embodiment of the application discloses a data synchronization method, which comprises the following steps: if a target operation statement which is specific to a target data table and belongs to a target operation type is detected, determining at least one second storage node which has a correlation with the target data table; wherein the target operation statement is used for changing the content in the target data table; determining a data synchronization mode of each second storage node; and sending the target operation statement to the corresponding second storage node based on each data synchronization mode so as to realize synchronous updating of the target data table in the first storage node and the second storage node. The embodiment of the application also discloses a data synchronization system and a storage medium.

Description

Data synchronization method, system and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data synchronization method, system, and storage medium.

Background

With the rapid development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changing to financial technology (Fintech), but higher requirements are also put forward on the technologies due to the requirements of the financial industry on safety and real-time performance. With the increasing demand of users, the distributed storage system is widely used to guarantee the user experience. In the application scenario of the distributed storage system, when data in a certain storage system needs to be changed, for example, addition, deletion, and modification, the copy data in other systems corresponding to the data needs to be changed correspondingly, so as to ensure data consistency.

At present, when a change operation is required for data in a certain storage system and corresponding changes are also required for copy data in other systems, a common method is to send a change request to other systems at the same time. However, in the implementation process, each other system needs to maintain the same code, but because the iteration speed of the service system is high, the maintenance personnel of each other system are usually inconsistent, so that the codes maintained by the systems together are inconsistent, and the change request cannot be responded in time, so that the data in the distributed storage system is inconsistent, and the data synchronization efficiency is low.

Content of application

In order to solve the above technical problems, embodiments of the present application are expected to provide a data synchronization method, a device, and a storage medium, so as to solve the problem of data synchronization efficiency of a current distributed storage system, effectively ensure consistency of data in the distributed storage system, and improve data synchronization efficiency in the distributed storage system.

The technical scheme of the application is realized as follows:

in a first aspect, a data synchronization method is applied to a first storage node, and the method includes:

if a target operation statement which is specific to a target data table and belongs to a target operation type is detected, determining at least one second storage node which has a correlation with the target data table; wherein the target operation statement is used for changing the content in the target data table;

determining a data synchronization mode of each second storage node;

and sending the target operation statement to the corresponding second storage node based on each data synchronization mode so as to realize synchronous updating of the target data table in the first storage node and the second storage node.

In a second aspect, a data synchronization system, the system comprising: a first storage node and at least one second storage node; wherein:

the first storage node is used for determining at least one second storage node having a correlation with a target data table if a target operation statement which is specific to the target data table and belongs to a target operation type is detected; wherein the target operation statement is used for changing the content in the target data table; determining a data synchronization mode of each second storage node; sending the target operation statement to the corresponding second storage node based on each data synchronization mode;

the second storage node is used for receiving the target operation statement; and adopting the target operation statement to execute an updating operation on the target data table corresponding to the target operation statement in the second storage node to obtain an updating result.

In a third aspect, a storage medium has stored thereon a data synchronization program which, when executed by a processor, implements the steps of the data synchronization method as defined in any one of the above.

In the embodiment of the application, if a target operation statement which is specific to a target data table and belongs to a target operation type is detected, a first storage node determines at least one second storage node which has a correlation with the target data table, determines a data synchronization mode of each second storage node, and sends the target operation statement to the corresponding second storage node based on each data synchronization mode, so that the second storage node executes the target operation statement, synchronous update operation of the target data table with the first storage node is realized, the problem of data synchronization efficiency of the existing distributed storage system is solved, data consistency in the distributed storage system is effectively guaranteed, and data synchronization efficiency in the distributed storage system is improved.

Drawings

Fig. 1 is a schematic flowchart of a data synchronization method according to an embodiment of the present application;

fig. 2 is a schematic flowchart of another data synchronization method according to an embodiment of the present application;

fig. 3 is a schematic flowchart of another data synchronization method according to an embodiment of the present application;

fig. 4 is a schematic flowchart of a data synchronization method according to another embodiment of the present application;

fig. 5 is a schematic flowchart of another data synchronization method according to another embodiment of the present application;

fig. 6 is a schematic flowchart of another data synchronization method according to another embodiment of the present application;

fig. 7 is a schematic flowchart of a data synchronization method according to another embodiment of the present application;

fig. 8 is a schematic view of an application scenario provided in an embodiment of the present application;

fig. 9 is a schematic view of a process for implementing an alarm prompt according to an embodiment of the present application.

Fig. 10 is a schematic structural diagram of a data synchronization system according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

An embodiment of the present application provides a data synchronization method, which is shown in fig. 1 and is applied to a first storage node, and the method includes the following steps:

step 101, if a target operation statement belonging to a target operation type for a target data table is detected, determining at least one second storage node having a relationship with the target data table.

Wherein the target operation statement is used for changing the content in the target data table.

In the embodiment of the application, the first storage node is a distributed storage node for a user to change the content of a target data table stored by the first storage node. The target operation type is mainly the operation type used for changing the content in the data table. The at least one second storage node having an association with the target data table refers to a storage node storing the target data table.

And 102, determining a data synchronization mode of each second storage node.

In the embodiment of the present application, different data synchronization modes are configured for different second storage nodes, and in the synchronization process, the data synchronization mode of each second storage node may be adjusted according to an actual synchronization situation. The data synchronization mode at least comprises a real-time synchronization mode and an asynchronous synchronization mode.

And 103, sending the target operation statement to the corresponding second storage node based on each data synchronization mode.

And sending the target operation statement to the corresponding second storage node based on each data synchronization mode so as to realize the synchronous update of the first storage node and the second storage node aiming at the target data table.

In the embodiment of the application, after the first storage node determines the data synchronization mode of each second storage node, the target operation statement is sent to the corresponding second storage node based on the data synchronization mode of each second storage node, so that after the second storage node obtains the target operation statement, the target operation statement can be directly executed, and further, the corresponding data content of the target data table in the second storage node is adjusted and updated.

Based on the foregoing embodiments, an embodiment of the present application provides a data synchronization method, which is applied to a first storage node, and includes the following steps:

step 201, if a target operation statement belonging to a target operation type for the target data table is detected, determining at least one second storage node having a relationship with the target data table.

In this embodiment of the application, when the target operation statement is that the first storage node performs a corresponding operation instruction from a user on the target data table of the first storage node, the data service layer of the first storage node responds to the statement belonging to the target operation type in the operation statement of the operation instruction, and the data service layer may directly respond to the statement by using the database in the storage node. The target operation types include at least the following types: insert (Insert), Delete (Delete), and Update (Update), determining that the resulting target operation statement belongs to at least one of the above target operation types. When determining at least one second storage node having an association relationship with the target operation table, the second storage node may be determined from a relationship between a pre-stored data table and a storage node, for example, after determining that the target data table is obtained, the identification information of the target data table may be determined, and the storage node having the identification information of the target data table may be determined from a relationship between the pre-stored data table and a storage node, that is, the at least one second storage node may be obtained. The target operation statement may be, for example, a Structured Query Language (SQL) type statement that may respond directly to a database in the storage node, such as an Insert SQL statement for the target data table, a Delete SQL statement for the target data table, and an Update statement for the target data table.

Step 202, determining a data synchronization mode of each second storage node.

In the embodiment of the application, the configuration parameters of each second storage node are determined to obtain the data synchronization mode of each second storage node, so that the data synchronization mode of at least one second storage node can be obtained.

Step 203, based on each data synchronization mode, sending the target operation statement to the corresponding second storage node.

And the target operation statement is used for updating the target data table in the corresponding second storage node so as to realize synchronous updating of the target data table in the first storage node and the second storage node.

In this embodiment of the application, when the data synchronization mode of the second storage node is a real-time synchronization mode, the target operation statement may be sent to the corresponding second storage node in the real-time synchronization mode, and when the data synchronization mode of the second storage node is an asynchronous synchronization mode, the target operation statement may be sent to the second storage node in the asynchronous synchronization mode according to a preset asynchronous time period.

Based on the foregoing embodiments, in other embodiments of the present application, step 203 may be implemented by steps 203a to 203 c:

step 203a, if each data synchronization mode is a real-time synchronization mode, determining a table building statement corresponding to the target data table.

In the embodiment of the application, when the data synchronization mode of the second storage node is a real-time synchronization mode, the node configuration information of the second storage node is read, and the table building statement corresponding to the target data table is obtained from the node configuration information.

Step 203b, generating a first request message based on the table building statement and the target operation statement.

In the embodiment of the application, the table building statement and the target operation statement are respectively filled in the request message to obtain the first request message.

And 203c, sending the first request message to the corresponding second storage node in a real-time synchronous mode.

The first request message comprises a target operation statement, and the table building statement in the first request message is used for generating a target data table when the second storage node detects that the target data table does not exist after receiving the first request message.

In the embodiment of the application, the first request message is sent to the corresponding second storage node in real time in a real-time synchronization mode, so that the target operation statement is sent to the second storage node.

Based on the foregoing embodiment, in another embodiment of the present application, referring to fig. 2, before the first storage node performs step 203, the first storage node is further configured to perform step 204:

and step 204, determining the target data volume corresponding to the target operation statement.

In the embodiment of the application, the first storage node determines the data volume included in the data table content corresponding to the target operation statement, and after determining the number of fields included in the target data table, calculates the quotient between the data volume included in the data table content corresponding to the target operation statement and the number of the fields, so as to obtain the target data volume.

Correspondingly, step 203 may be implemented by step 203 d:

and 203d, sending the target operation statement to the corresponding second storage node based on the target data volume and each data synchronization mode.

In this embodiment of the present application, the default data synchronization mode of the second storage node may be a real-time data synchronization mode. Under different data synchronization modes, different preprocessing modes can be adopted to preprocess the target operation statement according to different target data volumes, so that the implementation efficiency of the data synchronization method is improved, and the sending efficiency of the target operation statement to the corresponding second storage node is improved.

Based on the foregoing embodiments, in other embodiments of the present application, the step 203d may be implemented by the steps a 11-a 13:

step a11, if each data synchronization mode is a real-time synchronization mode and the target data amount is less than the preset threshold, determining the table building statement corresponding to the target data table.

In the embodiment of the application, the preset threshold may be an empirical value obtained through a large number of experiments, or an empirical value obtained through setting by a user according to actual requirements, and during the use process, the preset threshold may be dynamically adjusted according to the response duration of the target operation statement. The preset threshold value does not affect the performance of the implementation process of the data synchronization method.

Step a12, generating a second request message based on the table building statement and the target operation statement.

Step a13, sending the second request message to the corresponding second storage node in a real-time synchronization manner.

The second request message includes a target operation statement, and the table building statement included in the second request message is used for generating a target data table when the second storage node detects that the target data table does not exist after receiving the second request message.

In this way, under the condition that the data synchronization mode corresponding to the second storage node is the real-time synchronization mode, if the target data volume of the target operation statement is smaller than the preset threshold, the target operation statement and the table building statement are directly generated into the second request message.

Based on the foregoing embodiments, in other embodiments of the present application, the step "determining a table building statement corresponding to a target data table" in step 203a or step a11 may be implemented through steps b11 to b 17:

and b11, determining the calling information corresponding to the target operation statement.

In the embodiment of the application, the calling information is data information of a data service layer responding to a target operation statement.

Step b12, determining the reference field information for defining the structure of the target data table from the calling information.

In the embodiment of the application, the corresponding node configuration information is determined from the calling information, and the reference field information corresponding to the structure of the target data table can be obtained by analyzing the node configuration information.

And b13, replacing the reference character in the reference field information with the target character to obtain the target field information.

In the embodiment of the present application, the target characters may be "% and blank characters" in the SQL statement, for example, and the corresponding reference characters are "? "and" \\ \ s ] + "so that target field information is obtained after the replacement processing.

After the data synchronization apparatus performs step b13, it may select to perform steps b 14-b 16 or step b17, wherein if the target field information includes table field entity information, it selects to perform steps b 14-b 16, and if the target field information does not include table field entity information, it selects to perform step b 17.

And b14, traversing the target field information to obtain the target field name if the target field information comprises the table field entity information.

And b15, acquiring the reference field type corresponding to the target field name.

In the embodiment of the application, the corresponding reference field type is obtained from the table field entity information according to the target field name.

And b16, processing the reference field type by adopting the target statement rule to obtain the table building statement.

In the embodiment of the present application, the target statement rule is a requirement of the SQL statement format, for example, if the field is a character string type, a single quotation mark (") is added to the field, if the field is a date type, the field is converted into a date format stored in the target data table and the single quotation mark (") is added, and if the field is a significant number exceeding 16 bits, the field is converted into the character string type and the single quotation mark (") is added by a large number self-contained character string conversion method, so that the table building statement can be obtained.

And b17, if the target field information does not include the table field entity information, obtaining the statement corresponding to the target operation statement from the target field storage area to obtain the table building statement.

In the embodiment of the present application, the target field storage area may be a JavaBean, for example.

Based on the foregoing embodiments, in other embodiments of the present application, referring to fig. 3, after the first storage node performs step 204, the first storage node is further configured to perform step 205:

step 205, if each data synchronization mode is a real-time synchronization mode and the target data amount is greater than or equal to the preset threshold, updating the corresponding data synchronization mode to be an asynchronous synchronization mode.

In this embodiment of the application, when the data synchronization mode of the second storage node is the real-time synchronization mode, if the target data amount is greater than or equal to the preset threshold, if the target operation statement is directly sent to the second storage node in the real-time synchronization mode, the operation performance of the second storage node is seriously affected, and therefore, the data synchronization mode of the second storage node is adjusted and updated to the asynchronous synchronization mode.

Based on the foregoing embodiments, in other embodiments of the present application, step 203 or step 203d may also be implemented by steps c11 to c 16:

and c11, if each data synchronization mode is an asynchronous synchronization mode and the target data volume is greater than or equal to a preset threshold value, generating a target operation file based on the target operation statement.

In this embodiment of the application, when the data synchronization mode of the second storage node is an asynchronous synchronization mode, if the target data size is greater than or equal to a preset threshold, a file is created, and a target operation statement is placed in the created file to obtain a target operation file. The file may be named by using the identification information of the target data table, and further, since the update operation on the target data table may be continuous, in order to distinguish the target operation statements corresponding to different time periods, the file may be named by using the identification information of the target data table, and may further include a current time and a randomly generated number, that is, the file in which the target operation statement is stored is named by using the identification information of the target data table, the current time, and the randomly generated number, so as to obtain the target operation file.

And if each data synchronization mode is an asynchronous synchronization mode and the target data volume is smaller than a preset threshold value, directly sending the target operation statement to the corresponding second storage node by adopting the asynchronous synchronization mode. Illustratively, under the condition that the data synchronization mode of the second storage node is an asynchronous synchronization mode, if the target data volume of the target operation statement is smaller than a preset threshold, the first storage node sends the target operation statement to the second storage node every preset asynchronous time period, so that the second storage node executes the target operation statement, and synchronous update operation for the target data table and the target data table in the first storage node is realized.

And c12, sending the target operation file to the file server.

In an embodiment of the present application, the file server is a server for storing files between the first storage node and the at least one second storage node.

And c13, receiving indication information sent by the file server for indicating the storage position of the target operation file.

In this embodiment of the application, the indication information may be Identity identification Information (ID) and a corresponding Hash value (Hash) of a storage file, which is generated by the file server after receiving the target operation file and used for identifying the target operation file. Correspondingly, the generated ID for identifying the storage file of the target operation file and the corresponding Hash may be determined by an upload frame rate (FPS).

And step c14, storing the indication information.

And c15, generating an acquisition instruction according to the preset asynchronous time period.

The obtaining instruction is used for indicating the indication information of the file server to obtain the target operation file, and obtaining the target operation statement to execute the corresponding updating operation.

In the embodiment of the present application, the preset asynchronous time period may be a time period empirical value set according to an actual situation, or may be a time empirical value obtained through a large number of experiments. The preset asynchronous time period may be, for example, every 5 minutes, every 30 minutes, or 6:00 a day, etc.

And c16, sending a obtaining instruction to the corresponding second storage node.

In this embodiment of the application, the first storage node sends the obtaining instruction to the corresponding second storage node immediately after the obtaining instruction is generated every preset asynchronous time period, so that the second storage node obtains the target operation file at the storage location indicated by the indication information from the file server, so that the second storage node executes the corresponding target operation statement in the target operation file.

In some application scenarios, after receiving the indication information, the first storage node generates an acquisition instruction, and sends the acquisition instruction to the corresponding second storage node at intervals of a preset asynchronous time period, so as to implement an asynchronous and synchronous mode between the first storage node and at least one second storage node for the target data table.

It should be noted that, the steps c11 to c16 may be executed after the step 205, that is, after the data synchronization method corresponding to the second storage node is updated from the real-time synchronization method to the asynchronous synchronization method, the steps c11 to c16 are continuously executed.

Based on the foregoing embodiments, in other embodiments of the present application, referring to fig. 4, after the first storage node performs step 203, the first storage node is further configured to perform steps 206 to 207:

and step 206, receiving the updating result sent by the corresponding second storage node.

And the updating result is the operation result of the second storage node executing the target operation statement.

In this embodiment of the present application, the update result may be sent by the second storage node to the first storage node in the form of a response packet.

Under the condition that the data synchronization mode is a real-time synchronization mode, after receiving a second request message, the second storage node analyzes the second request message to obtain a target operation statement and a table building statement; judging whether the second storage node has a corresponding target data table in the target operation statement or not according to the target operation statement; if the target data table does not exist in the second storage node, the second storage node executes the table building statement obtained by analysis, creates a corresponding target data table, and then executes the target operation statement aiming at the target data table to obtain an operation result, namely an updating result; if the second storage node has the target data table, the second storage node judges whether the field information in the target data table is consistent with the field information in the table building statement, wherein the field information at least comprises information contents such as field names, field types, field lengths and the like; if the field information in the target data table is consistent with the field information in the table building statement, after caching the target data table by the second storage node, executing the target operation statement aiming at the target data table, and updating the content in the target data table to obtain an updating result, wherein when the purpose of caching before executing the target operation statement aiming at the target data table is that the target operation statement execution exception occurs in other second storage nodes, the rollback operation can be realized by utilizing the cached target data table; and if the field information in the target data table is inconsistent with the field information in the table building statement, the second storage node acquires the table building statement of the target data table in the second storage node, and the second storage node fails to execute based on the table building statement and the target operation statement of the target data table in the second storage node, so that an updating result is obtained.

In the case that the data synchronization mode is an asynchronous synchronization mode, if the target operation statement is received by the second storage node every preset asynchronous time period, the execution process of the target operation statement by the second storage node is consistent with the execution process of the target operation statement by the second storage node in the case of the real-time synchronization mode, which is not specifically limited here.

Under the condition that the data synchronization mode is the asynchronous synchronization mode, if the target operation file is received by the second storage node every other preset asynchronous time period, the second storage node analyzes the target operation file, and sequentially acquires a preset number of operation statements from the target operation file for execution each time according to the time sequence of the target operation statements, and the corresponding execution process is consistent with the execution process of the target operation statements by the second storage node under the condition of the real-time synchronization mode, and is not specifically limited here.

And step 207, updating the data synchronization mode of the corresponding second storage node based on the updating result.

In the embodiment of the present application, under different data synchronization modes, different update results are different in the update modes of the data synchronization mode corresponding to the update of the second storage node.

Based on the foregoing embodiments, in other embodiments of the present application, step 207 may be implemented by step 207a or step 207 b:

step 207a, in case that the data synchronization mode is the real-time synchronization mode, if the update result indicates that the second storage node executes the target operation statement abnormally, the update data synchronization mode is the asynchronous synchronization mode.

In this embodiment of the application, when the data synchronization mode is the real-time synchronization mode, if the update result indicates that the second storage node successfully executes the target operation statement, the data synchronization mode of the corresponding second storage node does not need to be updated, and the data synchronization mode of the corresponding second storage node is continuously maintained as the real-time synchronization mode.

And step 207b, under the condition that the data synchronization mode is the asynchronous synchronization mode, if the update result shows that the second storage node successfully executes the target operation statement, the data synchronization mode is updated to be the real-time synchronization mode.

In this embodiment of the application, when the data synchronization mode is the asynchronous synchronization mode, if the update result indicates that the second storage node fails to execute the target operation statement, the data synchronization mode is not updated, and the data synchronization mode of the corresponding second storage node is continuously maintained as the asynchronous synchronization mode.

Based on the foregoing embodiments, in other embodiments of the present application, referring to fig. 5, after the first storage node performs step 206, it is further configured to perform steps 208 to 209:

and 208, if the updating result shows that the second storage node executes the target operation statement abnormally, determining the abnormal priority corresponding to the updating result.

In this embodiment of the application, when the update result indicates that the second storage node successfully executes the target operation statement, the synchronization process is ended. And under the condition that the updating result indicates that the second storage node executes the target operation statement abnormally, determining the abnormal priority corresponding to the updating result according to the content of the updating result.

And step 209, executing the alarm operation corresponding to the abnormal priority.

In the embodiment of the present application, the assumption of the exception priority includes that the representation levels are respectively expressed as: first stage, second stage, … …, n stage. Assuming that the abnormal priority includes three levels in total, the first level represents the highest priority, correspondingly, the alarm operation may be, for example, establishing a voice communication connection with the communication device of the relevant person in charge to prompt timely handling of the abnormal condition, the second level represents the second highest priority, the corresponding alarm operation may be, for example, establishing a communication connection with the communication device of the relevant person in charge, prompting the relevant person in charge to handle the abnormal condition when the relevant person is available by sending a prompt message, and the third level represents the lowest priority, and the corresponding alarm operation may be, for example, generating the prompt message and recording the prompt message for the relevant person in charge to refer.

Based on the foregoing embodiments, in other embodiments of the present application, referring to fig. 6, after the first storage node performs step 206, it is further configured to perform steps 210 to 211:

step 210, if the update result indicates that the second storage node executes the target operation statement abnormally, determining the second storage node, of which the update result indicates that the second storage node executes the target operation statement successfully, from the at least one second storage node, and obtaining at least one third storage node.

In this embodiment of the application, when the update result indicates that the second storage node executes the target operation statement abnormally, the update result indicates that the synchronization operation on the second storage node is not successfully executed, and therefore, this time of synchronization for the target data table is unsuccessful, at this time, the second storage node whose update result indicates that the execution of the target operation statement is successful is determined from at least one second storage node having an association relationship with the target data table, and at least one third storage node is obtained.

And step 211, sending a data rollback control command to at least one third storage node.

And the data rollback control instruction is used for instructing the third storage node to roll back to the data before the target operation statement is executed.

In this embodiment of the application, the first storage node sends the data rollback instruction to the at least one third storage node, so that the at least one third storage node responds to the data rollback instruction, and replaces the target data table after the target operation statement is executed with the target data table cached before the target operation statement is executed, thereby implementing the data rollback operation.

Based on the foregoing embodiments, in other embodiments of the present application, referring to fig. 7, after the first storage node performs step 206, the first storage node is further configured to perform steps 212 to 213:

and 212, under the condition that the update result sent by each second storage node indicates that the second storage node successfully executes the target operation statement, counting the data processing consumed duration of each second storage node to obtain at least one data processing consumed duration.

Step 213, updating the preset threshold based on at least one data processing consumed duration.

In the embodiment of the application, the first storage node analyzes at least one data processing consumed time to determine whether the current preset threshold is reasonable, if not, analyzes according to an indication of the processing consumed time to determine a new threshold to update the preset threshold, and if so, the preset threshold does not need to be updated. In some application scenarios, after obtaining the at least one data processing consumption duration, the at least one data processing consumption duration may also be sent to the relevant responsible person, so that the relevant responsible person analyzes the at least one data processing consumption duration, and resets the preset threshold.

Based on the foregoing embodiments, in other embodiments of the present application, step 201 may be implemented by steps 201a to 201 d:

step 201a, obtaining a mapper declaration parameter and a parameter type parameter in a data service layer.

Step 201b, processing the emitter declaration parameters and the parameter type parameters by adopting the target statement rules to obtain reference operation statements.

In the embodiment of the application, the target statement rule, namely the SQL statement rule, is adopted to analyze the emitter declaration parameters and the parameter type parameters, and the contents obtained after analysis are spliced, so that the SQL statement can be obtained.

Step 201c, if the operation type of the reference operation statement is the target operation type, determining that the target operation statement is the reference operation statement.

In the embodiment of the application, the reference operation statement is analyzed, and if the reference operation statement is determined to be of the target operation type, namely Insert, Delete or Update, the reference operation statement is determined to be the target operation statement.

Step 201d, at least one second storage node having a relationship with the target data table is determined.

Based on the foregoing embodiments, an embodiment of the present application provides a schematic structural diagram of a data synchronization system, which is shown in fig. 8 and includes: a main operation library system, a target standby library system 1, a target standby library system 2 and a target standby library system 3. The main operating library system, the target standby library system 1, the target standby library system 2 and the target standby library system 3 all include: a business logic application layer, a data service layer, an interceptor, and a master operations Database (DB). Wherein, the interceptor includes: the device comprises a data synchronization module, a synchronization result processing module, a data issuing abnormity monitoring and feedback processing module and a threshold setting module. It should be noted that fig. 8 only illustrates the structural composition of the main operating system library completely, the structural compositions of the target backup library system 1, the target backup library system 2, and the target backup library system 3 are the same as the structural composition of the main operating system library, the main operating system corresponds to the first storage node, the target backup library system 1, the target backup library system 2, and the target backup library system 3 correspond to the at least one second storage node, and the role functions between the first storage node and the second storage node may be replaced with each other.

Based on the schematic structural diagram of the data synchronization system shown in fig. 8, an embodiment of the present application provides a data synchronization method, and the specific implementation process is as follows:

after a business logic layer of the main operation database system receives an operation request sent by a user and used for requesting to update a target data table in the main operation database, the business logic layer sends the operation request to a data service layer, and a corresponding data operation statement in the data service layer responds to the operation request to realize the operation on the target data table. An interceptor in the main operation library system intercepts corresponding data operation statements in the data service layer, analyzes the intercepted data operation statements, and determines to obtain target operation statements which belong to target operation types such as Insert types, Delete types, Update types and the like and aim at the target data table. After the interceptor in the main operation library system determines that the target operation statement is obtained, determining at least one second storage node having a relationship with the target data table as a target standby library system 1, a target standby library system 2 and a target standby library system 3, determining a data synchronization mode corresponding to the target standby library system 1, the target standby library system 2 and the target standby library system 3, implementing a synchronization process on the target operation statement by adopting the data synchronization mode corresponding to each of the target standby library system 1, the target standby library system 2 and the target standby library system 3, and receiving an update result sent by the target standby library system 1, the target standby library system 2 and the target standby library system 3. The main operation library system can also update the data synchronization mode of the target standby library system 1, the target standby library system 2 and the target standby library system 3 according to the update results sent by the target standby library system 1, the target standby library system 2 and the target standby library system 3, and adjust the preset threshold value and the like, so that a complete closed-loop feedback system is realized, and an effective synchronization strategy is ensured.

At least one second storage node having an association relationship with the target data table is determined, and the at least one second storage node may be determined from a relationship between the pre-stored data table and the storage node. The relationship between the pre-stored data table and the storage node may be determined from an annotation included in a corresponding frame script, and may be determined from an annotation included in an Aspect Oriented Programming (AOP) of a Spring frame, where the annotation identifies a target library system to be issued and to specify the issuing of the corresponding data table. The relationship between the pre-stored data table and the storage node can be represented in a list form, for example, as shown in table 1, DcnRouter is identification information of the target data table, SysName is system name information of the system having the target data table, DcnNo is system number information of the system having the target data table, and ServiceId is a corresponding mapping relationship when information is transmitted between the systems having the target data table.

The interceptor in the main operating library system may be, for example, a persistent layer framework such as plug-in (plugin) functionality provided by MyBatis. The main operating library system and the target standby library system can be realized in a remote calling mode.

The process of the interceptor obtaining the target operation statement may be, for example: the interceptor obtains parameters in a data operation statement of the data service layer responding to the operation request, and resolves a command type, such as SqlCommandTypeSQL, included in a mapper (mapper) declaration (mapper State) from the parameters. And determining target operation statements of an Insert type, a Delete type, an Update type and the like according to the command type such as SqlCommandTypeSQL.

TABLE 1

The method includes the steps of determining at least one second storage node having an association relationship with a target data table, wherein the interceptor may obtain a class name according to an intercepted mapping statement identifier, such as MappedStatementId, mapping class object information according to the class name, judging whether the class has an added annotation and an annotation parameter according to the class object information, and if the class has the added annotation and the annotation parameter, determining to obtain at least one second storage node having an association relationship with the target data table.

The specific process of the interceptor determining the target operation statements of the Insert type, the Delete type, the Update type and the like according to the command type, such as SqlCommandTypeSql, may be as follows: the interceptor obtains bound sql for mappedstatus and parameter type (ParamObject) in the data service layer. And analyzing according to the two parameters of MappedStatement and ParamObject, and splicing into an SQL statement of a target operation type which can be directly executed to obtain a target operation statement. Further, the target data volume required to be executed in the SQL statement is obtained by dividing the data volume in the parameter list of bound SQL by the number of fields of the table structure.

The interceptor analyzes the node configuration information in the data operation statement of the data service layer responding to the operation request, and because the node configuration information comprises each field information defining the structure of the target data table, each field information in the node configuration information can be traversed, and the table building statement of the target data table is spliced. The specific implementation process can be as follows: all placeholders in the replacement node configuration information "? "is"% ", all blank characters" \ \ s ] + "are replaced, and the node configuration information after replacement is obtained; judging whether the table field entity class is used as a parameter in the replaced node configuration information; if the table field entity class is used as a parameter in the replaced node configuration information, sequentially circulating the parameter list, and determining to obtain a parameter set which belongs to the user and a parameter mapping relation set; acquiring field names and field types from parameter sets and parameter mapping relation sets which belong to user transmission; acquiring the field type from the corresponding set according to the acquired field name, wherein the specific implementation process comprises the following steps: if the field is of a character string type, adding a single quotation mark ("), if the field is of a date type, converting the field into a date format stored in a target data table and adding the single quotation mark ("), and if the field is of a significant number which exceeds 16 bits, converting the field into the character string type and adding the single quotation mark (") through a character string conversion method carried by the significant number to obtain a table building statement; if the table field entity class is not used as the parameter in the replaced node configuration information, the entity class batch parameter is obtained from the JavaBean, and the table building statement can be obtained. Wherein, the table building statement is also SQL statement.

And a data synchronization module of the interceptor determines the data synchronization mode of each target backup library system. The initial data synchronization mode of each target standby library system is defaulted to a real-time synchronization mode. And a data synchronization module of the interceptor checks whether the target data volume reaches a preset threshold, when the data synchronization mode is a real-time synchronization mode, if the target data volume is less than the preset threshold, the data synchronization mode is kept as the real-time synchronization mode, and if the target data volume is greater than or equal to the preset threshold, the data synchronization mode is updated from the real-time synchronization mode to an asynchronous synchronization mode. When the data synchronization mode is an asynchronous synchronization mode, if the target data volume is smaller than a preset threshold, the data synchronization mode is determined to be the asynchronous data synchronization mode, and if the target data volume is larger than or equal to the preset threshold, the data synchronization mode is determined to be the asynchronous file synchronization mode. The data synchronization mode can be realized by two parameters of a table name and the data synchronization mode during recording. Further, in the asynchronous and synchronous mode, the format shown in table 2 may be adopted to implement a specific asynchronous and synchronous mode. In table 2, the table _ name parameter is used to indicate the table name of the target data table; the dcn _ no parameter is used for representing the number of the target standby library system receiving the target operation data; the SQL parameter is used for recording an SQL script which needs to execute synchronization when the process _ type is SQL, otherwise, the SQL script is null; the ID _ Hash is used for recording the ID and the Hash value of the storage target operation FILE obtained by conversion when the process _ type is FILE and the storage target operation FILE is uploaded to the FILE server; the process _ type is used for identifying a specific execution type of an asynchronous and synchronous mode, and is identified by FILE representing an asynchronous FILE synchronous mode and SQL representing an asynchronous data synchronous mode; create _ time is used to represent the current time of generation of the asynchronous synchronization mode, which can be used as the logical order of the subsequent synchronization; the sys _ seq is used for representing a current serial number corresponding to the asynchronous and synchronous mode, and is used for the operation and maintenance personnel to use the serial number to search a call chain of the whole process after monitoring the abnormality so as to quickly locate the problem, execute a processing scheme and evaluate the influence range.

TABLE 2

table_name

sql

id_hash

dcn_no

process_type

create_time

sys_seq

Illustratively, if the data synchronization module of the interceptor determines that the data synchronization mode of the target backup library system 1 is the real-time synchronization mode, the interceptor fills the determined target operation statement and the table building statement into the real-time request message, issues the real-time request message to the real-time message queue corresponding to the system number corresponding to the target backup library system 1, and sends the real-time request message to the target backup library system 1 in the form of a queue message. After receiving the real-time request message, the target standby library system 1 judges the real-time request message as follows: judging whether a target data table of a table structure corresponding to the table building statement exists in the target database preparation system 1; if the matched target data table does not exist in the target standby library system 1, executing a table building statement in the real-time request message, creating to obtain a target data table, then executing a target operation statement in the real-time request message to obtain an updating result of the executed target operation statement, and filling the updating result into a response message to be sent to the main operation library system; if the matched target data table exists in the target database backup system 1, further checking whether the field name, the field type and the field length in the table building statement are consistent with those of the target data table in the target database backup system 1; if the field name, the field type and the field length are consistent, firstly caching a target data table in the target standby library system 1, then executing a target operation statement to obtain an update result, filling the update result into a response message, and sending the response message to the main operation library system; if at least one of the field name, the field type and the field length is inconsistent, obtaining the table structure information of the target data table in the target backup library system 1 through a specific statement such as a dataSource to obtain a table building statement of the target data table in the target backup library system 1, filling the table building statement of the target data table in the target backup library system 1, which is not passed through in execution, and the table building statement of the target data table in the target backup library system 1 and the reason which specifically causes the returned data to be not passed into a response message, and sending the response message to the main operation library system.

After the synchronization result processing module of the primary operating library system interceptor receives the response message sent by the target standby library system 1, if the response message indicates that the target standby library system 1 is successfully executed in a synchronization manner (i.e., the execution of the target operation statement is successful), the data synchronization operation for the target standby library system 1 is completed. If the response message indicates that the target backup library system 1 fails to execute synchronously (namely the execution target operand statement is abnormal), firstly updating a data synchronous mode corresponding to the target backup library system into an asynchronous synchronous mode; and then finding the subsystem of the standby library system which is successfully executed in the past through the system number to which the current updating result belongs, finally recording the synchronous target operation statement and the system number record corresponding to the target standby library system 1 according to the table 2, and updating the process _ type field in the table 2 to be SQL. Meanwhile, when the response message indicates that the synchronous execution of the target standby library system 1 fails, the exception information needs to be persisted, the information structure of the persisted exception information can be as shown in table 3, part of content identifiers in table 3 are the same as those in table 2, and details are not repeated here, and the route _ sys _ name in table 3 is used for identifying the name of the system to which the standby library which is executed abnormally belongs, so that operation and maintenance personnel can conveniently locate the system log; the remark is used for representing remarks and is mainly used for recording the reason of execution failure as a key log of alarm; urgent _ type is used to identify the urgent type, where H denotes high priority, L denotes low priority, and D has completed processing.

TABLE 3

table_name

sql

id_hash

dcn_no

route_sys_name

remark

create_time

urgent_type

If the reason of the synchronization failure is that at least one of the field name, the field type and the field length is inconsistent, the data issuing abnormity monitoring and feedback processing module of the interceptor determines that the urgent _ type is an abnormal information record with high priority, namely H, aiming at the urgent _ type with high priority, voice communication can be triggered, for example, relevant responsible persons are reminded, so that the relevant responsible persons can confirm whether the upstream and downstream fields are inconsistent and can adjust consistency, and the urgent _ type can be modified into D to be processed after the adjustment is consistent. If the failure reason is failure caused in the synchronization process, the data issuing abnormality monitoring and feedback processing module of the interceptor determines that the urgent _ type is the abnormal information record of the low priority L, and for the urgent _ type of the low priority, a prompt message such as text information or voice information can be sent to prompt relevant responsible persons so that the relevant responsible persons can arrange time in a centralized manner for processing, and the urgent _ type is modified into D after the relevant responsible persons process the urgent _ type.

On the premise that the synchronization of the target backup library system 1 fails and the synchronization of the target backup library system 2 and the target backup library system 3 is successful, the data sending abnormality monitoring and feedback processing module of the interceptor needs to perform data rollback operation on the target backup library system 2 and the target backup library system 3 because the synchronization of the target backup library system 1 fails. The main operation library system determines the system numbers of the target standby library system 2 and the target standby library system 3, assembles SQL regression statements to be executed into rollback notification messages, and issues the rollback notification messages to the data rollback queue to send the rollback notification messages to the corresponding target standby library system 2 and the target standby library system 3. After receiving the rollback notification message, the target backup library system 2 and the target backup library system 3 perform corresponding data rollback: for example, when the target operation statement is of an Insert type, the newly added successful data is deleted; if the target operation statement is of the Update type, finding out backup data before updating through the updating condition, and recovering the data; if the target operation statement is of a Delete type, the backup data before deletion is found out through conditions to be restored.

When the data synchronization module of the interceptor determines that the data synchronization mode of the target backup library system 1 is the asynchronous data synchronization mode, the data synchronization module of the interceptor of the main operating library system fills the target operation statement into the asynchronous request message, issues the asynchronous request message into the asynchronous message queue corresponding to the system number corresponding to the target backup library system 1, and sends the message in the asynchronous message queue to the target backup library system 1 according to a preset asynchronous time period, for example, every 30 minutes, wherein the implementation process of the target backup library system 1 after receiving the asynchronous request message is the same as the processing step of the target backup library system 1 receiving the real-time request message, and details are not repeated here.

When the data synchronization module of the interceptor determines that the data synchronization mode of the target backup library system 1 is an asynchronous file synchronization mode, the data synchronization module of the interceptor of the main operation library system firstly creates a file in a local system, and names the created file by using the name of a target data table, the current date and a randomly produced number; writing the target operation statement into the created file; uploading the file including the target operation statement to a file server, and the file server returns a unique identifier of a storage location of the file including the target operation statement, namely, the id _ hash fields in tables 2 and 3. Then, the data synchronization module of the interceptor fills the returned unique identifier into the asynchronous request message, and issues the asynchronous request message into an asynchronous message queue corresponding to the system number corresponding to the target standby library system 1, and sends the message in the asynchronous message queue to the target standby library system 1 according to a preset asynchronous time period, for example, every 30 minutes, where the implementation process after the target standby library system 1 receives the asynchronous request message is the same as the processing step of the target standby library system 1 receiving the real-time request message, and is not described in detail here. The specific implementation process of writing the target operation statement into the created file may be to write all statements in the target operation statement into the created file, or to group the operation statements in the target operation statement according to a certain number to generate at least one file, so as to implement asynchronous synchronization.

The asynchronous processing result aiming at the asynchronous synchronous mode is basically consistent with the result processing of the synchronous mode, and the difference is that the data synchronous mode is updated to real-time synchronous updating after a return message of successful data synchronization is received.

The threshold setting module of the interceptor of the main operation library system collects time-consuming duration from the operation request of the user to the period of successful synchronization detection and the target data volume contained in the synchronization when the execution data synchronization is successfully performed based on the note @ collectionReccateTime () of the AOP of Spring, respectively collects and observes the two values of the three data operation types, can obtain the data time-consuming recording result shown in Table 4, so that the relevant responsible person observes the time-consuming of different data volumes of different operation types for the statistical result shown in Table 4 to adjust the preset threshold, and sends the adjusted preset threshold to the main operation library system through a dynamic configuration item, thereby realizing the subsequent synchronization process.

TABLE 4

op_type

data_count

timeout

The op _ type is used for identifying the operation type, the data _ count is used for identifying the data volume of the corresponding operation type, and the timeout is used for identifying the consumed time of the operation type. The time consumption corresponding to one operation type is the time consumption of analyzing a request, sigma (time consumption for acquiring data + time consumption for processing data) + time consumption for assembling a return result, and sigma (time consumption for acquiring data + time consumption for processing data) is the time consumption of operation statements with data amount corresponding to the operation type.

In the data issuing abnormality monitoring and feedback processing module in the main operation library system, the process of realizing the abnormality alarm can be realized based on a text information platform and a virtual telephone outbound platform, as shown in fig. 9, each new piece of abnormal information with the emergency type of high priority is added, an immediate trigger telephone reminding mode can be adopted to remind a relevant responsible person, if the relevant responsible person processes the abnormal information with high priority, the emergency type in the abnormal information is changed into processed, if the relevant responsible person does not process the abnormal information with high priority, the emergency type in the abnormal recording information is kept as high priority, a trigger logic is put into a delay queue, the virtual telephone outbound platform is triggered again after a certain time delay, for example, 5 minutes, so that the telephone reminding mode is realized to remind the relevant responsible person until the emergency type in the abnormal information record is changed into processed. And aiming at the record with the emergency type of low priority, triggering an abnormal information scanning operation after the data synchronization is finished, namely scanning the record with the emergency type of low priority in the abnormal information record in a timing polling mode, converting the scanned record with the emergency type of low priority in the abnormal information record into a request message format and sending the request message format to a text information platform, and realizing a text information reminding mode until the emergency type in the abnormal information record is changed into processed. The text information platform may be, for example, a short message platform.

The description of the same steps and the same contents in this embodiment as those in other embodiments may refer to the description in other embodiments, and will not be repeated here.

Based on the foregoing embodiments, an embodiment of the present application provides a data synchronization system, which may be applied to the data synchronization method provided in the embodiments corresponding to fig. 1 to 7, and as shown in fig. 10, the data synchronization system 3 may include: a first storage node 31 and at least one second storage node 32, wherein:

a first storage node 31, configured to store a target data table and execute a data synchronization procedure to implement the following steps: if a target operation statement which is specific to the target data table and belongs to the target operation type is detected, determining at least one second storage node which has a correlation with the target data table; wherein, the target operation statement is used for changing the content in the target data table; determining a data synchronization mode of each second storage node; based on each data synchronization mode, sending a target operation statement to a corresponding second storage node so as to realize synchronous updating aiming at a target data table in the first storage node and the second storage node;

a second storage node 32 for receiving a target operation statement; and adopting the target operation statement to execute the updating operation on the target data table corresponding to the target operation statement in the second storage node to obtain an updating result.

In other embodiments of the present application, when the first storage node 31 executes the steps to send the target operation statement to the corresponding second storage node based on each data synchronization manner, the steps may be implemented by:

if each data synchronization mode is a real-time synchronization mode, determining a table building statement corresponding to the target data table;

generating a first request message based on the table building statement and the target operation statement;

sending a first request message to a corresponding second storage node in a real-time synchronization mode; the first request message comprises a target operation statement, and the table building statement in the first request message is used for generating a target data table when the second storage node detects that the target data table does not exist after receiving the first request message.

In other embodiments of the present application, the step performed by the first storage node 31 is further configured to, before sending the target operation statement to the corresponding second storage node based on each data synchronization manner, perform the following steps:

determining a target data volume corresponding to the target operation statement;

correspondingly, based on each data synchronization mode, sending the target operation statement to the corresponding second storage node includes:

and sending the target operation statement to the corresponding second storage node based on the target data volume and each data synchronization mode.

In other embodiments of the present application, when the first storage node 31 executes the steps to send the target operation statement to the corresponding second storage node based on the target data volume and each data synchronization manner, the steps may be implemented by:

if each data synchronization mode is a real-time synchronization mode and the target data volume is smaller than a preset threshold value, determining a table building statement corresponding to the target data table;

generating a second request message based on the table building statement and the target operation statement;

sending a second request message to a corresponding second storage node in a real-time synchronization mode; the second request message includes a target operation statement, and the table building statement included in the second request message is used for generating a target data table when the second storage node detects that the target data table does not exist after receiving the second request message.

In other embodiments of the present application, when the first storage node 31 performs the step of determining the table building statement corresponding to the target data table, the following steps may be implemented:

determining calling information corresponding to the target operation statement;

determining reference field information for defining the structure of the target data table from the calling information;

replacing the reference character in the reference field information with the target character to obtain target field information;

if the target field information comprises table field entity information, traversing the target field information to obtain a target field name;

acquiring a reference field type corresponding to a target field name;

processing the type of the reference field by adopting a target statement rule to obtain a table building statement;

and if the target field information does not comprise table field entity information, obtaining a statement corresponding to the target operation statement from the target field storage area to obtain the table building statement.

In other embodiments of the present application, the first storage node 31 is further configured to perform the following steps:

and if each data synchronization mode is a real-time synchronization mode and the target data volume is greater than or equal to a preset threshold value, updating the corresponding data synchronization mode to be an asynchronous synchronization mode.

if each data synchronization mode is an asynchronous synchronization mode and the target data volume is greater than or equal to a preset threshold value, generating a target operation file based on the target operation statement;

sending the target operation file to a file server;

receiving indication information which is sent by a file server and used for indicating the storage position of a target operation file;

storing the indication information;

generating an acquisition instruction according to a preset asynchronous time period; the obtaining instruction is used for indicating the indication information of the file server to obtain a target operation file, and a target operation statement is obtained to execute corresponding updating operation;

and sending the acquisition instruction to the corresponding second storage node.

In other embodiments of the present application, after the step performed by the first storage node 31 sends the target operation statement to the corresponding second storage node based on each data synchronization manner, the step is further configured to perform the following steps:

receiving an updating result sent by the corresponding second storage node; the updating result is the operation result of the second storage node executing the target operation statement;

and updating the data synchronization mode of the corresponding second storage node based on the updating result.

In other embodiments of the present application, when the first storage node 31 executes the step of updating the data synchronization mode of the corresponding second storage node based on the update result, the following steps may be implemented:

under the condition that the data synchronization mode is a real-time synchronization mode, if the updating result shows that the second storage node executes the target operation statement abnormally, the data synchronization mode is updated to be an asynchronous synchronization mode;

and under the condition that the data synchronization mode is an asynchronous synchronization mode, if the updating result shows that the second storage node successfully executes the target operation statement, the data synchronization mode is updated to be a real-time synchronization mode.

In other embodiments of the present application, after the first storage node 31 performs the step of receiving the update result sent by the corresponding second storage node, the following steps are further performed:

if the updating result indicates that the second storage node executes the target operation statement abnormally, determining the abnormal priority corresponding to the updating result;

and executing the alarm operation corresponding to the abnormal priority.

if the updating result indicates that the second storage node executes the target operation statement abnormally, determining the second storage node of which the updating result indicates that the second storage node executes the target operation statement successfully from at least one second storage node to obtain at least one third storage node;

sending a data rollback control instruction to at least one third storage node; and the data rollback control instruction is used for instructing the third storage node to roll back to the data before the target operation statement is executed.

under the condition that the updating result sent by each second storage node indicates that the second storage node successfully executes the target operation statement, counting the data processing consumed duration of each second storage node to obtain at least one data processing consumed duration;

the preset threshold is updated based on at least one data processing elapsed time.

In other embodiments of the present application, the step executed by the first storage node 31 may be implemented by the following steps if a target operation statement belonging to a target operation type for a target data table is detected, and at least one second storage node having an association relationship with the target data table is determined:

obtaining a mapper declaration parameter and a parameter type parameter in a data service layer;

processing the emitter declaration parameters and the parameter type parameters by adopting a target statement rule to obtain a reference operation statement;

if the operation type of the reference operation statement is the target operation type, determining the target operation statement as the reference operation statement;

at least one second storage node having an association with the target data table is determined.

It should be noted that, when one or more programs in the embodiment of the present application are executed by a processor, the method provided in the embodiment corresponding to fig. 1 to 7 may be implemented, and details are not described herein again.

Based on the foregoing embodiments, embodiments of the present application provide a computer-readable storage medium, referred to as a storage medium for short, where one or more programs are stored in the computer-readable storage medium, and the one or more programs can be executed by one or more processors to implement the data synchronization method implementation process provided in the embodiments corresponding to fig. 1 to 7, and details are not described here again.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable first storage node to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable first storage node, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable first storage node to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable first storage node to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only a preferred embodiment of the present application, and is not intended to limit the scope of the present application.

Claims

1. A data synchronization method applied to a first storage node, the method comprising:

determining a data synchronization mode of each second storage node;

2. The method according to claim 1, wherein said sending the target operation statement to the corresponding second storage node based on each data synchronization manner comprises:

sending the first request message to the corresponding second storage node in the real-time synchronization mode; the first request packet includes the target operation statement, and the table building statement included in the first request packet is used to generate the target data table when detecting that the target data table does not exist after the second storage node receives the first request packet.

3. The method according to claim 1 or 2, wherein before sending the target operation statement to the corresponding second storage node based on each data synchronization manner, the method further comprises:

correspondingly, the sending the target operation statement to the corresponding second storage node based on each data synchronization mode includes:

4. The method according to claim 3, wherein said sending the target operation statement to the corresponding second storage node based on the target data amount and each data synchronization manner comprises:

if each data synchronization mode is the real-time synchronization mode and the target data volume is smaller than a preset threshold value, determining a table building statement corresponding to the target data table;

sending the second request message to the corresponding second storage node in the real-time synchronization mode; the second request packet includes the target operation statement, and the table building statement included in the second request packet is used to generate the target data table when detecting that the target data table does not exist after the second storage node receives the second request packet.

5. The method according to claim 2 or 4, wherein the determining the table building statement corresponding to the target data table comprises:

traversing the target field information to obtain a target field name if the target field information comprises table field entity information;

acquiring a reference field type corresponding to the target field name;

processing the reference field type by adopting a target statement rule to obtain the table building statement;

and if the target field information does not comprise the table field entity information, obtaining the statement corresponding to the target operation statement from a target field storage area to obtain the table building statement.

6. The method of claim 4, further comprising:

and if each data synchronization mode is the real-time synchronization mode and the target data volume is greater than or equal to the preset threshold, updating the corresponding data synchronization mode to be an asynchronous synchronization mode.

7. The method of claim 2 or 6, further comprising:

if each data synchronization mode is an asynchronous synchronization mode and the target data volume is greater than or equal to the preset threshold value, generating a target operation file based on the target operation statement;

sending the target operation file to a file server;

receiving indication information which is sent by the file server and used for indicating the storage position of the target operation file;

storing the indication information;

generating an acquisition instruction according to a preset asynchronous time period; the obtaining instruction is used for indicating that the target operation file is obtained from the indication information of the file server, and the target operation statement is obtained so as to execute the corresponding updating operation;

8. The method according to claim 1, wherein after sending the target operation statement to the corresponding second storage node based on each data synchronization manner, the method further comprises:

receiving an updating result sent by the corresponding second storage node; wherein the update result is an operation result of the second storage node executing the target operation statement;

9. The method according to claim 8, wherein the updating the data synchronization mode of the corresponding second storage node based on the update result comprises:

under the condition that the data synchronization mode is a real-time synchronization mode, if the updating result shows that the second storage node executes the target operation statement abnormally, updating the data synchronization mode to be an asynchronous synchronization mode;

and under the condition that the data synchronization mode is the asynchronous synchronization mode, if the updating result shows that the second storage node successfully executes the target operation statement, updating the data synchronization mode to be the real-time synchronization mode.

10. The method according to claim 8, wherein after receiving the update result sent by the corresponding second storage node, the method further comprises:

if the update result indicates that the second storage node executes the target operation statement abnormally, determining the abnormal priority corresponding to the update result;

and executing the alarm operation corresponding to the abnormal priority.

11. The method according to claim 8, wherein after receiving the update result sent by the corresponding second storage node, the method further comprises:

if the update result indicates that the second storage node executes the target operation statement abnormally, determining a second storage node, which indicates that the second storage node executes the target operation statement successfully, from at least one second storage node to obtain at least one third storage node;

sending a data rollback control instruction to the at least one third storage node; the data rollback control instruction is used for instructing the third storage node to roll back to the data before the target operation statement is executed.

12. The method of claim 8, further comprising:

updating a preset threshold based on at least one of the data processing consumption durations.

13. The method of claim 1, wherein determining at least one second storage node having an association relationship with the target data table if a target operation statement belonging to the target operation type is detected for the target data table comprises:

if the operation type of the reference operation statement is the target operation type, determining that the target operation statement is the reference operation statement;

and determining at least one second storage node having an association relation with the target data table.

14. A data synchronization system, the system comprising: a first storage node and at least one second storage node; wherein:

15. A storage medium having stored thereon a data synchronization program which, when executed by a processor, implements the steps of the data synchronization method of any one of claims 1 to 13.