CN115495527A - Data synchronization management system and method - Google Patents
Data synchronization management system and method Download PDFInfo
- Publication number
- CN115495527A CN115495527A CN202211288936.XA CN202211288936A CN115495527A CN 115495527 A CN115495527 A CN 115495527A CN 202211288936 A CN202211288936 A CN 202211288936A CN 115495527 A CN115495527 A CN 115495527A
- Authority
- CN
- China
- Prior art keywords
- data synchronization
- task
- target
- data
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization management system and a method, wherein the system comprises: the system comprises a data synchronization task configuration module, a storage module, a scheduling module and an execution module; the data synchronization task configuration module is used for determining a data synchronization task to be processed, configuring configuration parameters of the data synchronization task and generating a finally configured data synchronization task; the storage module is used for storing the finally configured data synchronization task to a database; the scheduling module is used for polling the target data synchronization tasks in the database in accordance with the preset period and transmitting the target data synchronization tasks to the execution module; and the execution module is used for performing instantiation operation according to the received target data synchronization task, executing the instantiated data synchronization program, generating a data synchronization result and performing message notification according to the data synchronization result. Due to the fact that various data synchronization functions are integrated, data synchronization strategies are enriched, data synchronization operation thresholds are reduced, and data synchronization efficiency is improved.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to a data synchronization management system and a data synchronization management method.
Background
With the development of the information age, data is more and more applied in various industries, the application range is wider and wider, and the requirements for data synchronization and migration among different environments, different systems and different databases are more and more.
In the prior art, most of the current common data synchronization schemes still have the problems of using scripts, hard coding programs, single functions and the like, and have the defects of high requirement on using technical thresholds and complex operation flow, so that the data synchronization efficiency is reduced.
Disclosure of Invention
The embodiment of the application provides a data synchronization management system and a data synchronization management method. The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview and is intended to neither identify key/critical elements nor delineate the scope of such embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
In a first aspect, an embodiment of the present application provides a data synchronization management system, where the system includes:
the system comprises a data synchronization task configuration module, a storage module, a scheduling module and an execution module; wherein the content of the first and second substances,
the data synchronization task configuration module, the storage module, the scheduling module and the execution module are in communication connection;
the data synchronization task configuration module is used for determining a data synchronization task to be processed, configuring configuration parameters of the data synchronization task and generating a finally configured data synchronization task;
the storage module is used for storing the finally configured data synchronization task to a database so that the scheduling module can schedule at a preset time;
the scheduling module is used for polling target data synchronization tasks in the database within a preset period and transmitting the target data synchronization tasks to the execution module;
and the execution module is used for performing instantiation operation according to the received target data synchronization task, executing the instantiated data synchronization program, generating a data synchronization result and performing message notification according to the data synchronization result.
Optionally, configuring the configuration parameters of the data synchronization task, and generating the finally configured data synchronization task includes:
creating a target executor, a target data source plug-in, a target data conversion plug-in and a target data receiving plug-in of a data synchronization task;
acquiring and configuring scheduling parameters and scheduling periods of data synchronization tasks;
and determining the target actuator, the target data source plug-in, the target data conversion plug-in, the target data receiving plug-in, the scheduling parameter and the scheduling period as configuration parameters of the data synchronization task, and obtaining the finally configured data synchronization task after the configuration is finished.
Optionally, creating a target executor, a target data source plug-in, a target data conversion plug-in, and a target data receiving plug-in of the data synchronization task includes:
determining an actuator of the data synchronization task, and acquiring and configuring execution parameters required by the actuator to obtain a target actuator;
determining a data source plug-in of the data synchronization task, and acquiring and configuring data source information of the data source plug-in to obtain a target data source plug-in;
determining the actuator type of a target actuator, and configuring the target actuator into a target data conversion plug-in according to the actuator type;
and determining a data target plug-in of the data synchronization task, acquiring data target information required to be written by the data synchronization task, and configuring the data target information into the data target plug-in to obtain a target data receiving plug-in.
Optionally, polling a target data synchronization task in the database that meets the preset period includes:
reading a data synchronization task meeting preset conditions in a database according to preset timing task reading time;
generating an example task at the corresponding moment of the next day according to the configuration parameters of the data synchronization task meeting the preset conditions;
marking the instance task as a running state and storing the instance task to a database;
querying and acquiring a target instance task meeting the current moment in real time in a database by adopting a polling method;
and when all timing tasks depended by the target instance task are successfully operated and the target instance task reaches the execution time, determining the target instance task as a target data synchronization task meeting the preset period.
Optionally, when any timing task of all timing tasks depended on by the target instance task fails to run, the target instance task is marked as a waiting dependency state or a waiting running state, and the target instance task is rewritten into the database.
Optionally, the execution module includes an instance module; wherein the content of the first and second substances,
and the instance module is used for performing instantiation operation according to the received target data synchronization task to obtain an instantiated target instance task and packaging the instantiated target instance task into a data synchronization program.
Optionally, performing an instantiation operation according to the received target data synchronization task, including:
acquiring an actuator type according to the target instance task, and constructing the actuator type by using a reflection strategy;
setting a data source plug-in, a conversion plug-in and a data target plug-in of an actuator class to generate an actuator class synchronization task;
and determining the executor class synchronization task as the instantiated target instance task.
Optionally, executing the instantiated data synchronization program to generate a data synchronization result, including:
executing the executor class synchronization task to generate an execution result;
when the execution result is execution failure and the execution failure times exceed a preset failure time threshold, marking the target instance task as an execution failure state, rewriting the target instance task into the database, and determining the execution failure result as a data synchronization result; alternatively, the first and second electrodes may be,
when the execution result is successful, marking the target instance task as a successful execution state, rewriting the target instance task into the database, calling the depended downstream instance task, continuously executing the steps of inquiring in the database in real time by adopting a polling method and acquiring the target instance task meeting the current moment, and simultaneously determining the successful execution result as a data synchronization result;
and pushing the data synchronization result to a message notification module.
Optionally, the system further comprises a message notification module;
and the message notification module is used for determining the received data synchronization result as an example operation result and notifying the example operation result of the data synchronization operation result.
In a second aspect, an embodiment of the present application provides a data synchronization management method, where the method includes:
the data synchronization task configuration module determines a data synchronization task to be processed, configures configuration parameters of the data synchronization task, and generates a finally configured data synchronization task;
the storage module stores the finally configured data synchronization task to a database for the scheduling module to schedule at a preset time;
the scheduling module polls a target data synchronization task which is in accordance with a preset period in the database and transmits the target data synchronization task to the execution module;
the execution module performs instantiation operation according to the received target data synchronization task, executes the instantiated data synchronization program, generates a data synchronization result, and performs message notification according to the data synchronization result.
The technical scheme provided by the embodiment of the application can have the following beneficial effects:
in the embodiment of the application, firstly, a data synchronization task configuration module determines a data synchronization task to be processed, configures configuration parameters of the data synchronization task, generates a finally configured data synchronization task, then, a storage module stores the finally configured data synchronization task in a database for a scheduling module to schedule at a preset time, then, the scheduling module polls a target data synchronization task in the database according with a preset period, transmits the target data synchronization task to an execution module, and finally, the execution module performs instantiation operation according to the received target data synchronization task, executes an instantiated data synchronization program, generates a data synchronization result, and performs message notification according to the data synchronization result. The method and the device integrate various data synchronization functions, so that data synchronization strategies are enriched, the data synchronization operation threshold is reduced, and meanwhile, the scheduling module automatically polls the data synchronization tasks meeting the requirements in the database to automatically perform data synchronization operation, so that the data synchronization efficiency is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
Fig. 1 is a system structural diagram of a data synchronization management system according to an embodiment of the present application;
FIG. 2 is a schematic block diagram illustrating a process flow of a data synchronization task configuration module provided in the present application;
FIG. 3 is a flowchart illustrating the overall processing of a scheduling module and an execution module provided herein;
fig. 4 is a schematic flowchart of a data synchronization management method according to an embodiment of the present application.
Detailed Description
The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.
It should be understood that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
In the description of the present invention, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
The present application provides a data synchronization management system and method to solve the above-mentioned problems in the related art. In the technical scheme provided by the application, various data synchronization functions are integrated, so that data synchronization strategies are enriched, the data synchronization operation threshold is reduced, meanwhile, the scheduling module automatically polls the data synchronization tasks meeting the requirements in the database to automatically perform data synchronization operation, and therefore the data synchronization efficiency is improved, and detailed description is provided by adopting an exemplary embodiment.
Referring to fig. 1, a system structure diagram of a data synchronization management system is provided in an embodiment of the present application. The system comprises: the system comprises a data synchronization task configuration module, a storage module, a scheduling module and an execution module; the data synchronization task configuration module, the storage module, the scheduling module and the execution module are in communication connection.
In the embodiment of the application, the data synchronization task configuration module is used for determining a data synchronization task to be processed, configuring configuration parameters of the data synchronization task, and generating a finally configured data synchronization task; the storage module is used for storing the finally configured data synchronization task to a database so that the scheduling module can schedule at a preset time; the scheduling module is used for polling target data synchronization tasks in the database within a preset period and transmitting the target data synchronization tasks to the execution module; and the execution module is used for performing instantiation operation according to the received target data synchronization task, executing the instantiated data synchronization program, generating a data synchronization result and performing message notification according to the data synchronization result.
In the embodiment of the present application, a data synchronization task configuration module is specifically described, for example, when configuring configuration parameters of a data synchronization task and generating a finally configured data synchronization task, a target executor, a target data source plug-in, a target data conversion plug-in, and a target data receiving plug-in of the data synchronization task are first created, then scheduling parameters and a scheduling period of the data synchronization task are obtained and configured, then the target executor, the target data source plug-in, the target data conversion plug-in, the target data receiving plug-in, the scheduling parameters, and the scheduling period are determined as configuration parameters of the data synchronization task, and the finally configured data synchronization task is obtained after configuration is completed.
Further, when a target executor, a target data source plugin, a target data conversion plugin and a target data receiving plugin of the data synchronization task are created, firstly, an executor of the data synchronization task is determined, execution parameters required by the executor are obtained and configured, the target executor is obtained, then, the data source plugin of the data synchronization task is determined, data source information of the data source plugin is obtained and configured, the target data source plugin is obtained, then, the executor type of the target executor is determined, the target executor is configured into the target data conversion plugin according to the executor type, finally, the data target plugin of the data synchronization task is determined, data target information required to be written by the data synchronization task is obtained, and the data target information is configured into the data target plugin, so that the target data receiving plugin is obtained.
In a possible implementation manner, for example, as shown in fig. 2, fig. 2 is a schematic block diagram of a processing flow of a data synchronization task configuration module provided in the present application, an executor is first selected, it may be determined that data synchronization is performed by a spark frame, a datax frame, or an executor corresponding to another frame according to an implementation form of a data synchronization task, and after the executor is determined, an executor parameter may be configured according to a specific implementation frame, for example, when the executor corresponds to a spark frame, the spark memory size, the cpu core number, and the like may be configured, so as to maximize parallelism. After the configuration of the parameters of the actuator is completed, a data source plug-in of a data synchronization task needs to be determined, and the parameters of the data synchronization task are configured, wherein the parameters include the address of a data source, account information and other related information of the data source. After the configuration of the data source plug-in is finished, when the target data conversion plug-in is available, for example, an executor realized by a spark framework, the target data conversion plug-in is configured into a corresponding conversion sql statement, and the sql statement is determined as the target data conversion plug-in; for example, in the case of an executor implemented by a datax framework, the executor is configured as a conversion method that needs to be used by a corresponding field, and the conversion method is determined as a target data conversion plug-in. After the target data conversion plug-in is generated, a synchronous data target plug-in needs to be selected, and parameters of the synchronous data target plug-in are configured, for example, data target related information needing to be written in by a data synchronization task is configured, the information includes a target address of written data, account information, table information and the like, and a target data receiving plug-in can be obtained after configuration is finished. After the target data receiving plug-in is obtained, the scheduling parameters and the period parameters of the data synchronization task may be configured, for example, the scheduling parameters and the scheduling period information of the data synchronization task are configured to determine the execution period of the data synchronization task and determine the scheduling parameters of the data synchronization task. After the configuration of the scheduling parameters and the period parameters of the data synchronization task is completed, the finally configured data synchronization task can be stored in a database so that the scheduling module can schedule the data synchronization task at the configured running time.
For example, in an actual application scenario, taking a data synchronization task as live data synchronization to MySQL data as an example, the data synchronization task configuration module explains the whole configuration of the data synchronization task, and first selects a spark actuator, and then configures execution parameters required by the spark actuator, where the parameters include 1g of memory and 5 execution or run-client execution modes. And then selecting a data source plug-in hive, configuring hive data source related information, including configuring hive url addresses, configuring hive table data synchronization condition select id, name from table1 where dt = $ { day }, configuring a data source as a temporary view tmp _ a, and configuring the data source as the temporary view tmp _ a. When the data conversion plug-in is reconfigured, for example, as an executor implemented by spark framework, it is configured to have the corresponding conversion sql statement as select id, md5 (name) name _ md5 from temp _ a. And secondly, selecting a data target plug-in mysql, configuring relevant information of the mysql data target, such as configuring an msyql url address, configuring account information and configuring a data target table2, and obtaining a target data receiving plug-in after configuration is finished. And reconfiguring the scheduling parameters and the period parameters, for example, configuring the scheduling parameters day = temp, depending on the task a in the same period, performing error retry for 2 times at an interval of 3 minutes, executing the configuration period parameters for 6 minutes and 30 minutes every day, obtaining a finally configured data synchronization task after all configuration is finished, and finally storing the finally configured data synchronization task in a database to store the timing task.
In the embodiment of the present application, a scheduling module is specifically described, for example, when a polling database conforms to a target data synchronization task in a preset period, first, a data synchronization task meeting a preset condition is read from the database according to a preset timing task reading time, then, an instance task at a time corresponding to the next day is generated according to configuration parameters of the data synchronization task meeting the preset condition, next, the instance task is marked as a running state and is stored in the database, then, a polling method is adopted to query and obtain the target instance task meeting the current time in the database in real time, and finally, when all timing tasks depended by the target instance task are successfully run and the target instance task reaches an execution time, the target instance task is determined as the target data synchronization task conforming to the preset period.
Further, when any timing task of all timing tasks depended by the target instance task fails to run, the target instance task is marked to be in a waiting dependency state or a waiting running state, and the target instance task is rewritten into the database.
In a possible implementation manner, for example, as shown in fig. 3, fig. 3 is an overall processing flow diagram of a scheduling module and an execution module provided in the present application, in the scheduling module, at 23 days, first, effective timing tasks in a database are read, then, an instance task corresponding to a time of the next day is generated according to a period parameter of each timing task, then, the generated instance is marked as a waiting-to-run state, and is stored in the database, then, the database is polled to query, an instance task whose execution time is within the polling period is obtained, and finally, whether all the timing tasks depended on by the instance task are successfully run and reach the execution time is determined, if any one of the tasks depended on by the instance task is not successfully run, the instance task is marked as a waiting-to-depend state, and then, the instance task database is written back; if the example tasks are successfully operated but the execution time is not reached, the task operation is not carried out; and if the timing tasks depended by the instance tasks run successfully and the execution time is reached, continuing to perform the processing flow of the execution module.
For example, in an actual application scenario, the data synchronization task configuration module takes the data synchronization task as live data to synchronize to MySQL data as an example, and describes a processing process of the scheduling module, the scheduling module reads the timing task when the current day (20220101) 23 is used to generate an example task 1 of 30 minutes at 6 days (20220102) this time, marks the example task 1 as a non-running state, stores the example task 1 to the storage system, and performs a polling function of the scheduling module, reads the example task 1 from 20220102 days 6 minutes, determines whether the example task 1 depends on the example task a (the same-period example of the timing task a) to be successfully executed, marks the example task 1 as a waiting-dependent state if the execution is unsuccessful, determines whether the current time of the system is greater than 6 minutes 30 minutes if the execution is successful, ends if the execution is smaller, and performs a processing process of the execution module if the execution is greater.
In the embodiment of the present application, an execution module is specifically described, where the execution module includes an instance module; the instance module is used for performing instantiation operation according to the received target data synchronization task to obtain an instantiated target instance task and packaging the instantiated target instance task into a data synchronization program.
Specifically, when instantiation is performed according to a received target data synchronization task, firstly, an actuator type is obtained according to the target instance task, an actuator class is constructed by using a reflection strategy, then, a data source plug-in, a conversion plug-in and a data target plug-in of the actuator class are set, an actuator class synchronization task is generated, and finally, the actuator class synchronization task is determined as the instantiated target instance task.
Specifically, when the execution module executes the instantiated data synchronization program and generates a data synchronization result, the execution module firstly executes the executor class synchronization task to generate an execution result, when the execution result is an execution failure and the execution failure frequency exceeds a preset failure frequency threshold, the target instance task is marked as an execution failure state, the target instance task is rewritten into the database, and the execution failure result is determined as the data synchronization result; or when the execution result is successful, marking the target instance task as a successful execution state, rewriting the target instance task into the database, calling the depended downstream instance task, continuously executing the steps of inquiring in the database in real time by adopting a polling method and acquiring the target instance task meeting the current moment, and simultaneously determining the successful execution result as a data synchronization result; and finally, pushing the data synchronization result to a message notification module.
In a possible implementation manner, for example, as shown in fig. 3, fig. 3 is an overall processing flow diagram of a scheduling module and an execution module provided in the present application, in the execution module, first obtaining an actuator type, generating an actuator class by using reflection, then setting the actuator to configure a data source plug-in, a conversion plug-in, and a data target plug-in, then calling the actuator synchronization method, executing the actuator class synchronization task, and finally executing an instance program generated by the module operation, where a spark actuator submits the instance program to yarn for clustered operation, and a datax program submits the instance program to a local computer for stand-alone operation, specifically, reading a data source, and performing operations such as corresponding data conversion and filtering, and when an execution result is an execution failure and the execution failure number exceeds a preset failure number threshold, marking a target instance task as an execution failure state, rewriting the target instance task into a database, and determining the execution failure result as a data synchronization result; or when the execution result is successful, marking the target instance task as a successful execution state, rewriting the target instance task into the database, calling the depended downstream instance task, continuously executing the steps of inquiring in the database in real time by adopting a polling method and acquiring the target instance task meeting the current moment, and simultaneously determining the successful execution result as a data synchronization result; and finally, pushing the data synchronization result to a message notification module.
For example, in an actual application scenario, a data synchronization task configuration module describes a processing process of an execution module by taking an example that a data synchronization task is live data synchronized to MySQL data as an example, an instance module in the execution module performs instantiation operation on the data synchronization task according to input related information (an executor, a synchronization data source, a conversion operation, and a synchronization data target), and encapsulates the data synchronization task into a data synchronization program, instantiates a spark executor class by using reflection according to the configured spark executor, sets related parameters (a data source plug-in, a conversion plug-in, and a data target plug-in) of the spark executor class, and encapsulates the plug-in combination into a spark program code (fusing spark reading, conversion, and writing operations) according to the set parameters. The execution module starts a corresponding data synchronization program:
1. starting a spark program according to the configured spark parameters;
2. reading the partition data corresponding to the hive table and registering the partition data as a temporary table
3. Executing spark translation statement
4. Writing the converted data into the MySQL specified table by utilizing the spark integrated MySQL writing function
5. If the execution of the steps fails, judging whether the current failure times reach an error time threshold value for 2 times, and if not, sleeping for configuration time for 3 minutes. After 3 minutes, steps 1-5 are re-executed until the run is successful or an error threshold 2 is reached.
If the execution is successful, marking the instance as a successful state, and calling a timing task depending on the instance task 1 to continuously execute the corresponding step; if the execution fails, the instance is marked as a failed state.
In the embodiment of the application, the system further comprises a message notification module; and the message notification module is used for determining the received data synchronization result as an example operation result and notifying the example operation result of the data synchronization operation result.
In a possible implementation manner, after each instance program is executed, the running result of the instance program is notified of the data synchronization running result through the message module. E.g., synchronization program execution is complete, and an associated execution completion notification is sent.
It should be noted that the method and the device can be used in common data synchronization scenarios, and based on page operations, the method and the device screen data synchronization actuator components, screen data synchronization source components matched with the selected actuator components according to the selected actuator components, and fill in data synchronization source related information, then (optionally) screen related conversion operation components required by synchronization data, such as encryption, conversion, and masking, screen data synchronization target components matched with the actuator again, fill in data synchronization target related information, and finally configure scheduling information (such as once, hour, day, and the like) of the data synchronization task. According to the screening component, the data synchronization program chain task is assembled by the back-end service and is persisted. And the scheduling module polls constantly to acquire a data synchronization task which accords with the execution period, and an actuator assembly, a synchronization source assembly, a conversion operation assembly and a synchronization target assembly which are configured according to the timing task are packaged into a data synchronization program to form the data synchronization example. And then starting the data synchronization instance to perform corresponding data synchronization work. According to the technical scheme, the use threshold of data synchronization is reduced by packaging different data synchronization actuators, various data synchronization source components, data synchronization conversion components and data synchronization target components, page operation is provided, and data synchronization with different functions can be completed through different actuator components, wherein the data synchronization comprises but is not limited to traditional database synchronization, big data synchronization, text file synchronization, folder synchronization and the like.
In the embodiment of the application, firstly, a data synchronization task configuration module determines a data synchronization task to be processed, configures configuration parameters of the data synchronization task, generates a finally configured data synchronization task, then, a storage module stores the finally configured data synchronization task to a database for a scheduling module to schedule at a preset time, then, the scheduling module polls a target data synchronization task in the database according with a preset period, transmits the target data synchronization task to an execution module, and finally, the execution module performs instantiation operation according to the received target data synchronization task, executes an instantiated data synchronization program, generates a data synchronization result, and performs message notification according to the data synchronization result. The method and the system integrate various data synchronization functions, so that data synchronization strategies are enriched, the data synchronization operation threshold is reduced, and meanwhile, the scheduling module automatically polls the data synchronization tasks meeting the requirements in the database to automatically perform data synchronization operation, so that the data synchronization efficiency is improved.
Referring to fig. 4, a flow chart of a data synchronization management method is provided in the embodiment of the present application. As shown in fig. 4, the method of the embodiment of the present application may include the following steps:
s101, a data synchronization task configuration module determines a data synchronization task to be processed, configures configuration parameters of the data synchronization task, and generates a finally configured data synchronization task;
s102, the storage module stores the finally configured data synchronization task to a database so that a scheduling module can schedule at a preset time;
s103, the scheduling module polls a target data synchronization task which is in accordance with a preset period in the database and transmits the target data synchronization task to the execution module;
and S104, the execution module performs instantiation operation according to the received target data synchronization task, executes the instantiated data synchronization program, generates a data synchronization result, and performs message notification according to the data synchronization result.
In the embodiment of the application, firstly, a data synchronization task configuration module determines a data synchronization task to be processed, configures configuration parameters of the data synchronization task, generates a finally configured data synchronization task, then, a storage module stores the finally configured data synchronization task to a database for a scheduling module to schedule at a preset time, then, the scheduling module polls a target data synchronization task in the database according with a preset period, transmits the target data synchronization task to an execution module, and finally, the execution module performs instantiation operation according to the received target data synchronization task, executes an instantiated data synchronization program, generates a data synchronization result, and performs message notification according to the data synchronization result. The method and the system integrate various data synchronization functions, so that data synchronization strategies are enriched, the data synchronization operation threshold is reduced, and meanwhile, the scheduling module automatically polls the data synchronization tasks meeting the requirements in the database to automatically perform data synchronization operation, so that the data synchronization efficiency is improved.
The present invention also provides a computer readable medium, on which program instructions are stored, and the program instructions, when executed by a processor, implement the data synchronization management method provided by the above-mentioned method embodiments.
The present invention also provides a computer program product containing instructions which, when run on a computer, cause the computer to perform the data synchronization management method of the above-described method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above may be implemented by a computer program to instruct associated hardware, and the program for data synchronization management may be stored in a computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory or a random access memory.
The above disclosure is only for the purpose of illustrating the preferred embodiments of the present application and is not to be construed as limiting the scope of the present application, so that the present application is not limited thereto, and all equivalent variations and modifications can be made to the present application.
Claims (10)
1. A data synchronization management system, the system comprising:
the system comprises a data synchronization task configuration module, a storage module, a scheduling module and an execution module; wherein the content of the first and second substances,
the data synchronization task configuration module, the storage module, the scheduling module and the execution module are in communication connection;
the data synchronization task configuration module is used for determining a data synchronization task to be processed, configuring configuration parameters of the data synchronization task and generating a finally configured data synchronization task;
the storage module is used for storing the finally configured data synchronization task to a database so that the scheduling module can schedule the data synchronization task at a preset time;
the scheduling module is used for polling target data synchronization tasks in a database according with a preset period and transmitting the target data synchronization tasks to the execution module;
the execution module is used for performing instantiation operation according to the received target data synchronization task, executing the instantiated data synchronization program, generating a data synchronization result, and performing message notification according to the data synchronization result.
2. The data synchronization management system of claim 1, wherein configuring the configuration parameters of the data synchronization task to generate the finally configured data synchronization task comprises:
creating a target executor, a target data source plug-in, a target data conversion plug-in and a target data receiving plug-in of the data synchronization task;
acquiring and configuring scheduling parameters and scheduling periods of the data synchronization tasks;
and determining the target actuator, the target data source plug-in, the target data conversion plug-in, the target data receiving plug-in, the scheduling parameter and the scheduling period as the configuration parameters of the data synchronization task, and obtaining the finally configured data synchronization task after the configuration is finished.
3. The method of claim 2, wherein creating a target executor, a target data source plug-in, a target data transformation plug-in, a target data receiving plug-in for the data synchronization task comprises:
determining an actuator of the data synchronization task, and acquiring and configuring execution parameters required by the actuator to obtain a target actuator;
determining a data source plug-in of the data synchronization task, and acquiring and configuring data source information of the data source plug-in to obtain a target data source plug-in;
determining the actuator type of the target actuator, and configuring the target actuator into a target data conversion plug-in according to the actuator type;
and determining a data target plug-in of the data synchronization task, acquiring data target information required to be written by the data synchronization task, and configuring the data target information into the data target plug-in to obtain a target data receiving plug-in.
4. The data synchronization management system of claim 1, wherein the polling database conforms to the target data synchronization task within a preset period, and comprises:
reading a data synchronization task meeting preset conditions in a database according to preset timing task reading time;
generating an example task at the corresponding moment of the next day according to the configuration parameters of the data synchronization task meeting the preset conditions;
marking the instance task as a running state and storing the instance task to a database;
querying and acquiring a target instance task meeting the current moment in real time in a database by adopting a polling method;
and when all timing tasks depended by the target instance task are successfully operated and the target instance task reaches the execution time, determining the target instance task as a target data synchronization task in accordance with a preset period.
5. A data synchronization management system according to claim 4,
and when any timing task of all timing tasks depended by the target instance task fails to run, marking the target instance task as a waiting dependency state or a waiting running state, and rewriting the target instance task into a database.
6. A data synchronization management system according to claim 1,
the execution module comprises an instance module; wherein the content of the first and second substances,
and the instance module is used for performing instantiation operation according to the received target data synchronization task to obtain an instantiated target instance task and packaging the instantiated target instance task into a data synchronization program.
7. The data synchronization management system of claim 4, wherein the instantiating operation according to the received target data synchronization task comprises:
acquiring an actuator type according to the target instance task, and constructing the actuator type by using a reflection strategy;
setting a data source plug-in, a conversion plug-in and a data target plug-in of the actuator class to generate an actuator class synchronization task;
and determining the executor class synchronization task as an instantiated target instance task.
8. The data synchronization management system of claim 7, wherein the executing the instantiated data synchronization program generates a data synchronization result, comprising:
executing the executor class synchronization task to generate an execution result;
when the execution result is execution failure and the execution failure times exceed a preset failure time threshold, marking the target instance task as an execution failure state, rewriting the target instance task into a database, and determining the execution failure result as a data synchronization result; alternatively, the first and second electrodes may be,
when the execution result is successful, marking the target instance task as a successful execution state, rewriting the target instance task into a database, calling a depended downstream instance task, continuously executing the step of querying in the database in real time by adopting a polling method and acquiring the target instance task meeting the current moment, and simultaneously determining the successful execution result as a data synchronization result;
and pushing the data synchronization result to a message notification module.
9. The data synchronization management system of claim 7, further comprising a message notification module;
and the message notification module is used for determining the received data synchronization result as an example operation result and notifying the example operation result of the data synchronization operation result.
10. A method for managing data synchronization, the method comprising:
the data synchronization task configuration module determines a data synchronization task to be processed, configures configuration parameters of the data synchronization task, and generates a finally configured data synchronization task;
the storage module stores the finally configured data synchronization task to a database so that the scheduling module can schedule at a preset time;
the scheduling module polls a target data synchronization task which is in accordance with a preset period in a database and transmits the target data synchronization task to the execution module;
and the execution module performs instantiation operation according to the received target data synchronization task, executes the instantiated data synchronization program, generates a data synchronization result, and performs message notification according to the data synchronization result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211288936.XA CN115495527A (en) | 2022-10-20 | 2022-10-20 | Data synchronization management system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211288936.XA CN115495527A (en) | 2022-10-20 | 2022-10-20 | Data synchronization management system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115495527A true CN115495527A (en) | 2022-12-20 |
Family
ID=84474924
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211288936.XA Pending CN115495527A (en) | 2022-10-20 | 2022-10-20 | Data synchronization management system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115495527A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116521636A (en) * | 2023-05-16 | 2023-08-01 | 三峡科技有限责任公司 | Automatic synchronization method and system for operation data |
-
2022
- 2022-10-20 CN CN202211288936.XA patent/CN115495527A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116521636A (en) * | 2023-05-16 | 2023-08-01 | 三峡科技有限责任公司 | Automatic synchronization method and system for operation data |
| CN116521636B (en) * | 2023-05-16 | 2023-11-28 | 三峡科技有限责任公司 | Automatic synchronization method and system for operation data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108280023B (en) | Task execution method and device and server | |
| CN110825420A (en) | Configuration parameter updating method, device, equipment and storage medium for distributed cluster | |
| CN109815291B (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN109298924B (en) | Timing task management method, computer readable storage medium and terminal device | |
| CN111400407A (en) | Data synchronization method and device, storage medium and electronic device | |
| CN103927314A (en) | Data batch processing method and device | |
| CN113656503A (en) | Data synchronization method, device and system and computer readable storage medium | |
| CN115495527A (en) | Data synchronization management system and method | |
| CN111784318A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN115185787A (en) | Method and device for processing transaction log | |
| CN110780981B (en) | Task scheduling management system and method for hadoop big data platform | |
| CN113468143A (en) | Data migration method, system, computing device and storage medium | |
| CN113485814A (en) | Batch task scheduling method and device | |
| CN114281757A (en) | Database migration method and system and computer readable storage medium | |
| CN114090198A (en) | Distributed task scheduling method and device, electronic equipment and storage medium | |
| CN113419872A (en) | Application system interface integration system, integration method, equipment and storage medium | |
| CN116414914A (en) | Data synchronization method and device, processor and electronic equipment | |
| US20180341521A1 (en) | Managing job schedules | |
| CN111831394B (en) | Method and device for deploying stateful pod by kubernetes | |
| CN114840586A (en) | Data conversion method, equipment, server and computer readable storage medium | |
| CN115185659A (en) | Task instance running method and device and computer readable storage medium | |
| CN112015798B (en) | Data processing method and device for guaranteeing data non-repetition and delay monitoring | |
| CN110674214B (en) | Big data synchronization method, device, computer equipment and storage medium | |
| CN114020368A (en) | Information processing method and device based on state machine and storage medium | |
| CN111679899A (en) | Task scheduling method, device, platform equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |