CN115146000A - Database data synchronization method and device, electronic equipment and storage medium - Google Patents
Database data synchronization method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115146000A CN115146000A CN202210700258.7A CN202210700258A CN115146000A CN 115146000 A CN115146000 A CN 115146000A CN 202210700258 A CN202210700258 A CN 202210700258A CN 115146000 A CN115146000 A CN 115146000A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- database
- data synchronization
- written
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a database data synchronization method, a database data synchronization device, electronic equipment and a storage medium. The method comprises the following steps: determining a target database to be read and a target database to be written corresponding to the target data synchronization task according to preconfigured target data synchronization task information; calling a reading interface of a target database to be read, reading first data of the target database to be read, and converting a data structure corresponding to the first data into a data structure corresponding to data in the target database to be written to obtain second data; and calling a write interface of the target database to be written, and writing the second data into the target database to be written. By operating the technical scheme provided by the embodiment of the invention, the problem that when the databases to be synchronized are of multiple types, corresponding programs need to be written according to the databases of different types and the synchronization is carried out for multiple times, the synchronization efficiency is reduced, and the beneficial effects of improving the applicability and efficiency of data synchronization are achieved.
Description
Technical Field
The present invention relates to computer technologies, and in particular, to a database data synchronization method and apparatus, an electronic device, and a storage medium.
Background
The development of the internet technology brings great convenience to various fields of life of people, the application of user-oriented services is countless, and the application can accurately provide personalized accurate services for different types of users by relying on a big data technology. The big data platform has the operational capability of mass data, and can perform structured processing on data of different systems by relying on big data technology components, so that cross-platform conversion and application of the data are realized.
When there is a need to synchronize data in different databases, in the prior art, data to be synchronized is usually a single-type database, and when the databases to be synchronized are of multiple types, corresponding programs need to be written according to the different types of databases and multiple times of synchronization are performed, so that synchronization efficiency is reduced.
Disclosure of Invention
The invention provides a database data synchronization method, a database data synchronization device, electronic equipment and a storage medium, and aims to improve the applicability and efficiency of data synchronization.
According to an aspect of the present invention, there is provided a database data synchronization method, including:
determining a target database to be read and a target database to be written corresponding to the target data synchronization task according to preconfigured target data synchronization task information;
calling a reading interface of the target database to be read according to the target data synchronization task information, reading first data of the target database to be read, and converting a data structure corresponding to the first data into a data structure corresponding to data in the target database to be written to obtain second data;
and calling a writing interface of the target database to be written according to the target data synchronization task information, and writing the second data into the target database to be written.
According to another aspect of the present invention, there is provided a database data synchronization apparatus, comprising:
the database determining module is used for determining a target database to be read and a target database to be written corresponding to the target data synchronization task according to the preconfigured target data synchronization task information;
the data reading module is used for calling a reading interface of the target database to be read according to the target data synchronization task information, reading first data of the target database to be read, and converting a data structure corresponding to the first data into a data structure corresponding to data in the target database to be written to obtain second data;
and the data writing module is used for calling a writing interface of the target database to be written according to the target data synchronization task information and writing the second data into the target database to be written.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the database data synchronization method according to any of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the database data synchronization method according to any one of the embodiments of the present invention when the computer instructions are executed.
According to the technical scheme of the embodiment of the invention, a target database to be read and a target database to be written which correspond to a target data synchronization task are determined according to the pre-configured target data synchronization task information; calling a reading interface of the target database to be read according to the target data synchronization task information, reading first data of the target database to be read, and converting a data structure corresponding to the first data into a data structure corresponding to data in the target database to be written to obtain second data; and calling a writing interface of the target database to be written according to the target data synchronization task information, and writing the second data into the target database to be written. The problem that when the databases to be synchronized are of multiple types, corresponding programs need to be written according to the databases of different types and multiple times of synchronization are carried out, and the synchronization efficiency is reduced is solved, and the beneficial effects of improving the applicability and efficiency of data synchronization are achieved.
It should be understood that the statements in this section are not intended to identify key or critical features of the embodiments of the present invention, nor are they intended to limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
Fig. 1 is a flowchart of a database data synchronization method according to an embodiment of the present invention;
fig. 2 is a flowchart of a database data synchronization method according to a second embodiment of the present invention;
fig. 3 is a schematic diagram of splitting and allocating a data synchronization overall task according to a second embodiment of the present invention;
FIG. 4 is a diagram illustrating a multi-source data synchronization engine according to a second embodiment of the present invention;
fig. 5 is a schematic structural diagram of a database data synchronization apparatus according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device for implementing an embodiment of the invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," "target," and the like in the description and claims of the present invention and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example one
Fig. 1 is a flowchart of a database data synchronization method according to an embodiment of the present invention, where this embodiment is applicable to a case where data in databases with different structures are synchronized by a pre-constructed multi-source data synchronization engine, and the method may be executed by a database data synchronization apparatus according to an embodiment of the present invention, and the apparatus may be implemented in a software and/or hardware manner. Referring to fig. 1, the database data synchronization method provided in this embodiment includes:
s110, determining a target database to be read and a target database to be written corresponding to the target data synchronization task according to the preset target data synchronization task information.
The target data synchronization task information is information contained in a target data synchronization task which needs to be executed at this time, and examples of the information include task starting time and an alarm notification object; and technical parameters related to the task, such as maximum number of nodes, channel capacity, maximum delay, etc.; but also the source database of the data and the information of the database into which the data needs to be imported. For example, if the source database is MySQL, the target data synchronization task information may include the relationship between the table related to the source database and other tables, such as links and field points.
And determining databases related to the target data synchronization task according to the target data synchronization task information, namely a target database to be read and a target database to be written corresponding to the target data synchronization task, wherein the target database to be read is a source database of data, and the target database to be written is a database into which the data needs to be imported. The number of the target databases to be read and the target databases to be written can be multiple, and when the number is multiple, the types of the databases can be the same or different.
S120, calling a reading interface of the target database to be read according to the target data synchronization task information, reading first data of the target database to be read, and converting a data structure corresponding to the first data into a data structure corresponding to data in the target database to be written to obtain second data.
The reading interface of the target database to be read may be pre-constructed according to the target database to be read, for example, an Oracle database reading interface, a MySQL database reading interface, an HBase database reading interface, a DB2 database reading interface, a SyBase2 database reading interface, and the like. The method comprises the steps of determining a reading interface of a target database to be read through target data synchronization task information, calling the reading interface through a multi-source data synchronization engine, and reading first data of the target database to be read, wherein the first data are data which need to be read from the target database to be read and are synchronized to the target database to be written, can be all data in the target database to be read, and can also be partial data in the target database to be read.
And converting a data structure corresponding to the first data into a data structure corresponding to the data to be written into the target database through a plurality of automatic conversion functions aiming at different types of database structured data types integrated in the target data synchronization task information in advance, and recording the data after structure conversion as second data.
S130, calling a writing interface of the target database to be written according to the target data synchronization task information, and writing the second data into the target database to be written.
The writing interface of the target database to be written may be pre-constructed according to the target database to be written, for example, an Oracle database writing interface, a MySQL database writing interface, an HBase database writing interface, a DB2 database writing interface, a SyBase2 database writing interface, and the like. And determining a writing interface of the target database to be written through the target data synchronization task information, calling the writing interface through the multi-source data synchronization engine, and writing second data into the target database to be written, so that data content corresponding to the second data exists in the target database to be read and the target database to be written at the same time, and realizing data synchronization of the target database to be read and the target database to be written.
Optionally, after the second data is written into the target to-be-written database, the execution condition of the target data synchronization task may be counted and presented in a report form, so that a user can know the second data timely and thoroughly.
The report content may include a task starting time, a task ending time, a task total consumed time, a task average flow, a task writing speed, a total number of reading records, a total number of reading failures, and the like.
In this embodiment, optionally, invoking a reading interface of the target to-be-read database according to the target data synchronization task information, and reading the first data of the target to-be-read database includes:
calling a reading interface of the target database to be read according to the target data synchronization task information, and reading the first data to a preset cache position;
correspondingly, calling a write interface of the target database to be written according to the target data synchronization task information, and writing the second data into the target database to be written, including:
and calling a write-in interface of the target database to be written according to the target data synchronization task information, and writing the second data in the preset cache position into the target database to be written.
The method comprises the steps of calling a reading interface of a target database to be read, reading first data to a preset cache position, determining a data reading processing mode according to target data synchronization task information, for example, processing data reading of the same database in parallel through a plurality of nodes, establishing data synchronization task long connection according to the target data synchronization task information, reading the first data to the data cache position in batches according to the data reading processing mode, and preparing to write the first data to the target database to be written. And performing structured conversion on the first data in the cache to form a structure consistent with a corresponding data table of the target database to be written, writing the structure into the target database to be written in batches, and disconnecting the data synchronization task length connection after all the data are written.
By reading the first data to the preset cache position and writing the second data in the preset cache position into the target database to be written, the structure conversion can be performed on other first data in the preset cache position while writing the second data, and the data synchronization efficiency is improved.
In this embodiment, optionally, the method further includes:
determining whether a synchronization exception condition exists; if yes, generating synchronous abnormal information to record the synchronous abnormal condition;
after the second data is written into the target database to be written, determining a first abnormal repairing mode corresponding to the synchronous abnormal condition according to the synchronous abnormal information;
and repairing the synchronous abnormal condition according to the first abnormal repairing mode.
Confirming whether a synchronization abnormal condition exists in the data synchronization process; if the abnormal synchronous code exists, the abnormal synchronous information is generated to record the abnormal synchronous condition, and the recording mode can be that the abnormal data is recorded into the monitoring log, and the abnormal code corresponding to the abnormal synchronous condition is determined.
Optionally, the synchronization abnormal information is counted and presented in a report form, so that a user can know the abnormal condition of the execution of the data synchronization task conveniently. For example, the report content may include specific data numbers corresponding to data read timeout, data write failure, structure conversion error, field information missing, data repeated write, and the like, and may correspondingly show an exception code corresponding to an exception condition, which is not limited in this embodiment.
The synchronization exception condition may be data conversion failure, data read or write timeout, data read and write error, and the like, which is not limited in this embodiment.
After the second data is written into the target to-be-written database, the first exception recovery method corresponding to the synchronous exception condition is determined according to the synchronous exception information, which may be an exception recovery method corresponding to an exception code that is determined in advance according to the exception code, for example, an exception recovery method for performing re-reading and writing on an exception condition when data reading or writing is overtime, and the like, which is not limited in this embodiment. The exception repair may also include field correction, field filling, and the like.
And repairing the synchronous abnormal condition according to the first abnormal repairing mode, and automatically processing the synchronous abnormal condition without human intervention after the execution process of the data synchronization task, thereby improving the correctness and the integrity of the data synchronization.
In this embodiment, optionally, after writing the second data into the target database to be written, the method further includes:
determining the total write data amount of the target database to be written and the total read data amount of the target database to be read;
judging whether the difference quantity between the total write data quantity and the total read data quantity is within a preset quantity range or not;
if not, determining a second abnormal repairing mode corresponding to the data total quantity difference, and repairing the data total quantity difference according to the second abnormal repairing mode.
And determining the total data amount of the target database to be written before and after the data is written through the database query statement, so as to obtain the total written data amount. Obtaining the total amount of read data of the target database to be read by determining the data to be read in the target database to be read, illustratively, if the data to be read is all data in the target database to be read, the total amount of read data is the total amount of data in the target database to be read; and if the data to be read is partial data in the target database to be read, the total amount of the read data is the total amount of the partial data.
And judging whether the difference quantity of the total quantity of the written data and the total quantity of the read data is within a preset quantity range, if so, determining that the data are synchronous and consistent, otherwise, determining a second abnormal repairing mode corresponding to the difference of the total quantity of the data, and repairing the difference of the total quantity of the data according to the second abnormal repairing mode because different databases possibly have errors when the data are mutually synchronous. For example, resynchronization or alarm prompting, etc., which is not limited in this embodiment. Therefore, excessive data omission in the execution of the data synchronization task is avoided, and the consistency and the accuracy of data synchronization are improved.
According to the technical scheme provided by the embodiment, the target database to be read and the target database to be written which correspond to the target data synchronization task are determined according to the preconfigured target data synchronization task information; calling a reading interface of a target database to be read according to the target data synchronization task information, reading first data of the target database to be read, and converting a data structure corresponding to the first data into a data structure corresponding to data in the target database to be written to obtain second data; and calling a write-in interface of the target database to be written according to the target data synchronization task information, and writing the second data into the target database to be written. Data synchronization among different databases can be completed only by knowing basic database information, data reading and writing programs do not need to be independently written for each data synchronization task, a unified scheme of data synchronization among different types of databases is realized by a set of data reading and writing method, and the efficiency and the applicability of data synchronization are improved.
Example two
Fig. 2 is a flowchart of a database data synchronization method according to a second embodiment of the present invention, and the technical solution is supplementary explained for a process before determining a target to-be-read database and a target to-be-written database corresponding to a target data synchronization task according to preconfigured target data synchronization task information. Compared with the scheme, the scheme is specifically optimized in that before determining the target database to be read and the target database to be written corresponding to the target data synchronization task according to the preconfigured target data synchronization task information, the method further comprises the following steps:
responding to information management operation and parameter definition operation to generate a task work order corresponding to the target data synchronization task; wherein the information management operation comprises a database information management operation and a table structure information management operation;
and sending the task work order to at least one target approval party to obtain an approval result corresponding to each target approval party, and determining whether to execute the target data synchronization task according to the approval result. Specifically, the flow chart of the database data synchronization method is shown in fig. 2:
s210, responding to information management operation and parameter definition operation to generate a task work order corresponding to the target data synchronization task; wherein the information management operation includes a database information management operation and a table structure information management operation.
Wherein the information management operation and the parameter definition operation can be issued by a maker of the target data synchronization task.
The information management operation may include a database information management operation and a table structure information management operation, by which information required for the target data synchronization task is managed in advance. The database information management is to manage and control a database supported by the multi-source data synchronization engine, for example, bringing the database corresponding to the current target data synchronization task into a management and control range.
The management of the table structure information manages and controls a special table structure in the database, the special table structure is a table structure which is lower in use frequency compared with a conventional table structure, the data conversion mode of the special table structure is prevented from being determined temporarily during data synchronization by adding the special table structure information, if the table structure information is configured, a default data reading strategy is adjusted to adapt to the special condition of the target database to be read, and data conversion and adjustment can be performed according to the database type of the target database to be written and the table structure information configuration during data writing, so that the data synchronization efficiency is improved.
The parameter definition operation is used for defining technical parameters required by the target data synchronization task, such as maximum node number, channel capacity, maximum delay and the like, wherein the maximum node number can be defined according to equipment performance, the parallel number of the data synchronization task is determined, and the task execution efficiency is determined; the channel capacity determines the maximum amount of temporarily stored data in the cache in the data synchronization process, and can be determined according to the size of the memory; the maximum delay determines the maximum waiting time of single data synchronization, and the exceeding is determined as the abnormal synchronization condition.
Optionally, the information management operation and the parameter definition operation can be conveniently performed by a user through the visualization platform, and the convenience degree of operation execution is improved.
And generating a task work order corresponding to the target data synchronization task through information management operation and parameter definition operation, wherein the task work order may include task details of the target data synchronization task, such as database types of a target database to be read and a target database to be written, task execution time, system parameters, and the like.
S220, the task work order is sent to at least one target approving party to obtain an approval result corresponding to each target approving party, and whether the target data synchronization task is executed or not is determined according to the approval result.
And sending the task work order to at least one target approving party to obtain an approving result corresponding to each target approving party, and determining whether to execute the target data synchronization task according to the approving result. The target approving party can be an approving party in different levels of different departments, so that multi-level approval is realized. And if the work order approval result is that the approval is passed, the target data synchronization task takes effect, and the target data synchronization task is started according to the task starting time defined in the work order.
And S230, if the target data synchronization task is determined to be executed, determining a target database to be read and a target database to be written corresponding to the target data synchronization task according to pre-configured target data synchronization task information.
And when the target data synchronization task is determined to be executed, the target data synchronization task information is the task information related to the approved task work order.
S240, calling a reading interface of the target database to be read according to the target data synchronization task information, reading first data of the target database to be read, and converting a data structure corresponding to the first data into a data structure corresponding to data in the target database to be written to obtain second data.
And S250, calling a writing interface of the target database to be written according to the target data synchronization task information, and writing the second data into the target database to be written.
In this embodiment, optionally, the method further includes:
splitting a data synchronization total task corresponding to each target database to be read into at least two data synchronization subtasks respectively;
determining a target task group corresponding to each data synchronization subtask according to the current working state of at least one candidate task group;
and executing the data synchronization subtask through the target task group.
The data synchronization total task corresponding to the target database to be read can be divided into a plurality of subtasks according to the table structure, the field information, the data volume and the like, wherein each target database to be read corresponds to the respective data synchronization total task. And determining a candidate task group corresponding to each data synchronization overall task according to the preset configuration, such as a candidate task group ABC corresponding to the data synchronization overall task, a candidate task group ABCD corresponding to the data synchronization overall task, and the like.
And determining whether the candidate task group is in an idle state according to the current working state of the candidate task group, and if so, determining any task group in the idle state as a target task group. And when the task group starts the subtasks, threads for reading data, managing channels and writing data are started fixedly to complete task synchronization. Wherein the channel management is used to determine a current working state of the candidate task group.
Fig. 3 is a schematic diagram of splitting and allocating a data synchronization overall task according to a second embodiment of the present invention. As shown in fig. 3, a data synchronization task is divided into data synchronization subtasks 1-11 (abbreviated as subtasks in the figure). The candidate task groups comprise ABCD, and each candidate task group comprises data reading, channel management and data writing threads. And scheduling the subtasks according to the current working state of each candidate task group, wherein the subtask 1 is scheduled to the task group B, the subtask 2 is scheduled to the task group A, the subtask 3 is scheduled to the task group D, the subtask 4 is scheduled to the task group B, the subtask 5 is scheduled to the task group C, the subtask 6 is scheduled to the task group D, the subtask 7 is scheduled to the task group C, the subtask 8 is scheduled to the task group D, the subtask 9 is scheduled to the task group B, the subtask 10 is scheduled to the task group D, and the subtask 11 is scheduled to the task group C, namely, the target task groups corresponding to the subtasks 1, 4 and 9 are the task group B, the target task group corresponding to the subtask 2 is the task group A, the target task groups corresponding to the subtasks 5, 7 and 11 are the task group C, and the target task groups corresponding to the subtasks 3, 6, 8 and 10 are the task group D.
Therefore, each data synchronization subtask is distributed to a target task group which is most suitable for executing the task at present, each target task group endows reading, writing and data channel capabilities to the corresponding data synchronization subtask, system resources are effectively regulated and controlled, parallel and efficient execution of the subtasks is achieved, and task execution efficiency is improved.
In this embodiment, optionally, the method further includes:
responding to a database interface generation request, and determining the database type of a database corresponding to a target generation interface;
determining whether a generation condition for generating the target generation interface is met or not according to the database type and a preset interface generation template;
if not, generating the target generation interface according to the preset interface generation template and preset table structure information;
and registering the target generation interface to an interface scheduling service bus so as to call the target generation interface in the database data synchronization process.
The database interface generation request may be sent by a user, or may be sent automatically when the database interface of the target to-be-read database or the target to-be-written database is not generated, which is not limited in this embodiment. And responding to a database interface generation request, determining a database type of a database corresponding to the target generation interface, determining whether a generation condition for generating the target generation interface is met or not according to the database type and a preset interface generation template, namely judging whether a generation method for generating the target generation interface associated with the database type exists in the preset interface generation template or not. And if so, directly generating a target generation interface according to a preset interface generation template. The target generation interface is an interface needing to be generated.
If not, generating a target generation interface according to a preset interface generation template and preset table structure information, wherein the preset table structure information is information of a pre-configured special table structure, generating basic information based on the interface generation template, and adding the special table information according to the preset table structure information so as to generate the target generation interface.
And registering the generated target generation interface to an interface scheduling service bus in the multi-source data synchronization engine, so that the corresponding work, such as write work or read work, can be executed by calling the interface through the interface scheduling service bus in the data synchronization process. A standard import and export function is researched and developed and embedded, and a conventional interface template is formed by packaging based on the function, so that a data read-write interface can be added through a template configuration mode and/or preset table structure information, and the interface generation efficiency is improved. The newly generated interface is imported into the multi-source data synchronization engine according to the communication protocol, and can be called after registration, so that the convenience of interface use is improved.
The multi-source data synchronization engine integrates read-write interfaces of different databases in a centralized manner, analyzes parameters of a data synchronization task work order and calls a corresponding interface to execute, provides an interface rapid manufacturing module, rapidly generates a database read-write interface through conventional template configuration, and can also define special table structure information in the database through tools.
The multi-source data synchronization engine is a central control node of the whole target data synchronization task and plays roles of data cleaning, subtask segmentation, task group management and the like. After the central hub management node is started for the target data synchronization task, a process is started to complete the whole data synchronization task, and then the node completing the task is the central hub management node.
Fig. 4 is a schematic diagram of a multi-source data synchronization engine according to a second embodiment of the present invention. As shown in fig. 4, the multi-source data synchronization engine includes an interface integration scheduling module, a task management and control module, and an interface manufacturing module, and integrates read-write interfaces of different databases, analyzes parameters of the data synchronization task order, calls corresponding interfaces to execute, and provides an interface manufacturing function.
The interface integration scheduling module comprises three components of a communication protocol, a parameter analysis component and an interface scheduling service bus, wherein the communication protocol is used for carrying out information transmission with the outside, and supporting working scenes such as new interface import, task work order receiving, data same-step-length connection establishment and the like; the parameter analysis is used for analyzing the configuration parameters of the task work order and informing an interface scheduling service bus to call an interface to execute corresponding work; the data read and write interfaces must register with the interface dispatch service bus to be effective.
The task management and control module comprises a subtask splitting component, a subtask scheduling component and a task group management component. The subtask splitting is used for splitting the data synchronization total task into subtasks, the subtask scheduling is used for scheduling the subtasks to a target task group, and the task group management is used for managing specific information of the task group.
The interface manufacturing module is used for packaging data to form a conventional interface template based on research and development and embedded with a standardized heterogeneous data import and export function, newly-added data read-write interfaces can be edited through a template configuration mode and/or table information, logs can be recorded, and the new interfaces can be imported into the multisource data synchronization engine according to a communication protocol and can be called after registration.
According to the embodiment of the invention, the task work order corresponding to the target data synchronization task is generated in response to the information management operation and the parameter definition operation before the target data synchronization task is executed, the data synchronization method is abstracted and presented in the form of the task, and a data reading and writing program does not need to be independently written according to each data synchronization requirement. And the convenience of data synchronization is improved. And the task work order is sent to at least one target approval party to obtain the approval results corresponding to the target approval parties, whether the target data synchronization task is executed or not is determined according to the approval results, a work order approval workflow is established, the complete life cycle management and control are carried out on the task work order from generation to taking effect, a multi-level cooperative working mode is supported, and the work order approval efficiency and effectiveness are improved.
EXAMPLE III
Fig. 5 is a schematic structural diagram of a database data synchronization apparatus according to a third embodiment of the present invention. The device can be realized in a hardware and/or software mode, can execute the database data synchronization method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. As shown in fig. 5, the apparatus includes:
a database determining module 510, configured to determine, according to preconfigured target data synchronization task information, a target to-be-read database and a target to-be-written database that correspond to the target data synchronization task;
a data reading module 520, configured to call a reading interface of the target database to be read according to the target data synchronization task information, read first data of the target database to be read, and convert a data structure corresponding to the first data into a data structure corresponding to data in the target database to be written, so as to obtain second data;
and a data writing module 530, configured to call a writing interface of the target to-be-written database according to the target data synchronization task information, and write the second data into the target to-be-written database.
On the basis of the above technical solutions, optionally, the data reading module includes:
the data reading unit is used for calling a reading interface of the target database to be read according to the target data synchronization task information and reading the first data to a preset cache position;
correspondingly, the data writing module comprises:
and the data writing unit is used for calling a writing interface of the target database to be written according to the target data synchronization task information and writing the second data in the preset cache position into the target database to be written.
On the basis of the above technical solutions, optionally, the apparatus further includes:
the abnormal condition determining module is used for determining whether a synchronous abnormal condition exists; if yes, generating synchronous abnormal information to record the synchronous abnormal condition;
a first abnormal recovery mode determining module, configured to determine, according to the synchronous abnormal information, a first abnormal recovery mode corresponding to the synchronous abnormal condition after writing the second data into the target database to be written;
and the first abnormity repairing module is used for repairing the synchronous abnormal condition according to the first abnormity repairing mode.
On the basis of the above technical solutions, optionally, the apparatus further includes:
the task work order generation module is used for responding to information management operation and parameter definition operation to generate a task work order corresponding to the target data synchronization task before the database determination module; wherein the information management operation comprises a database information management operation and a table structure information management operation;
and the approval result acquisition module is used for sending the task work order to at least one target approving party so as to obtain an approval result corresponding to each target approving party and determine whether to execute the target data synchronization task according to the approval result.
On the basis of the above technical solutions, optionally, the apparatus further includes:
a data total amount determining module, configured to determine, after the data writing module, a total amount of data written into the target database to be written and a total amount of data read from the target database to be read;
a difference quantity judging module, configured to judge whether a difference quantity between the total write data quantity and the total read data quantity is within a preset quantity range;
and the second abnormity repairing module is used for determining a second abnormity repairing mode corresponding to the total data quantity difference if the difference quantity judging module judges that the difference quantity is not the same, and repairing the total data quantity difference according to the second abnormity repairing mode.
On the basis of the above technical solutions, optionally, the apparatus further includes:
the database type determining module is used for responding to the database interface generation request and determining the database type of the database corresponding to the target generation interface;
the interface generation condition satisfaction determining module is used for determining whether the generation condition for generating the target generation interface is satisfied or not according to the database type and a preset interface generation template;
the interface generation module is used for generating the target generation interface according to the preset interface generation template and the preset table structure information if the interface generation condition meets the judgment that the determination module determines that the interface generation condition is negative;
and the interface registration module is used for registering the target generation interface to an interface scheduling service bus so as to call the target generation interface in the database data synchronization process.
On the basis of the above technical solutions, optionally, the apparatus further includes:
the subtask splitting module is used for splitting the data synchronization total task corresponding to each target database to be read into at least two data synchronization subtasks respectively;
the target task group determining module is used for determining a target task group corresponding to each data synchronization subtask according to the current working state of at least one candidate task group;
and the subtask execution module is used for executing the data synchronization subtask through the target task group.
Example four
FIG. 6 illustrates a schematic diagram of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 6, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 may also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
In some embodiments, the database data synchronization method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the database data synchronization method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the database data synchronization method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above, reordering, adding or deleting steps, may be used. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A database data synchronization method, comprising:
determining a target database to be read and a target database to be written corresponding to the target data synchronization task according to preconfigured target data synchronization task information;
calling a reading interface of the target database to be read according to the target data synchronization task information, reading first data of the target database to be read, and converting a data structure corresponding to the first data into a data structure corresponding to data in the target database to be written to obtain second data;
and calling a writing interface of the target database to be written according to the target data synchronization task information, and writing the second data into the target database to be written.
2. The method according to claim 1, wherein calling a reading interface of the target database to be read according to the target data synchronization task information to read the first data of the target database to be read comprises:
calling a reading interface of the target database to be read according to the target data synchronization task information, and reading the first data to a preset cache position;
correspondingly, calling a write interface of the target database to be written according to the target data synchronization task information, and writing the second data into the target database to be written, including:
and calling a write-in interface of the target database to be written according to the target data synchronization task information, and writing the second data in the preset cache position into the target database to be written.
3. The method of claim 1, further comprising:
determining whether a synchronization exception condition exists; if yes, generating synchronous abnormal information to record the synchronous abnormal condition;
after the second data is written into the target database to be written, determining a first abnormal repairing mode corresponding to the synchronous abnormal condition according to the synchronous abnormal information;
and repairing the synchronous abnormal condition according to the first abnormal repairing mode.
4. The method according to claim 1, before determining the target to-be-read database and the target to-be-written database corresponding to the target data synchronization task according to the preconfigured target data synchronization task information, further comprising:
responding to information management operation and parameter definition operation to generate a task work order corresponding to the target data synchronization task; wherein the information management operation comprises a database information management operation and a table structure information management operation;
and sending the task work order to at least one target approval party to obtain an approval result corresponding to each target approval party, and determining whether to execute the target data synchronization task according to the approval result.
5. The method of claim 1, wherein after writing the second data to the target database-to-be-written, the method further comprises:
determining the total amount of write data of the target database to be written and the total amount of read data of the target database to be read;
judging whether the difference quantity of the total write data quantity and the total read data quantity is within a preset quantity range or not;
if not, determining a second abnormal restoration mode corresponding to the data total amount difference, and restoring the data total amount difference according to the second abnormal restoration mode.
6. The method of claim 1, further comprising:
responding to a database interface generation request, and determining the database type of a database corresponding to a target generation interface;
determining a template according to the database type and a preset interface generation template whether a generation condition for generating the target generation interface is met;
if not, generating the target generation interface according to the preset interface generation template and preset table structure information;
and registering the target generation interface to an interface scheduling service bus so as to call the target generation interface in the database data synchronization process.
7. The method of claim 1, further comprising:
splitting a data synchronization total task corresponding to each target database to be read into at least two data synchronization subtasks respectively;
determining a target task group corresponding to each data synchronization subtask according to the current working state of at least one candidate task group;
and executing the data synchronization subtask through the target task group.
8. A database data synchronization apparatus, comprising:
the database determining module is used for determining a target database to be read and a target database to be written corresponding to the target data synchronization task according to the preconfigured target data synchronization task information;
the data reading module is used for calling a reading interface of the target database to be read according to the target data synchronization task information, reading first data of the target database to be read, and converting a data structure corresponding to the first data into a data structure corresponding to data in the target database to be written to obtain second data;
and the data writing module is used for calling a writing interface of the target database to be written according to the target data synchronization task information and writing the second data into the target database to be written.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the database data synchronization method of any one of claims 1-7.
10. A computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, the computer instructions are for causing a processor to, when executed, implement the database data synchronization method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210700258.7A CN115146000A (en) | 2022-06-20 | 2022-06-20 | Database data synchronization method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210700258.7A CN115146000A (en) | 2022-06-20 | 2022-06-20 | Database data synchronization method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115146000A true CN115146000A (en) | 2022-10-04 |
Family
ID=83409193
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210700258.7A Pending CN115146000A (en) | 2022-06-20 | 2022-06-20 | Database data synchronization method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115146000A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116204587A (en) * | 2023-02-21 | 2023-06-02 | 中国人民解放军海军工程大学 | Data synchronization task generation method, device and computer readable storage medium |
| CN117724664A (en) * | 2024-02-18 | 2024-03-19 | 芯来智融半导体科技(上海)有限公司 | Data writing method, device, equipment and storage medium |
-
2022
- 2022-06-20 CN CN202210700258.7A patent/CN115146000A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116204587A (en) * | 2023-02-21 | 2023-06-02 | 中国人民解放军海军工程大学 | Data synchronization task generation method, device and computer readable storage medium |
| CN116204587B (en) * | 2023-02-21 | 2024-01-30 | 中国人民解放军海军工程大学 | Data synchronization task generation method, device and computer readable storage medium |
| CN117724664A (en) * | 2024-02-18 | 2024-03-19 | 芯来智融半导体科技(上海)有限公司 | Data writing method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115146000A (en) | Database data synchronization method and device, electronic equipment and storage medium | |
| US20230144100A1 (en) | Method and apparatus for managing and controlling resource, device and storage medium | |
| CN112347071A (en) | Power distribution network cloud platform data fusion method and power distribution network cloud platform | |
| CN112685499A (en) | Method, device and equipment for synchronizing process data of work service flow | |
| CN110837496A (en) | Data quality management method and system based on dynamic sql | |
| CN115599769A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN115408546A (en) | Time sequence data management method, device, equipment and storage medium | |
| CN115291806A (en) | Processing method, processing device, electronic equipment and storage medium | |
| CN113094406A (en) | Power marketing data management method and system | |
| CN112925811B (en) | Method, apparatus, device, storage medium and program product for data processing | |
| CN116383207A (en) | Data tag management method and device, electronic equipment and storage medium | |
| CN115730022A (en) | Data processing construction method and platform system adopting event triggering and process arrangement | |
| CN115905322A (en) | Service processing method and device, electronic equipment and storage medium | |
| CN115438056A (en) | Data acquisition method, device, equipment and storage medium | |
| CN116821217A (en) | Data distribution conversion method, device, equipment and storage medium | |
| CN113225228B (en) | Data processing method and device | |
| CN116431698B (en) | Data extraction method, device, equipment and storage medium | |
| CN114706578A (en) | Data processing method, device, equipment and medium | |
| CN117633116A (en) | Data synchronization method, device, electronic equipment and storage medium | |
| CN117668114A (en) | Data synchronization method, device, equipment and medium | |
| CN114638935A (en) | Method and device for generating dimension monitoring task and monitoring data quality | |
| CN117807049A (en) | Data lake-based data warehouse optimization method and system | |
| CN115665256A (en) | Task processing method and device, electronic equipment and storage medium | |
| CN115455692A (en) | Method, device, equipment and medium for pushing radar digital twin data | |
| CN117271104A (en) | Resource arrangement method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |