CN118093719A - Data synchronization method, device, electronic equipment and computer readable medium - Google Patents
Data synchronization method, device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN118093719A CN118093719A CN202410274597.2A CN202410274597A CN118093719A CN 118093719 A CN118093719 A CN 118093719A CN 202410274597 A CN202410274597 A CN 202410274597A CN 118093719 A CN118093719 A CN 118093719A
- Authority
- CN
- China
- Prior art keywords
- job
- execution
- target
- statement
- target job
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000012800 visualization Methods 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 238000013480 data collection Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000003442 weekly effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013079 data visualisation Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization method, a data synchronization device, electronic equipment and a computer readable medium, and relates to the technical field of data synchronization. The method comprises the following steps: scanning a job table, and inquiring each target job with a job state to be executed and the execution date of the next execution being the current date from the job table; for each project label job, locating a statement file corresponding to the target job according to a storage path of the statement file corresponding to the target job, and reading a first database statement from the statement file; for each project standard job, executing a first database statement corresponding to the target job, writing an execution result into a result table corresponding to the target job, modifying the job state of the target job to be completed, and generating a new job according to the target job and adding the new job into the job table. The embodiment can solve the technical problems of complex synchronization flow or complex configuration flow.
Description
Technical Field
The present invention relates to the field of data synchronization technologies, and in particular, to a data synchronization method, a data synchronization device, an electronic device, and a computer readable medium.
Background
The MySQL database can be used for storing data and carrying out data query, has good processing capacity for structured data and complex query, and supports various advanced query and aggregation functions. Hive would be more suitable for large-scale data sets, as well as compatible semi-structured data, unstructured data, and structured data. The timeliness of big data pursuit, namely, the data which is generally frequently queried and processed does not exceed a long time, so that if the historical data is stored in MySQL, the query processing of the latest data can be influenced.
MySQL is convenient for data query but is not suitable for storing large data volume, but MySQL has wider applicability, and in many business scenarios, data can be stored in MySQL, so there is a need to migrate data into Hive for excessive data and infrequently used historical data. Currently, a common data synchronization method includes performing a synchronization operation through a command line at a console and performing the synchronization operation using an Sqoop tool.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
The method of executing the synchronization operation through the command line is very complicated, and each file needs to be input with a script to execute the synchronization operation, and if a plurality of data tables exist, batch processing is very unfavorable. For the method using the Sqoop tool, the method is suitable for data tables with similar data structures, the data with different requirements cannot be synchronized more flexibly, the Sqoop is required to be configured to be compatible with MySQL and Hive, field-level mapping is required, and the configuration flow is complicated.
Disclosure of Invention
In view of the above, the embodiments of the present invention provide a data synchronization method, apparatus, electronic device, and computer readable medium, so as to solve the technical problem that the synchronization process is complicated or the configuration process is complicated.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a data synchronization method including:
Scanning a job table, and inquiring each target job with a job state to be executed and the execution date of the next execution being the current date from the job table; the basic attributes of each job are configured in the job table, the basic attributes comprise execution frequency, execution period value, job state, a storage path of a statement file, a table name of a result table and execution date of next execution, the statement file stores a first database statement, the job table is stored in a first database, and the result table is stored in a second database;
For each project label job, locating a statement file corresponding to the target job according to a storage path of the statement file corresponding to the target job, and reading a first database statement from the statement file;
For each project standard job, executing a first database statement corresponding to the target job, writing an execution result into a result table corresponding to the target job, modifying the job state of the target job to be completed, and generating a new job according to the target job and adding the new job into the job table.
Optionally, executing the first database statement corresponding to the target job, writing an execution result into a result table corresponding to the target job, and modifying the job status of the target job to be completed, including:
modifying the job state of the target job into execution;
executing a first database statement corresponding to the target job;
judging whether the first database statement is successfully executed or not;
If yes, writing the execution result into a result table corresponding to the target job and modifying the job state of the target job to be completed;
if not, modifying the operation state of the target operation to be failed.
Optionally, generating a new job according to the target job and adding the new job to the job table, including:
And generating a new job according to the execution frequency, the execution period value and the current date of the target job, and adding the new job into the job table.
Optionally, generating a new job according to the execution frequency, the execution cycle value and the current date of the target job includes:
determining the execution date of the next execution according to the execution frequency, the execution period value and the current date of the target job;
Generating a new job, and taking the execution date of the next execution as the execution date of the next execution of the new job; the execution frequency, the execution period value, the storage path of the statement file and the table name of the result table of the new job are the same as those of the target job, and the job state of the new job is to be executed.
Optionally, before scanning the job table, the method further comprises:
and configuring the basic attribute of each job into a job table according to the job requirement of each job.
Optionally, the method further comprises:
transmitting the execution result of the target job written in the result table to a visualization platform, so as to update the execution result of the target job in a visualization interface.
In addition, according to another aspect of the embodiment of the present invention, there is provided a data synchronization apparatus including:
The query module is used for scanning a job table, and querying each target job with a job state to be executed and a next execution date as a current date from the job table; the basic attributes of each job are configured in the job table, the basic attributes comprise execution frequency, execution period value, job state, a storage path of a statement file, a table name of a result table and execution date of next execution, the statement file stores a first database statement, the job table is stored in a first database, and the result table is stored in a second database;
The reading module is used for positioning the statement file corresponding to the target job according to the storage path of the statement file corresponding to the target job for each project label job, and reading a first database statement from the statement file;
And the execution module is used for executing a first database statement corresponding to the target job for each project standard job, writing an execution result into a result table corresponding to the target job, modifying the job state of the target job to be completed, and generating a new job according to the target job and adding the new job into the job table.
Optionally, the execution module is further configured to:
modifying the job state of the target job into execution;
executing a first database statement corresponding to the target job;
judging whether the first database statement is successfully executed or not;
If yes, writing the execution result into a result table corresponding to the target job and modifying the job state of the target job to be completed;
if not, modifying the operation state of the target operation to be failed.
Optionally, the execution module is further configured to:
And generating a new job according to the execution frequency, the execution period value and the current date of the target job, and adding the new job into the job table.
Optionally, the execution module is further configured to:
determining the execution date of the next execution according to the execution frequency, the execution period value and the current date of the target job;
Generating a new job, and taking the execution date of the next execution as the execution date of the next execution of the new job; the execution frequency, the execution period value, the storage path of the statement file and the table name of the result table of the new job are the same as those of the target job, and the job state of the new job is to be executed.
Optionally, the apparatus further comprises a configuration module for:
and configuring the basic attribute of each job into a job table according to the job requirement of each job.
Optionally, the execution module is further configured to:
transmitting the execution result of the target job written in the result table to a visualization platform, so as to update the execution result of the target job in a visualization interface.
According to another aspect of an embodiment of the present invention, there is also provided an electronic device including:
one or more processors;
storage means for storing one or more programs,
The one or more processors implement the method of any of the embodiments described above when the one or more programs are executed by the one or more processors.
According to another aspect of an embodiment of the present invention, there is also provided a computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method according to any of the embodiments described above.
According to another aspect of embodiments of the present invention, there is also provided a computer program product comprising a computer program which, when executed by a processor, implements the method according to any of the embodiments described above.
One embodiment of the above invention has the following advantages or benefits: because the scanning operation table is adopted, each target operation with the operation state to be executed and the execution date of the next execution being the current date is inquired from the operation table, the sentence file corresponding to the target operation is positioned according to the storage path of the sentence file corresponding to the target operation, the first database sentence is read from the sentence file, the first database sentence corresponding to the target operation is executed, the execution result is written into the result table corresponding to the target operation, and a new operation is generated according to the target operation and added into the operation table, the technical problems of complex synchronous flow or complex configuration flow in the prior art are overcome. According to the embodiment of the invention, through a script development method, each job is configured in the job table according to different requirements, when the job meets the execution conditions, the first database statement of the job is executed, and the execution result is written into the second database, so that the complicated command execution flow is reduced, and the initialized configuration process is simplified.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:
FIG. 1 is a flow chart of a data synchronization method according to an embodiment of the invention;
FIG. 2 is a flow chart of a data synchronization method according to one referenceable embodiment of the invention;
FIG. 3 is a flow chart of a data synchronization method according to another referenceable embodiment of the invention;
FIG. 4 is a flow chart of a data synchronization method according to yet another alternative embodiment of the invention;
FIG. 5 is a schematic diagram of a data synchronization device according to an embodiment of the present invention;
FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
Fig. 7 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In the technical scheme of the invention, the aspects of acquisition, analysis, use, transmission, storage and the like of the related user personal information all meet the requirements of related laws and regulations, are used for legal and reasonable purposes, are not shared, leaked or sold outside the aspects of legal use and the like, and are subjected to supervision and management of a supervision department. Necessary measures should be taken for the personal information of the user to prevent illegal access to such personal information data, ensure that personnel having access to the personal information data comply with the regulations of the relevant laws and regulations, and ensure the personal information of the user. Once these user personal information data are no longer needed, the risk should be minimized by limiting or even prohibiting the data collection and/or deletion.
User privacy is protected by de-identifying data when applicable, including in certain related applications, such as by removing specific identifiers (e.g., name, gender, date of birth, etc.), controlling the amount or specificity of stored data, controlling how data is stored, and/or other methods.
Fig. 1 is a flowchart of a data synchronization method according to an embodiment of the present invention. As an embodiment of the present invention, as shown in fig. 1, the data synchronization method may include:
Step 101, scanning a job table, and inquiring each target job with a job state to be executed and a next execution date as a current date from the job table; the basic attributes of each job are configured in the job table, and the basic attributes comprise an execution frequency, an execution period value, a job state, a storage path of a statement file, a table name of a result table and an execution date of the next execution, wherein the statement file stores a first database statement, the job table is stored in a first database, and the result table is stored in a second database.
Data collection is a key ring in large data processing flows, and data storage is often scattered among various physical subsystems. User base data is often stored in important data center data bins, and in data processing supporting multiple applications, the latest user base data needs to be continuously acquired from a data center.
Unlike the simple and cumbersome method using a console and the complex configuration method using tools, the data synchronization method provided by the embodiment of the present invention is performed using an automation script developed by python, i.e., steps 101-103 are performed by the automation script.
Before step 101, basic attributes of each job may be preconfigured, where the basic attributes include an execution frequency, an execution period value, a job state, a storage path of a statement file, a table name of a result table, and an execution date of the next execution, where the execution frequency may be daily, weekly, monthly, and the like, the execution period value may be 1, 2, 3, 4, 5, … …,30, and the like, the job state includes to be executed, being completed, or failing, and the statement file stores a first database statement (such as an SQL statement). It should be noted that, only one first database statement is typically stored in one statement file, for example, only one SQL statement is typically stored in one statement file. The embodiment of the invention only needs to configure the basic attribute of the job, does not need to perform field level configuration, and simplifies the configuration flow.
In step 101, a job table is scanned, and each target job whose job status is to be executed and whose execution date of the next execution is the current date is queried from the job table. It should be noted that, the system executes steps 101-103 each day through the automation script, so that each target job whose job status is to be executed and whose execution date of the next execution is the current date can be queried according to the current date (i.e. the date of the current day).
Step 102, for each project label job, locating the statement file corresponding to the target job according to the storage path of the statement file corresponding to the target job, and reading a first database statement from the statement file.
Because each statement file stores SQL statements, the statement file corresponding to the target job can be located according to the storage path of the statement file in the basic attribute of the target job, and then the first database statement is read from the statement file, for example, the SQL statement corresponding to the target job is read from the statement file.
Step 103, for each project standard job, executing a first database statement corresponding to the target job, writing an execution result into a result table corresponding to the target job, modifying the job state of the target job to be completed, and generating a new job according to the target job and adding the new job into the job table.
After a first database statement (such as an SQL statement) corresponding to a target job is read, executing the first database statement corresponding to the target job, writing an execution result of the first database statement into a result table configured in a basic attribute of the target job, modifying a job state in the basic attribute of the target job to be completed, generating a new job according to the basic attribute of the target job, and adding the newly generated job into the job table.
Optionally, executing the first database statement corresponding to the target job, writing an execution result into a result table corresponding to the target job, and modifying the job status of the target job to be completed, including: modifying the job state of the target job into execution; executing a first database statement corresponding to the target job; judging whether the first database statement is successfully executed or not; if yes, writing the execution result into a result table corresponding to the target job and modifying the job state of the target job to be completed; if not, modifying the operation state of the target operation to be failed. Before executing a target job, firstly modifying the job state of the target job from to-be-executed to in-process, executing a first database statement corresponding to the target job after the state modification is successful, then judging whether the first database statement corresponding to the target job is successfully executed, if so, writing the execution result of the first database statement corresponding to the target job into a result table configured in the basic attribute of the target job, and modifying the job state of the target job from in-process to completed; if the execution fails, the job status of the target job is modified from being executed to failed.
Optionally, generating a new job according to the target job and adding the new job to the job table, including: and generating a new job according to the execution frequency, the execution period value and the current date of the target job, and adding the new job into the job table. If the first database statement corresponding to the target job is successfully executed, a new job is generated according to the execution frequency, the execution period value and the current date of the target job, and the job is added into a job table, so that when the execution date of the next execution of the job is reached, the job can be queried from the job table and executed, and the data synchronization is ensured. Therefore, the embodiment of the invention not only can synchronize data in time, but also can realize daily and periodical data synchronization requirements.
Optionally, generating a new job according to the execution frequency, the execution cycle value and the current date of the target job includes: determining the execution date of the next execution according to the execution frequency, the execution period value and the current date of the target job; generating a new job, and taking the execution date of the next execution as the execution date of the next execution of the new job; the execution frequency, the execution period value, the storage path of the statement file and the table name of the result table of the new job are the same as those of the target job, and the job state of the new job is to be executed. The new job is generated according to the target job which has been successfully executed, so that the execution frequency, the execution period value, the storage path of the sentence file, the table name of the result table are the same as the execution frequency, the execution period value, the storage path of the sentence file and the table name of the result table of the target job, but the execution date of the next execution of the new job is different from the execution date of the next execution of the target job, the execution date of the next execution of the target job is the current date, the execution date of the next execution of the new job is generated according to the execution frequency, the execution period value and the current date of the target job, and the job state of the new job is to be executed.
Therefore, the embodiment of the invention can carry out timing synchronization configuration on the data query operation which is generally and repeatedly executed by programming the automatic running program. The first database can be a MySQL database, and the second database can be a Hive database.
MySQL is a relational database management system for storing a large amount of data, and the relational database is a database based on a relational model, and is characterized by high performance, reliability, expandability and the like by processing the data in the database by means of mathematical concepts and methods such as set algebra and the like.
Hive is a data warehouse tool based on Hadoop, and is used for extracting, converting and loading data, and is a mechanism capable of storing, querying and analyzing large-scale data stored in Hadoop. The Hive data warehouse tool can map a structured data file into a database table, provide an SQL query function, convert SQL sentences into MapReduce tasks to be executed, and is very suitable for carrying out statistical analysis on a data warehouse. In the Hadoop ecosystem, HDFS is used to store data, yarn is used for resource management, mapReduce is used for data processing, and Hive is a data warehouse tool built on top of Hadoop.
According to the embodiment of the invention, the job table is stored in the MySQL database, and the structure table is stored in the Hive (for example, in the Hive partition auto_run), so that the advantages of large data storage of the Hive and high MySQL execution speed are utilized.
According to the various embodiments described above, it can be seen that in the embodiments of the present invention, by scanning the job table, each target job whose job status is to be executed and whose execution date of the next execution is the current date is queried from the job table, the statement file corresponding to the target job is located according to the storage path of the statement file corresponding to the target job, the first database statement is read from the statement file, the first database statement corresponding to the target job is executed, the execution result is written into the result table corresponding to the target job, and a new job is generated according to the target job and added into the job table, so that the technical problem of complicated synchronization flow or complicated configuration flow in the prior art is solved. According to the embodiment of the invention, through a script development method, each job is configured in the job table according to different requirements, when the job meets the execution conditions, the first database statement of the job is executed, and the execution result is written into the second database, so that the complicated command execution flow is reduced, and the initialized configuration process is simplified.
Fig. 2 is a flow chart of a data synchronization method according to one referenceable embodiment of the invention. As yet another embodiment of the present invention, as shown in fig. 2, the data synchronization method may include:
step 201, according to the job requirement of each job, the basic attribute of each job is configured into a job table.
The basic attributes comprise execution frequency, execution period values, operation states, a storage path of a statement file, table names of a result table and execution dates of next execution, wherein the statement file stores a first database statement, the operation table is stored in a first database, and the result table is stored in a second database. The execution frequency may be daily, weekly, monthly, etc., the execution cycle value may be 1,2,3, 4, 5, … …,30, etc., the job status includes to be executed, in execution, completed or failed, and the statement file stores a first database statement (such as SQL statement). Alternatively, the first database may be a MySQL database and the second database may be a Hive database.
Step 202, a job table is scanned, and each target job with a job state to be executed and a next execution date as a current date is queried from the job table.
Because each job in the job table is configured with basic attributes, such as execution frequency, execution period value, job state, storage path of statement file, table name of result table and execution date of next execution, each target job whose job state is to be executed and execution date of next execution is current date can be queried by scanning the job table. It should be noted that, the system executes steps 402-404 each day through the automation script, so that each target job whose job status is to be executed and whose execution date of the next execution is the current date can be queried according to the current date (i.e. the date of the current day).
Step 203, for each project label job, locating the statement file corresponding to the target job according to the storage path of the statement file corresponding to the target job, and reading the first database statement from the statement file.
After the target job meeting the execution condition is queried from the job table, step 203 and step 204 are executed for each project target job. The target jobs may be executed together or sequentially. For example, the number of synchronously executed jobs may be set in advance, such as to be set to 3, 4, 5, or 6, etc., and the target job is synchronously executed according to the preset number of synchronously executed jobs, which may improve the data synchronization efficiency.
Step 204, for each project standard job, executing the first database statement corresponding to the target job, writing the execution result into the result table corresponding to the target job, modifying the job state of the target job to be completed, and generating a new job according to the target job and adding the new job into the job table.
According to the embodiment of the invention, the MySQL database is used for storing and managing the operation table of the automatic running batch, the Hive SQL needing operation is stored in a unified place, and the automatic running batch method is used for updating the SQL command needing complicated operation in batches, and the operation state of the target operation is updated after the SQL command is executed.
In addition, in one embodiment of the present invention, the implementation of the data synchronization method is described in detail in the above data synchronization method, and thus the description thereof will not be repeated here.
Fig. 3 is a flow chart of a data synchronization method according to another referenceable embodiment of the invention. As another embodiment of the present invention, as shown in fig. 3, the data synchronization method may include:
Step 301, a job table is scanned, and each target job whose job status is to be executed and whose execution date of the next execution is the current date is queried from the job table.
The basic attributes of each job are configured in the job table, and the basic attributes comprise an execution frequency, an execution period value, a job state, a storage path of a statement file, a table name of a result table and an execution date of the next execution, wherein the statement file stores a first database statement, the job table is stored in a first database, and the result table is stored in a second database.
For each project label job, steps 302-310 are performed.
Step 302, locating the statement file corresponding to the target job according to the storage path of the statement file corresponding to the target job, and reading the first database statement from the statement file.
Step 303, modifying the job status of the target job to be executing.
Step 304, executing the first database statement corresponding to the target job.
Step 305, judging whether the first database statement is executed successfully; if yes, go to step 306; if not, step 310 is performed.
And step 306, writing the execution result into a result table corresponding to the target job and modifying the job state of the target job to be completed.
Step 307, determining the execution date of the next execution according to the execution frequency, the execution period value and the current date of the target job.
Step 308, generating a new job, and taking the execution date of the next execution as the execution date of the next execution of the new job.
The execution frequency, the execution period value, the storage path of the statement file and the table name of the result table of the new job are the same as those of the target job, and the job state of the new job is to be executed.
Step 309, adding the new job to the job table.
Step 310, modifying the job status of the target job to fail.
In addition, in another embodiment of the present invention, the implementation of the data synchronization method is described in detail in the above data synchronization method, so that the description is not repeated here.
Fig. 4 is a flowchart of a data synchronization method according to still another exemplary embodiment of the present invention. As still another embodiment of the present invention, as shown in fig. 4, the data synchronization method may include:
Step 401, a job table is scanned, and each target job with a job state to be executed and a next execution date of the next execution being a current date is queried from the job table; the basic attributes of each job are configured in the job table, and the basic attributes comprise an execution frequency, an execution period value, a job state, a storage path of a statement file, a table name of a result table and an execution date of the next execution, wherein the statement file stores a first database statement, the job table is stored in a first database, and the result table is stored in a second database.
For each project label job, steps 402-409 are performed.
Step 402, locating the statement file corresponding to the target job according to the storage path of the statement file corresponding to the target job, and reading the first database statement from the statement file.
Step 403, modifying the job status of the target job to be executing.
Step 404, executing the first database statement corresponding to the target job.
Step 405, judging whether the first database statement is executed successfully; if yes, go to step 406; if not, go to step 408.
And step 406, writing the execution result into a result table corresponding to the target job and modifying the job state of the target job to be completed.
Step 407, generating a new job according to the execution frequency, the execution period value and the current date of the target job, and adding the new job into the job table. The execution frequency, the execution period value, the storage path of the statement file and the table name of the result table of the new job are the same as those of the target job, and the job state of the new job is to be executed.
Step 408, transmitting the execution result of the target job written in the result table to a visualization platform, so as to update the execution result of the target job in a visualization interface.
Because the first database and the second database are both connected with the visualization platform, after the tables in the databases are updated, the charts of the visualization platform are automatically updated. Wherein the graph of the visualization platform corresponds to each execution result in the structure table.
Step 409, modifying the job status of the target job to fail.
According to the embodiment of the invention, through a script development method, each job is configured in the job table according to different requirements, when the job meets the execution conditions, the first database statement of the job is executed, the execution result is written into the second database, and the execution result is automatically displayed on the visualization platform for data visualization, so that the complicated command execution flow is reduced, and the initialized configuration process is simplified.
In addition, in still another embodiment of the present invention, the implementation of the data synchronization method is described in detail in the above-mentioned data synchronization method, and thus the description thereof will not be repeated here.
Fig. 5 is a schematic diagram of a data synchronization device according to an embodiment of the present invention. As shown in fig. 5, the data synchronization device 500 includes a query module 501, a reading module 502, and an executing module 503; the query module 501 is configured to scan a job table, and query each target job whose job status is to be executed and whose execution date of the next execution is the current date from the job table; the basic attributes of each job are configured in the job table, the basic attributes comprise execution frequency, execution period value, job state, a storage path of a statement file, a table name of a result table and execution date of next execution, the statement file stores a first database statement, the job table is stored in a first database, and the result table is stored in a second database; the reading module 502 is configured to locate, for each target job, a statement file corresponding to the target job according to a storage path of the statement file corresponding to the target job, and read a first database statement from the statement file; the execution module 503 is configured to execute, for each target job, a first database statement corresponding to the target job, write an execution result into a result table corresponding to the target job, modify a job status of the target job to be completed, and generate a new job according to the target job and add the new job to the job table.
Optionally, the execution module 503 is further configured to:
modifying the job state of the target job into execution;
executing a first database statement corresponding to the target job;
judging whether the first database statement is successfully executed or not;
If yes, writing the execution result into a result table corresponding to the target job and modifying the job state of the target job to be completed;
if not, modifying the operation state of the target operation to be failed.
Optionally, the execution module 503 is further configured to:
And generating a new job according to the execution frequency, the execution period value and the current date of the target job, and adding the new job into the job table.
Optionally, the execution module 503 is further configured to:
determining the execution date of the next execution according to the execution frequency, the execution period value and the current date of the target job;
Generating a new job, and taking the execution date of the next execution as the execution date of the next execution of the new job; the execution frequency, the execution period value, the storage path of the statement file and the table name of the result table of the new job are the same as those of the target job, and the job state of the new job is to be executed.
Optionally, the apparatus further comprises a configuration module for:
and configuring the basic attribute of each job into a job table according to the job requirement of each job.
Optionally, the execution module 503 is further configured to:
transmitting the execution result of the target job written in the result table to a visualization platform, so as to update the execution result of the target job in a visualization interface.
The specific implementation of the data synchronization device according to the present invention is described in detail in the above data synchronization method, and thus the description thereof will not be repeated here.
Fig. 6 illustrates an exemplary system architecture 600 in which a data synchronization method or data synchronization apparatus of an embodiment of the present invention may be applied.
As shown in fig. 6, the system architecture 600 may include terminal devices 601, 602, 603, a network 604, and a server 605. The network 604 is used as a medium to provide communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 605 via the network 604 using the terminal devices 601, 602, 603 to receive or send messages, etc. Various communication client applications such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 601, 602, 603.
The terminal devices 601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 605 may be a server providing various services, such as a background management server (by way of example only) providing support for shopping-type websites browsed by users using terminal devices 601, 602, 603. The background management server can analyze and other data such as the received article information inquiry request and feed back the processing result to the terminal equipment.
It should be noted that, the data synchronization method provided by the embodiment of the present invention is generally executed by the server 605, and accordingly, the data synchronization device is generally disposed in the server 605.
It should be understood that the number of terminal devices, networks and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 7, there is illustrated a schematic diagram of a computer system 700 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, ROM 702, and RAM703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output portion 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 708 including a hard disk or the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. The drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is mounted into the storage section 708 as necessary.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 709, and/or installed from the removable medium 711. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 701.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer programs according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor includes a query module, a read module, and an execution module, where the names of the modules do not constitute a limitation on the module itself in some cases.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, implement the method of: scanning a job table, and inquiring each target job with a job state to be executed and the execution date of the next execution being the current date from the job table; the basic attributes of each job are configured in the job table, the basic attributes comprise execution frequency, execution period value, job state, a storage path of a statement file, a table name of a result table and execution date of next execution, the statement file stores a first database statement, the job table is stored in a first database, and the result table is stored in a second database; for each project label job, locating a statement file corresponding to the target job according to a storage path of the statement file corresponding to the target job, and reading a first database statement from the statement file; for each project standard job, executing a first database statement corresponding to the target job, writing an execution result into a result table corresponding to the target job, modifying the job state of the target job to be completed, and generating a new job according to the target job and adding the new job into the job table.
As a further aspect, embodiments of the present invention also provide a computer program product comprising a computer program which, when executed by a processor, implements the method according to any of the above embodiments.
According to the technical scheme of the embodiment of the invention, as the scanning job table is adopted, each target job with the job state to be executed and the execution date of the next execution being the current date is queried from the job table, the statement file corresponding to the target job is positioned according to the storage path of the statement file corresponding to the target job, the first database statement is read from the statement file, the first database statement corresponding to the target job is executed, the execution result is written into the result table corresponding to the target job, and a new job is generated according to the target job and added into the job table, so that the technical problems of complex synchronous flow or complex configuration flow in the prior art are overcome. According to the embodiment of the invention, through a script development method, each job is configured in the job table according to different requirements, when the job meets the execution conditions, the first database statement of the job is executed, and the execution result is written into the second database, so that the complicated command execution flow is reduced, and the initialized configuration process is simplified.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (15)
1. A method of data synchronization, comprising:
Scanning a job table, and inquiring each target job with a job state to be executed and the execution date of the next execution being the current date from the job table; the basic attributes of each job are configured in the job table, the basic attributes comprise execution frequency, execution period value, job state, a storage path of a statement file, a table name of a result table and execution date of next execution, the statement file stores a first database statement, the job table is stored in a first database, and the result table is stored in a second database;
For each project label job, locating a statement file corresponding to the target job according to a storage path of the statement file corresponding to the target job, and reading a first database statement from the statement file;
For each project standard job, executing a first database statement corresponding to the target job, writing an execution result into a result table corresponding to the target job, modifying the job state of the target job to be completed, and generating a new job according to the target job and adding the new job into the job table.
2. The method of claim 1, wherein executing the first database statement corresponding to the target job, writing the execution result into the result table corresponding to the target job, and modifying the job status of the target job to completed, comprises:
modifying the job state of the target job into execution;
executing a first database statement corresponding to the target job;
judging whether the first database statement is successfully executed or not;
If yes, writing the execution result into a result table corresponding to the target job and modifying the job state of the target job to be completed;
if not, modifying the operation state of the target operation to be failed.
3. The method of claim 1, wherein generating a new job from the target job and adding it to the job table comprises:
And generating a new job according to the execution frequency, the execution period value and the current date of the target job, and adding the new job into the job table.
4. A method according to claim 3, wherein generating a new job based on the execution frequency, execution cycle value and current date of the target job comprises:
determining the execution date of the next execution according to the execution frequency, the execution period value and the current date of the target job;
Generating a new job, and taking the execution date of the next execution as the execution date of the next execution of the new job; the execution frequency, the execution period value, the storage path of the statement file and the table name of the result table of the new job are the same as those of the target job, and the job state of the new job is to be executed.
5. The method of claim 1, further comprising, prior to scanning the job table:
and configuring the basic attribute of each job into a job table according to the job requirement of each job.
6. The method as recited in claim 1, further comprising:
transmitting the execution result of the target job written in the result table to a visualization platform, so as to update the execution result of the target job in a visualization interface.
7. A data synchronization device, comprising:
The query module is used for scanning a job table, and querying each target job with a job state to be executed and a next execution date as a current date from the job table; the basic attributes of each job are configured in the job table, the basic attributes comprise execution frequency, execution period value, job state, a storage path of a statement file, a table name of a result table and execution date of next execution, the statement file stores a first database statement, the job table is stored in a first database, and the result table is stored in a second database;
The reading module is used for positioning the statement file corresponding to the target job according to the storage path of the statement file corresponding to the target job for each project label job, and reading a first database statement from the statement file;
And the execution module is used for executing a first database statement corresponding to the target job for each project standard job, writing an execution result into a result table corresponding to the target job, modifying the job state of the target job to be completed, and generating a new job according to the target job and adding the new job into the job table.
8. The apparatus of claim 7, wherein the execution module is further to:
modifying the job state of the target job into execution;
executing a first database statement corresponding to the target job;
judging whether the first database statement is successfully executed or not;
If yes, writing the execution result into a result table corresponding to the target job and modifying the job state of the target job to be completed;
if not, modifying the operation state of the target operation to be failed.
9. The apparatus of claim 7, wherein the execution module is further to:
And generating a new job according to the execution frequency, the execution period value and the current date of the target job, and adding the new job into the job table.
10. The apparatus of claim 9, wherein the execution module is further to:
determining the execution date of the next execution according to the execution frequency, the execution period value and the current date of the target job;
Generating a new job, and taking the execution date of the next execution as the execution date of the next execution of the new job; the execution frequency, the execution period value, the storage path of the statement file and the table name of the result table of the new job are the same as those of the target job, and the job state of the new job is to be executed.
11. The apparatus of claim 7, further comprising a configuration module configured to:
and configuring the basic attribute of each job into a job table according to the job requirement of each job.
12. The apparatus of claim 7, wherein the execution module is further to:
transmitting the execution result of the target job written in the result table to a visualization platform, so as to update the execution result of the target job in a visualization interface.
13. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
The one or more processors implement the method of any of claims 1-6 when the one or more programs are executed by the one or more processors.
14. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-6.
15. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410274597.2A CN118093719A (en) | 2024-03-11 | 2024-03-11 | Data synchronization method, device, electronic equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410274597.2A CN118093719A (en) | 2024-03-11 | 2024-03-11 | Data synchronization method, device, electronic equipment and computer readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118093719A true CN118093719A (en) | 2024-05-28 |
Family
ID=91163365
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410274597.2A Pending CN118093719A (en) | 2024-03-11 | 2024-03-11 | Data synchronization method, device, electronic equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118093719A (en) |
-
2024
- 2024-03-11 CN CN202410274597.2A patent/CN118093719A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019350B (en) | Data query method and device based on configuration information | |
| CN109189835B (en) | Method and device for generating data wide table in real time | |
| CN111190888A (en) | Method and device for managing graph database cluster | |
| CN111125107A (en) | Data processing method, device, electronic equipment and medium | |
| CN110555068A (en) | Data export method and device | |
| CN110689268A (en) | Method and device for extracting indexes | |
| CN111125064A (en) | Method and device for generating database mode definition statement | |
| CN112597126A (en) | Data migration method and device | |
| CN112000734A (en) | Big data processing method and device | |
| CN113190517B (en) | Data integration method and device, electronic equipment and computer readable medium | |
| CN116775613A (en) | Data migration method, device, electronic equipment and computer readable medium | |
| CN112148705A (en) | Data migration method and device | |
| CN111984686A (en) | Data processing method and device | |
| CN113760240B (en) | Method and device for generating data model | |
| CN113779122B (en) | Method and device for exporting data | |
| CN110688355A (en) | Method and device for changing container state | |
| CN118093719A (en) | Data synchronization method, device, electronic equipment and computer readable medium | |
| CN112835609B (en) | Method and device for modifying download address of dependent packet | |
| CN113722007A (en) | Configuration method, device and system of VPN branch equipment | |
| CN113742321A (en) | Data updating method and device | |
| CN112988857A (en) | Service data processing method and device | |
| CN113779048A (en) | Data processing method and device | |
| CN113535768A (en) | Production monitoring method and device | |
| CN112817930A (en) | Data migration method and device | |
| CN113127416A (en) | Data query method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |