CN109614446A - Method of data synchronization, device, electronic equipment and storage medium - Google Patents
Method of data synchronization, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN109614446A CN109614446A CN201811409607.XA CN201811409607A CN109614446A CN 109614446 A CN109614446 A CN 109614446A CN 201811409607 A CN201811409607 A CN 201811409607A CN 109614446 A CN109614446 A CN 109614446A
- Authority
- CN
- China
- Prior art keywords
- database
- source
- data
- information
- data table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment provides a kind of method of data synchronization, device, electronic equipment and storage mediums, are related to big data technical field.This method comprises: from the parameter information for obtaining source database, the source data table in source database and target database in configuration file；The metadata information of the metadata information of acquisition of information source database and source data table based on the parameter；The metadata information of metadata information, source data table based on source database and the parameter information of target database generate synchronous script；The data on source data table are synchronized to target database based on synchronous script, wherein source database is relevant database, and target database is the database in distributed system.The technical solution of the embodiment of the present invention automatically can synchronize script by Mass production, improve the synchronous efficiency of data.
Description
Technical field
The present invention relates to big data technical field, in particular to a kind of method of data synchronization, data synchronization unit,
Electronic equipment and computer readable storage medium.
Background technique
In the development of information technology, many enterprises or website platform have a large amount of data to deposit in various traditional relationals
In database, to guarantee the customary cold standby of information security needs progress of data, being to carry out data Denver Nuggets to need to establish data warehouse,
These operations require the data for being distributed in thousands of upper tables in various relevant databases being synchronized to distributed system example
In Hadoop.
Currently, tables of data to be synchronized is filtered out using open source data synchronization means Sqoop in a kind of technical solution,
Manual compiling Sqoop synchronizes script, manually performs synchronous script.It is artificial to compile in the case where needing synchronous tables of data very much
Script synchronous with sqoop is executed not only low efficiency is write, but also error rate is high, reduces the synchronous efficiency of data, on the other hand,
It is built in Hadoop in the requirement and various traditional Relational DataBases of table and builds the requirement of table and be not quite similar, such as table name specification, column
Name specification, data type, data encoding etc., manual compiling needs to understand in depth source table when synchronizing script, more time-consuming in this way, low
Effect.
Accordingly, it is desirable to provide a kind of method of data synchronization, the number of the one or more problems being able to solve in the above problem
According to synchronizing device, electronic equipment and computer readable storage medium.
It should be noted that information is only used for reinforcing the reason to background of the present invention disclosed in above-mentioned background technology part
Solution, therefore may include the information not constituted to the prior art known to persons of ordinary skill in the art.
Summary of the invention
The embodiment of the present invention be designed to provide a kind of method of data synchronization, data synchronization unit, electronic equipment and
Computer readable storage medium, and then overcome one caused by the limitation and defect due to the relevant technologies at least to a certain extent
A or multiple problems.
According to a first aspect of the embodiments of the present invention, a kind of method of data synchronization is provided, comprising: obtain from configuration file
Take the parameter information of source database, the source data table in the source database and target database；Information based on the parameter
Obtain the metadata information of the source database and the metadata information of the source data table；Member based on the source database
The parameter information of data information, the metadata information of the source data table and the target database generates synchronous script；Base
The data on the source data table are synchronized to the target database in the synchronous script, wherein the source database is
Relevant database, the target database are the database in distributed system.
In some embodiments of the invention, aforementioned schemes are based on, based on the parameter source database described in acquisition of information
Metadata information and the source data table metadata information, comprising: the source data is obtained from the parameter information
The database linkage information in library；Library link information connects the source database based on the data；After successful connection, obtain
The metadata information of the metadata information of the source database and the source data table.
In some embodiments of the invention, it is based on aforementioned schemes, it is metadata information based on the source database, described
The metadata information of source data table and the parameter information of the target database generate synchronous script, comprising: are based on the source
Wide area information server type obtains the class object of synchronizer corresponding with the type of database；Based on the source database
The parameter information of parameter information, the parameter information of the source data table and the target database instantiates the synchronizer
Class object；The synchronizer after Case-based Reasoning analyzes the metadata information and the source data table of the source database
Metadata information；Result based on analysis generates the synchronization that the data in the source database are synchronized to the target database
Script.
In some embodiments of the invention, aforementioned schemes are based on, the result based on analysis is generated the source database
In data be synchronized to the synchronization script of the target database, comprising: determined in the source database based on the result of analysis
Data and the target database in data transformational relation；It will be in the source database based on transformational relation generation
Data be synchronized to the synchronization script of the target database, the synchronous script is the synchronous script of Sqoop.
In some embodiments of the invention, aforementioned schemes are based on, it will be on the source data table based on the synchronous script
Data be synchronized to the target database, comprising: the source number is obtained by the transformational relation based on the synchronous script
According to the data on the source data table in library；The data that will acquire are synchronized in the corresponding tables of data in the target database.
In some embodiments of the invention, be based on aforementioned schemes, the transformational relation include: automatically skip sky table, from
Turn one of changes data character coding, automatic setting Data Concurrent degree, is compatible with non-Hive standard source data automatically or a variety of.
In some embodiments of the invention, be based on aforementioned schemes, the metadata information of the source database include: data table information,
View information, the metadata information of the source data table include field name, field data types, major key information.
In some embodiments of the invention, aforementioned schemes are based on, the parameter information in the configuration file includes: source number
According to the type in library, the IP address of source database, the port numbers of source database, source database title, source data table name, number of targets
According to library name.
According to a second aspect of the embodiments of the present invention, a kind of data synchronization unit is provided, comprising: parameter information obtains single
Member, for from the parameter for obtaining source database, the source data table in the source database and target database in configuration file
Information；Metadata information acquiring unit, for source database described in acquisition of information based on the parameter metadata information and
The metadata information of the source data table；Synchronous script generation unit, for the metadata information based on the source database, institute
The parameter information of the metadata information and the target database of stating tables of data generates synchronous script；Data synchronisation unit is used
In the data on the source data table are synchronized to the target database based on the synchronous script, wherein the source data
Library is relevant database, and the target database is the database in distributed system.
According to a third aspect of the embodiments of the present invention, a kind of electronic equipment is provided, comprising: processor；And memory,
It is stored with computer-readable instruction on the memory, is realized when the computer-readable instruction is executed by the processor as above
State method of data synchronization described in first aspect.
According to a fourth aspect of the embodiments of the present invention, a kind of computer readable storage medium is stored thereon with computer journey
Sequence realizes the method for data synchronization as described in above-mentioned first aspect when the computer program is executed by processor.
In the technical solution provided by some embodiments of the present invention, on the one hand, based on source database, source data table with
And the parameter information of target database obtains the metadata information of source database and the metadata information of source data table, it can be certainly
The metadata information of source database and source data table is obtained dynamicly；On the other hand, the metadata based on source database, source data table
The parameter information of information and target database generates synchronous script, automatically can synchronize script by Mass production；In another aspect,
It is synchronized to target database to by the data on source data table based on synchronous script, it can be based on synchronous script automatically to source data
Data on table analyzed with it is synchronous, the technical requirements to related personnel are reduced while reducing workload, greatly
Data production process is accelerated, and improves the synchronous efficiency of data, in addition, also improving robustness and to be applicable in scene wider
It is general.
It should be understood that above general description and following detailed description be only it is exemplary and explanatory, not
It can the limitation present invention.
Detailed description of the invention
The drawings herein are incorporated into the specification and forms part of this specification, and shows and meets implementation of the invention
Example, and be used to explain the principle of the present invention together with specification.It should be evident that the accompanying drawings in the following description is only the present invention
Some embodiments for those of ordinary skill in the art without creative efforts, can also basis
These attached drawings obtain other attached drawings.In the accompanying drawings:
Fig. 1 shows the flow diagram of method of data synchronization more according to the present invention；
Fig. 2 shows the schematic diagrames of the content of configuration file according to some embodiments of the present invention；
Fig. 3 shows the flow diagram of the synchronous script of generation according to some embodiments of the present invention；
Fig. 4 shows the schematic block diagram of the data synchronization unit of an exemplary embodiment according to the present invention；
Fig. 5 shows the structural schematic diagram for being suitable for the computer system for the electronic equipment for being used to realize the embodiment of the present invention.
Specific embodiment
Example embodiment is described more fully with reference to the drawings.However, example embodiment can be real in a variety of forms
It applies, and is not understood as limited to embodiment set forth herein；On the contrary, thesing embodiments are provided so that the present invention will be comprehensively and complete
It is whole, and the design of example embodiment is comprehensively communicated to those skilled in the art.Identical appended drawing reference indicates in figure
Same or similar part, thus repetition thereof will be omitted.
In addition, described feature, structure or characteristic can be incorporated in one or more implementations in any suitable manner
In example.In the following description, many details are provided to provide and fully understand to the embodiment of the present invention.However,
It will be appreciated by persons skilled in the art that technical solution of the present invention can be practiced without one or more in specific detail,
Or it can be using other methods, constituent element, device, step etc..In other cases, it is not shown in detail or describes known side
Method, device, realization or operation are to avoid fuzzy each aspect of the present invention.
Block diagram shown in the drawings is only functional entity, not necessarily must be corresponding with physically separate entity.
I.e., it is possible to realize these functional entitys using software form, or realized in one or more hardware modules or integrated circuit
These functional entitys, or these functional entitys are realized in heterogeneous networks and/or processor device and/or microcontroller device.
Flow chart shown in the drawings is merely illustrative, it is not necessary to including all content and operation/step,
It is not required to execute by described sequence.For example, some operation/steps can also decompose, and some operation/steps can close
And or part merge, therefore the sequence actually executed is possible to change according to the actual situation.
Fig. 1 shows the flow diagram of method of data synchronization more according to the present invention.Shown in referring to Fig.1, the number
It may comprise steps of according to synchronous method:
Step S110, from obtaining source database, the source data table in the source database and number of targets in configuration file
According to the parameter information in library；
Step S120, the based on the parameter metadata information of source database described in acquisition of information and the source data table
Metadata information；
Step S130, the metadata information of metadata information, the source data table based on the source database and institute
The parameter information for stating target database generates synchronous script；
Data on the source data table are synchronized to the target database based on the synchronous script by step S140,
Wherein, the source database is relevant database, and the target database is the database in distributed system.
According to the method for data synchronization of the example embodiment of Fig. 1, on the one hand, be based on source database, source data table and mesh
The parameter information for marking database obtains the metadata information of source database and the metadata information of source data table, can be automatically
Obtain the metadata information of source database and source data table；On the other hand, the metadata information based on source database, source data table
And the parameter information of target database generates synchronous script, automatically can synchronize script by Mass production；In another aspect, being based on
Synchronous script is synchronized to target database to by the data on source data table, can be based on synchronous script automatically on source data table
Data analyzed and synchronous, improve the synchronous efficiency of data.
In the following, the method for data synchronization in the example embodiment to Fig. 1 is described in detail.
In step s 110, from obtaining source database, source data table and mesh in the source database in configuration file
Mark the parameter information of database.
In the exemplary embodiment, referring to shown in Fig. 2, may include: in configuration file the type of source database i.e.
Dbloader.db.type, IP address, that is, dbloader.db.ip of source database, source database port numbers be
Dbloader.db.port, source database title, that is, dbloader.db.names, source data table name are
Dbloader.db.tables, Target database name, that is, dbloader.hive.db.names, source database user name be
The information such as password dbloader.db.password of dbloader.db.user, source database.
In the exemplary embodiment, the configuration file can (Extensible Markup Language, can expand for XML
Open up markup language) file, it can parse configuration file and obtain the parameter information of source database in configuration file, the source data
The parameter information of source data table in library and the parameter information of target database.For example, configuration parameter parsing class can be defined
As ConfigParamBean is parsed and verifies the parameter information in configuration file.
It should be noted that in the exemplary embodiment, source database can be relevant database such as MySQL data
The databases such as library, oracle database, DB2 database, SQLserver database, or the relationship type number of other forms
According to library, the present invention is to this without particular determination.The target database is the database in distributed system such as Hadoop
Or the database in Hive cluster.
Step S120, the based on the parameter metadata information of source database described in acquisition of information and the source data table
Metadata information.
In the exemplary embodiment, the database linkage information of the source database is obtained from the parameter information；It is based on
The database linkage information connects the source database；After successful connection, the metadata letter of the source database is obtained
The metadata information of breath and the source data table.For example, the metadata information of available source database (including is not limited to
Table, view relevant information), the metadata information of the source data table that obtains source database (including be not limited to column, type, major key
Etc. relevant informations).
Further, in some embodiments, it can also be used and the source database pair according to the type of source database
SQL (Structured Query Language, the structured query language) dialect answered obtains the total data row of source data table
Number, or the total line number for using the numeric type of iteration and two points of probe methods acquisition source databases to increase major-key table certainly.
Step S130, the metadata information of metadata information, the source data table based on the source database and institute
The parameter information for stating target database generates synchronous script.
In the exemplary embodiment, it can be obtained and the type of database pair based on the type of database of the source database
The syncsort answered；The metadata information of metadata information, the source data table based on the source database and the target
The parameter information of database instantiates the syncsort；Syncsort generation after Case-based Reasoning will be in the source database
Data be synchronized to the synchronization script of the target database.
In some embodiments, the synchronizer that the synchronizer of every kind of type of database can be configured in configuration file is realized
Class, for example, the identification information such as KEY of every kind of type of database can be configured in db_loader_class.properties
Value realizes class with corresponding synchronizer.It can realize that the synchronizer of various types of databases realizes class using factory mode.
For example, can using DBLoader class as synchronizer factory, parameter information, source data table based on source database
Parameter information and the parameter information of target database instantiation synchronizer realize the class object of class, described after Case-based Reasoning
Synchronizer analyzes the metadata information of the source database and the metadata information of the source data table；Result based on analysis
Generate the synchronization script that the data in the source database are synchronized to the target database.
The data in data and target database in source database are determined it is possible to further the result based on analysis
Transformational relation；The synchronization that the data in the source database are synchronized to the target database is generated based on the transformational relation
Script, the synchronous script are the synchronous script of Sqoop.Sqoop is mainly used in Hadoop (Hive) and traditional relationship type number
According to the transmitting for carrying out data between library such as MySQL, the data in MySQL can be led to the HDFS (Hadoop for entering Hadoop
Distributed File System, distributed file system) in.
Further, the table name specification of the structure determination source database based on analysis, column name specification, data type, data
The difference of the metadata of the metadata such as coding and target database generates transformational relation based on the difference.The transformational relation
It may include: automatic conversion data character coding, automatic setting Data Concurrent degree, be compatible with non-Hive standard source data, intelligence automatically
Special setting can be generated to guarantee script normal order, skip the relationships such as sky table automatically.
Data on the source data table are synchronized to the target database based on the synchronous script by step S140.
In the exemplary embodiment, the source in the source database is obtained by the transformational relation based on the synchronous script
Data in tables of data；The data that will acquire are synchronized in the corresponding tables of data in the target database.Further, also
The synchronous synchronized result report of data can be generated, in order to analyze according to synchronized result report data syn-chronization situation.
Fig. 3 shows the flow diagram of method of data synchronization according to some embodiments of the present invention.Referring to Fig. 3 institute
Show,
In step s310, the various parameters information in configuration file is configured.It in the exemplary embodiment, can be with people
Work configures the parameter information in configuration file, configured parameter information can also be written in configuration file.Match
Setting the parameter information in file may include: the IP address of source database, the type of source database, the title of source database.
In step s 320, synchronizer class, that is, DBLoader class is executed.The parameter of source database is read from configuration file
The class object of the instantiating synchronizer class of the parameter information of information, the parameter information of source data table and target database.Further
Ground, can be according to the corresponding synchronizer class of type automatic adaptation of source database, for example, being MySQL database in source database
When, class object, that is, MySQLoader of synchronizer corresponding with the type can be obtained according to the type of source database.
When executing MySQLLoader class, the metadata information of source database and first number of source data table can analyze
It is believed that breath, intelligence carries out parallel processing, the metadata based on source data information and source data table to source database to single table
The analysis result of information generates synchronous script.For example, can generate synchronous script for every tables of data, i.e. n tables of data are raw
At n synchronous script.
Further, the conversion of the data in data and target database in source database is determined based on the result of analysis
Relationship, the transformational relation may include: skip automatically sky table, automatic conversion data character coding, automatic setting Data Concurrent degree,
Automatically it is compatible with non-Hive standard source data, intelligence generates special setting to guarantee script normal order etc..Then, it is closed based on conversion
System generates the synchronization script that the data in the source database are synchronized to the target database, and the synchronous script is
Sqoop synchronizes script.
In step S330, the data on the source data table in source database are obtained by the synchronization script of generation.It is based on
Synchronous script obtains the data on the source data table in source database by the transformational relation；The data that will acquire are synchronized to institute
The corresponding tables of data in target database is stated, which can be the database in YARN cluster.
Next, during synchronization, synchronized result report can be generated, in order to according to same in step S340
Data syn-chronization situation is analyzed in step result report.
In addition, in an embodiment of the present invention, additionally providing a kind of data synchronization unit.Referring to shown in Fig. 4, the data are same
Step device 400 may include: parameter information acquiring unit 410, metadata information acquiring unit 420, synchronous script generation unit
430 and data synchronisation unit 440.Wherein, parameter information acquiring unit 410 be used for from configuration file obtain source database,
The parameter information of source data table in the source database and target database；Metadata information acquiring unit 420 is used for base
The metadata information of the source database and the metadata information of the source data table are obtained in the parameter information；Synchronous foot
This generation unit 430 is for metadata information, the metadata information of the tables of data and described based on the source database
The parameter information of target database generates synchronous script；Data synchronisation unit 440 is used to be based on the synchronous script for the source
Data in tables of data are synchronized to the target database, wherein the source database is relevant database, the number of targets
According to the database that library is in distributed system.
In some embodiments of the invention, aforementioned schemes are based on, metadata information acquiring unit 420 includes: connection letter
Acquiring unit is ceased, for obtaining the database linkage information of the source database from the parameter information；Database connection is single
Member connects the source database for library link information based on the data；Information acquisition unit, for successful connection it
Afterwards, the metadata information of the source database and the metadata information of the source data table are obtained.
In some embodiments of the invention, aforementioned schemes are based on, synchronous script generation unit 430 includes: that syncsort obtains
Unit is taken, the class pair of synchronizer corresponding with the type of database is obtained for the type of database based on the source database
As；Syncsort instantiating unit, for parameter information, the source data table based on the source database parameter information and
The parameter information of the target database instantiates the class object of the synchronizer；Data analysis unit is used for Case-based Reasoning
The synchronizer afterwards analyzes the metadata information of the source database and the metadata information of the source data table；Script is raw
At unit, the synchronization that the data in the source database are synchronized to the target database is generated for the result based on analysis
Script.
In some embodiments of the invention, aforementioned schemes are based on, script generation unit includes: that transformational relation determines list
Member determines that the conversion of the data in data and the target database in the source database is closed for the result based on analysis
System；Data in the source database are synchronized to the target data for generating based on the transformational relation by generation unit
The synchronization script in library, the synchronous script are the synchronous script of Sqoop.
In some embodiments of the invention, aforementioned schemes are based on, data synchronisation unit 440 includes: data capture unit,
For the data on the source data table in the source database to be obtained by the transformational relation based on the synchronous script；It is synchronous
Unit, the data for will acquire are synchronized in the corresponding tables of data in the target database.
In some embodiments of the invention, be based on aforementioned schemes, the transformational relation include: automatically skip sky table, from
Turn one of changes data character coding, automatic setting Data Concurrent degree, is compatible with non-Hive standard source data automatically or a variety of.
In some embodiments of the invention, aforementioned schemes are based on, the metadata information of the source database includes: data
Table information, view information, the metadata information of the source data table include field name, field data types, major key information.
In some embodiments of the invention, aforementioned schemes are based on, the parameter information in the configuration file includes: source number
According to the type in library, the IP address of source database, the port numbers of source database, source database title, source data table name, number of targets
According to library name.
Due to each functional module of the data synchronization unit 400 of example embodiments of the present invention side synchronous with above-mentioned data
The step of example embodiment of method, is corresponding, therefore details are not described herein.
In an exemplary embodiment of the present invention, a kind of electronic equipment that can be realized the above method is additionally provided.
Below with reference to Fig. 5, it illustrates the computer systems 500 for the electronic equipment for being suitable for being used to realize the embodiment of the present invention
Structural schematic diagram.The computer system 500 of electronic equipment shown in Fig. 5 is only an example, should not be to the embodiment of the present invention
Function and use scope bring any restrictions.
As shown in figure 5, computer system 500 includes central processing unit (CPU) 501, it can be read-only according to being stored in
Program in memory (ROM) 502 or be loaded into the program in random access storage device (RAM) 503 from storage section 508 and
Execute various movements appropriate and processing.In RAM 503, it is also stored with various programs and data needed for system operatio.CPU
501, ROM 502 and RAM 503 is connected with each other by bus 504.Input/output (I/O) interface 505 is also connected to bus
504。
I/O interface 505 is connected to lower component: the importation 506 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 507 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage section 508 including hard disk etc.；
And the communications portion 509 of the network interface card including LAN card, modem etc..Communications portion 509 via such as because
The network of spy's net executes communication process.Driver 510 is also connected to I/O interface 505 as needed.Detachable media 511, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 510, in order to read from thereon
Computer program be mounted into storage section 508 as needed.
Particularly, according to an embodiment of the invention, may be implemented as computer above with reference to the process of flow chart description
Software program.For example, the embodiment of the present invention includes a kind of computer program product comprising be carried on computer-readable medium
On computer program, which includes the program code for method shown in execution flow chart.In such reality
It applies in example, which can be downloaded and installed from network by communications portion 509, and/or from detachable media
511 are mounted.When the computer program is executed by central processing unit (CPU) 501, executes and limited in the system of the application
Above-mentioned function.
It should be noted that computer-readable medium shown in the present invention can be computer-readable signal media or meter
Calculation machine readable storage medium storing program for executing either the two any combination.Computer readable storage medium for example can be --- but not
Be limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or any above combination.Meter
The more specific example of calculation machine readable storage medium storing program for executing can include but is not limited to: have the electrical connection, just of one or more conducting wires
Taking formula computer disk, hard disk, random access storage device (RAM), read-only memory (ROM), erasable type may be programmed read-only storage
Device (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device,
Or above-mentioned any appropriate combination.In the present invention, computer readable storage medium can be it is any include or storage journey
The tangible medium of sequence, the program can be commanded execution system, device or device use or in connection.And at this
In invention, computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal,
Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but unlimited
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Any computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used for
By the use of instruction execution system, device or device or program in connection.Include on computer-readable medium
Program code can transmit with any suitable medium, including but not limited to: wireless, electric wire, optical cable, RF etc. are above-mentioned
Any appropriate combination.
Flow chart and block diagram in attached drawing are illustrated according to the system of various embodiments of the invention, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part of one module, program segment or code of table, a part of above-mentioned module, program segment or code include one or more
Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box
The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical
On can be basically executed in parallel, they can also be executed in the opposite order sometimes, and this depends on the function involved.Also it wants
It is noted that the combination of each box in block diagram or flow chart and the box in block diagram or flow chart, can use and execute rule
The dedicated hardware based systems of fixed functions or operations is realized, or can use the group of specialized hardware and computer instruction
It closes to realize.
Being described in unit involved in the embodiment of the present invention can be realized by way of software, can also be by hard
The mode of part realizes that described unit also can be set in the processor.Wherein, the title of these units is in certain situation
Under do not constitute restriction to the unit itself.
As on the other hand, present invention also provides a kind of computer-readable medium, which be can be
Included in electronic equipment described in above-described embodiment；It is also possible to individualism, and without in the supplying electronic equipment.
Above-mentioned computer-readable medium carries one or more program, when the electronics is set by one for said one or multiple programs
When standby execution, so that the electronic equipment realizes such as above-mentioned method of data synchronization as described in the examples.
For example, the electronic equipment may be implemented as shown in Figure 1: step S110 obtains source number from configuration file
According to the parameter information of source data table and target database in library, the source database；Step S120, believes based on the parameter
Breath obtains the metadata information of the source database and the metadata information of the source data table；Step S130, based on described
The parameter information of the metadata information of source database, the metadata information of the source data table and the target database generates
Synchronous script；Data on the source data table are synchronized to the target database based on the synchronous script by step S140,
Wherein, the source database is relevant database, and the target database is the database in distributed system.
It should be noted that although being referred to several modules for acting the device executed in the above detailed description
Or unit, but this division is not enforceable.In fact, embodiment according to the present invention, above-described two
Or more the feature and function of module or unit can be embodied in a module or unit.Conversely, above-described
One module or the feature and function of unit can be to be embodied by multiple modules or unit with further division.
Through the above description of the embodiments, those skilled in the art is it can be readily appreciated that example described herein is implemented
Mode can also be realized by software realization in such a way that software is in conjunction with necessary hardware.Therefore, according to the present invention
The technical solution of embodiment can be embodied in the form of software products, which can store non-volatile at one
Property storage medium (can be CD-ROM, USB flash disk, mobile hard disk etc.) in or network on, including some instructions are so that a calculating
Equipment (can be personal computer, server, touch control terminal or network equipment etc.) executes embodiment according to the present invention
Method.
Those skilled in the art after considering the specification and implementing the invention disclosed here, will readily occur to of the invention its
Its embodiment.This application is intended to cover any variations, uses, or adaptations of the invention, these modifications, purposes or
Person's adaptive change follows general principle of the invention and including the undocumented common knowledge in the art of the present invention
Or conventional techniques.The description and examples are only to be considered as illustrative, and true scope and spirit of the invention are by following
Claim is pointed out.
It should be understood that the present invention is not limited to the precise structure already described above and shown in the accompanying drawings, and
And various modifications and changes may be made without departing from the scope thereof.The scope of the present invention is limited only by the attached claims.
Claims (11)
1. a kind of method of data synchronization characterized by comprising
From the parameter letter for obtaining source database, the source data table in the source database and target database in configuration file
Breath；
The metadata information of the metadata information of source database described in acquisition of information and the source data table based on the parameter；
The metadata information of metadata information, the source data table based on the source database and the target database
Parameter information generates synchronous script；
The data on the source data table are synchronized to the target database based on the synchronous script, wherein the source number
It is relevant database according to library, the target database is the database in distributed system.
2. method of data synchronization according to claim 1, which is characterized in that source number described in acquisition of information based on the parameter
According to the metadata information in library and the metadata information of the source data table, comprising:
The database linkage information of the source database is obtained from the parameter information；
Library link information connects the source database based on the data；
After successful connection, the metadata information of the source database and the metadata information of the source data table are obtained.
3. method of data synchronization according to claim 1, which is characterized in that the metadata letter based on the source database
The parameter information of breath, the metadata information of the source data table and the target database generates synchronous script, comprising:
Type of database based on the source database obtains the class object of synchronizer corresponding with the type of database；
Parameter information, the parameter information of the source data table and the parameter of the target database based on the source database
The class object of synchronizer described in information instances；
The synchronizer after Case-based Reasoning analyzes the metadata information of the source database and the member of the source data table
Data information；
Result based on analysis generates the synchronization script that the data in the source database are synchronized to the target database.
4. method of data synchronization according to claim 3, which is characterized in that the result based on analysis is generated the source number
The synchronization script of the target database is synchronized to according to the data in library, comprising:
The transformational relation of the data in data and the target database in the source database is determined based on the result of analysis；
The synchronization script that the data in the source database are synchronized to the target database is generated based on the transformational relation,
The synchronous script is the synchronous script of Sqoop.
5. method of data synchronization according to claim 4, which is characterized in that be based on the synchronous script for the source data
Data on table are synchronized to the target database, comprising:
The data on the source data table in the source database are obtained by the transformational relation based on the synchronous script；
The data that will acquire are synchronized in the corresponding tables of data in the target database.
6. method of data synchronization according to claim 4, which is characterized in that the transformational relation includes: to skip sky automatically
Table, automatic conversion data character coding, automatic setting Data Concurrent degree, one of be compatible with non-Hive standard source data automatically or
It is a variety of.
7. method of data synchronization according to any one of claim 1 to 6, which is characterized in that the member of the source database
Data information includes: data table information, view information, and the metadata information of the source data table includes field name, Field Count
According to type, major key information.
8. method of data synchronization according to claim 1, which is characterized in that the parameter information packet in the configuration file
It includes: the type of source database, the IP address of source database, the port numbers of source database, source database title, source data table name
Claim, Target database name.
9. a kind of data synchronization unit characterized by comprising
Parameter information acquiring unit, for from obtained in configuration file source database, the source data table in the source database with
And the parameter information of target database；
Metadata information acquiring unit, metadata information and institute for source database described in acquisition of information based on the parameter
State the metadata information of source data table；
Synchronous script generation unit, the metadata information for metadata information, the tables of data based on the source database
And the parameter information of the target database generates synchronous script；
Data synchronisation unit, for the data on the source data table to be synchronized to the target data based on the synchronous script
Library, wherein the source database is relevant database, and the target database is the database in distributed system.
10. a kind of electronic equipment characterized by comprising processor；And memory, calculating is stored on the memory
Machine readable instruction is realized as described in any one of claims 1 to 8 when the computer-readable instruction is executed by the processor
Method of data synchronization.
11. a kind of computer readable storage medium, is stored thereon with computer program, the computer program is executed by processor
Shi Shixian method of data synchronization for example described in any item of the claim 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811409607.XA CN109614446A (en) | 2018-11-23 | 2018-11-23 | Method of data synchronization, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811409607.XA CN109614446A (en) | 2018-11-23 | 2018-11-23 | Method of data synchronization, device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109614446A true CN109614446A (en) | 2019-04-12 |
Family
ID=66004961
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811409607.XA Pending CN109614446A (en) | 2018-11-23 | 2018-11-23 | Method of data synchronization, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109614446A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110232095A (en) * | 2019-05-21 | 2019-09-13 | 中国平安财产保险股份有限公司 | A kind of method of data synchronization, device, storage medium and server |
| CN110543476A (en) * | 2019-07-03 | 2019-12-06 | 威富通科技有限公司 | Synchronization method and device of database table structure and server |
| CN111125254A (en) * | 2019-12-23 | 2020-05-08 | 北京懿医云科技有限公司 | Database synchronization method and device, electronic equipment and computer readable medium |
| CN112162992A (en) * | 2020-10-12 | 2021-01-01 | 浪潮软件集团有限公司 | Efficient database updating system and method |
| CN112287021A (en) * | 2020-07-13 | 2021-01-29 | 上海柯林布瑞信息技术有限公司 | Data real-time synchronization parameter generation method, data real-time synchronization parameter synchronization method, data real-time synchronization parameter generation device, data real-time synchronization parameter synchronization device, storage medium and terminal |
| CN112364101A (en) * | 2020-11-11 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Data synchronization method and device, terminal equipment and medium |
| CN112364049A (en) * | 2020-11-10 | 2021-02-12 | 中国平安人寿保险股份有限公司 | Data synchronization script generation method, system, terminal and storage medium |
| CN112860793A (en) * | 2021-02-03 | 2021-05-28 | 浪潮云信息技术股份公司 | Method for realizing metadata synchronization between different source databases |
| CN115033647A (en) * | 2022-08-11 | 2022-09-09 | 杭州湖畔网络技术有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN115391459A (en) * | 2022-08-24 | 2022-11-25 | 南京领行科技股份有限公司 | Data synchronization method and device, electronic equipment and computer readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102467529A (en) * | 2010-11-11 | 2012-05-23 | 金蝶软件（中国）有限公司 | Metadata synchronizing method and system |
| CN107766132A (en) * | 2017-06-25 | 2018-03-06 | 平安科技（深圳）有限公司 | Multi-task scheduling method, application server and computer-readable recording medium |
| CN107967316A (en) * | 2017-11-22 | 2018-04-27 | 平安科技（深圳）有限公司 | A kind of method of data synchronization, equipment and computer-readable recording medium |
-
2018
- 2018-11-23 CN CN201811409607.XA patent/CN109614446A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102467529A (en) * | 2010-11-11 | 2012-05-23 | 金蝶软件（中国）有限公司 | Metadata synchronizing method and system |
| CN107766132A (en) * | 2017-06-25 | 2018-03-06 | 平安科技（深圳）有限公司 | Multi-task scheduling method, application server and computer-readable recording medium |
| CN107967316A (en) * | 2017-11-22 | 2018-04-27 | 平安科技（深圳）有限公司 | A kind of method of data synchronization, equipment and computer-readable recording medium |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110232095B (en) * | 2019-05-21 | 2024-04-02 | 中国平安财产保险股份有限公司 | Data synchronization method, device, storage medium and server |
| CN110232095A (en) * | 2019-05-21 | 2019-09-13 | 中国平安财产保险股份有限公司 | A kind of method of data synchronization, device, storage medium and server |
| CN110543476A (en) * | 2019-07-03 | 2019-12-06 | 威富通科技有限公司 | Synchronization method and device of database table structure and server |
| CN111125254A (en) * | 2019-12-23 | 2020-05-08 | 北京懿医云科技有限公司 | Database synchronization method and device, electronic equipment and computer readable medium |
| CN112287021A (en) * | 2020-07-13 | 2021-01-29 | 上海柯林布瑞信息技术有限公司 | Data real-time synchronization parameter generation method, data real-time synchronization parameter synchronization method, data real-time synchronization parameter generation device, data real-time synchronization parameter synchronization device, storage medium and terminal |
| CN112287021B (en) * | 2020-07-13 | 2024-04-05 | 上海柯林布瑞信息技术有限公司 | Method and device for generating and synchronizing real-time data synchronization parameters, storage medium and terminal |
| CN112162992A (en) * | 2020-10-12 | 2021-01-01 | 浪潮软件集团有限公司 | Efficient database updating system and method |
| CN112364049A (en) * | 2020-11-10 | 2021-02-12 | 中国平安人寿保险股份有限公司 | Data synchronization script generation method, system, terminal and storage medium |
| CN112364049B (en) * | 2020-11-10 | 2024-05-17 | 中国平安人寿保险股份有限公司 | Data synchronization script generation method, system, terminal and storage medium |
| CN112364101A (en) * | 2020-11-11 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Data synchronization method and device, terminal equipment and medium |
| CN112860793A (en) * | 2021-02-03 | 2021-05-28 | 浪潮云信息技术股份公司 | Method for realizing metadata synchronization between different source databases |
| CN115033647A (en) * | 2022-08-11 | 2022-09-09 | 杭州湖畔网络技术有限公司 | Data synchronization method and device, electronic equipment and storage medium |
| CN115391459A (en) * | 2022-08-24 | 2022-11-25 | 南京领行科技股份有限公司 | Data synchronization method and device, electronic equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109614446A (en) | Method of data synchronization, device, electronic equipment and storage medium | |
| US11455306B2 (en) | Query classification and processing using neural network based machine learning | |
| CN106980669B (en) | A kind of storage of data, acquisition methods and device | |
| US20160246838A1 (en) | System and method for generating an effective test data set for testing big data applications | |
| CN108920698A (en) | A kind of method of data synchronization, device, system, medium and electronic equipment | |
| CN107590277A (en) | Method of data synchronization, device, electronic equipment and storage medium | |
| CN107967316A (en) | A kind of method of data synchronization, equipment and computer-readable recording medium | |
| NO20171080A1 (en) | Apparatus and methods of data synchronization | |
| US11294958B2 (en) | Managing a distributed knowledge graph | |
| US8489547B2 (en) | System and method for transforming configuration data items in a configuration management database | |
| CN109359026A (en) | Log reporting method, device, electronic equipment and computer readable storage medium | |
| CN108804306A (en) | Method and design for Auto-Test System | |
| CN107506218A (en) | The management method and management system of a kind of configuration file | |
| CN109522341A (en) | Realize method, apparatus, the equipment of the stream data processing engine based on SQL | |
| CN109614439A (en) | Method of data synchronization, device, electronic equipment and storage medium | |
| CN107861933A (en) | The method and apparatus for generating O＆M form | |
| CN110019346A (en) | A kind of data processing method and device based on double primary databases | |
| US9489423B1 (en) | Query data acquisition and analysis | |
| US20170116303A1 (en) | Unified data model | |
| CN109597810A (en) | A kind of task cutting method, device, medium and electronic equipment | |
| CN108984564A (en) | Data-storage system, method and apparatus | |
| CN105550351B (en) | The extemporaneous inquiry system of passenger's run-length data and method | |
| CN109375910A (en) | Class file generation method, device, electronic equipment and storage medium | |
| US20090037484A1 (en) | Programming system for occasionally-connected mobile business applications | |
| US11157267B1 (en) | Evaluation of dynamic relationships between application components |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190412 |
|
| RJ01 | Rejection of invention patent application after publication |