CN108628881A - Method of data synchronization and device - Google Patents
Method of data synchronization and device Download PDFInfo
- Publication number
- CN108628881A CN108628881A CN201710164627.4A CN201710164627A CN108628881A CN 108628881 A CN108628881 A CN 108628881A CN 201710164627 A CN201710164627 A CN 201710164627A CN 108628881 A CN108628881 A CN 108628881A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- target database
- synchronized
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application discloses method of data synchronization and device.One specific implementation mode of this method includes：The request for obtaining data from target database in response to receiving, obtains the regular expression of the address and the file sent for Filtration Goal database of target database；According to the address of target database, to target database transmission data synchronization request；The file sent from target database is received, and file is filtered according to regular expression and obtains data to be synchronized, wherein file is generated according to the data synchronization request received by target database；Data to be synchronized are sent to destination server.The embodiment, which realizes, in real time, efficiently and accurately realizes that destination server is synchronous with the data of target database.
Description
Technical field
This application involves field of computer technology, and in particular to Internet technical field more particularly to method of data synchronization
And device.
Background technology
As electric business platform data amount is growing, the data of network application are only only stored in a database, can be made
The processing capacity for obtaining database encounters bottleneck.Some system divides table technology to store data using a point library gradually at present
In multiple databases.But accompanying problem is that when wanting aggregate query, data in single library are simultaneously imperfect, in order to
Meet such demand to need to get up data aggregate of the hash in multiple databases.
The method for the storage engines storage data that the prior art uses does not support db transaction, and for the lock of data
Granularity is all table lock, in the application of big data high concurrent, is less applicable in.It is realized using the application of client-server model
The polymerization of data can increase the expense of network.It can once net when getting binary log file from database server
Network interacts, and network interaction can also occur when client and server-side interact, also had when purpose storage gets the message
Network interaction.In the case where data volume is big, frequent network interaction can so that data delay is serious, execute aggregate query
When, obtained result nor real-time results.When client or server-side either side when something goes wrong, the synchronization of data has can
Can have an impact, and increase the difficulty of investigation problem.
Invention content
The purpose of the application is to propose a kind of improved method of data synchronization and device, to solve background above technology department
Divide the technical issues of mentioning.
In a first aspect, this application provides a kind of method of data synchronization, this method includes：In response to receiving from number of targets
The request that data are obtained according to library, obtains the canonical table of the address and the file sent for Filtration Goal database of target database
Up to formula；According to the address of target database, to target database transmission data synchronization request；What reception was sent from target database
File, and file is filtered according to regular expression and obtains data to be synchronized, wherein file is by target database according to receiving
Data synchronization request generate；Data to be synchronized are sent to destination server.
In some embodiments, file is filtered according to regular expression and obtains data to be synchronized, including：Resolution file obtains
Data structure information set, wherein data structure information includes：Database name, table name and data content；From data structure information
The data structure information of database name and table name and regular expression matching is found out in set；The data structure found out is believed
It ceases corresponding data content and is combined into data to be synchronized.
In some embodiments, destination server is message-oriented middleware server；And it is waited for destination server transmission same
Step data, including：Data to be synchronized are packaged；It will be in the data publication to message to be synchronized after encapsulation by message queue
Between in part server in scheduled theme, wherein scheduled theme is subscribed to by distributed search engine.
In some embodiments, after sending data to be synchronized to destination server, this method further includes：Record file
Filename and data to be synchronized location information hereof, and using filename and location information as sync bit confidence
Breath.
In some embodiments, this method further includes：Sync bit information is sent to target database.
In some embodiments, this document is the binary log file of relevant database.
Second aspect, this application provides a kind of data synchronization unit, which includes：Acquiring unit, in response to
The request for obtaining data from target database is received, the address of target database is obtained and is sent for Filtration Goal database
File regular expression；Request unit is synchronized for the address according to target database to target database transmission data
Request；Receiving unit for receiving the file sent from target database, and filters file according to regular expression and obtains waiting for together
Step data, wherein file is generated according to the data synchronization request received by target database；Transmission unit, for
Destination server sends data to be synchronized.
In some embodiments, receiving unit is further used for：Resolution file obtains data structure information set, wherein
Data structure information includes：Database name, table name and data content；Found out from data structure information set database name and
The data structure information of table name and regular expression matching；The corresponding data content of the data structure information found out is combined into
Data to be synchronized.
In some embodiments, destination server is message-oriented middleware server；And transmission unit is further used for：It will
Data to be synchronized are packaged；It will be pre- in the data publication to be synchronized to message-oriented middleware server after encapsulation by message queue
In fixed theme, wherein scheduled theme is subscribed to by distributed search engine.
In some embodiments, which further includes：Recording unit, for sending data to be synchronized to destination server
Later, record the location information of the filename and data to be synchronized of file hereof, and using filename and location information as
Sync bit information.
In some embodiments, which further includes：Sync bit information transmitting unit, for being sent to target database
Sync bit information.
In some embodiments, this document is the binary log file of relevant database.
The third aspect, this application provides a kind of equipment, including：One or more processors；Storage device, for storing
One or more programs；When one or more programs are executed by one or more processors so that one or more processors are real
Now such as the method in first aspect any embodiment.
Fourth aspect, this application provides a kind of computer readable storage mediums, are stored thereon with computer program, the journey
It is realized such as the method in first aspect any embodiment when sequence is executed by processor.
Method of data synchronization and device provided by the present application, it is same by waiting for obtain to send out data synchronization request to target database
The file of step data composition, and data to be synchronized are filtered out from this document using preset regular expression, it is then issued to mesh
Mark server.It can real-time, quickly and accurately ensure that target database is synchronous with the data of destination server, reduce network and hand over
Mutually and reduce data delay.
Description of the drawings
By reading a detailed description of non-restrictive embodiments in the light of the attached drawings below, the application's is other
Feature, objects and advantages will become more apparent upon：
Fig. 1 is that this application can be applied to exemplary system architecture figures therein；
Fig. 2 is the flow chart according to one embodiment of the method for data synchronization of the application；
Fig. 3 is the flow chart according to another embodiment of the method for data synchronization of the application；
Fig. 4 is the structural schematic diagram according to one embodiment of the data synchronization unit of the application；
Fig. 5 is adapted for the structural schematic diagram of the computer system of the server for realizing the embodiment of the present application.
Specific implementation mode
The application is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining related invention, rather than the restriction to the invention.It also should be noted that in order to
Convenient for description, is illustrated only in attached drawing and invent relevant part with related.
It should be noted that in the absence of conflict, the features in the embodiments and the embodiments of the present application can phase
Mutually combination.The application is described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
Fig. 1 shows the exemplary system of the embodiment of the method for data synchronization or data synchronization unit that can apply the application
System framework 100.
As shown in Figure 1, system architecture 100 may include target database 101,102,103, and server 104, task scheduling
Center 105, data aggregate configuration center 106 and destination server 107 (that is, message-oriented middleware server).Target database
101, between 102,103 and server 105, server 105 and task scheduling center 105, data aggregate configuration center 106 it
Between, it between server 105 and destination server 107, is communicated with various connection types, such as wired, wireless communication link
Or fiber optic cables etc..
Server 104 can be with task scheduling center 105,106 interaction data of data aggregate configuration center.In task scheduling
A part as server 104 after the heart 105 and data aggregation configuration center 106 can also merge.In task scheduling
The heart 105 and data aggregation configuration center 106 can also merge into a server.Task scheduling center 105 is each for storing
The relevant information of task, for example, the information such as mission number, task status.A, task are stored in data aggregate configuration center 106
With the relationship of target database 101,102,103, including but not limited to：The address of target database 101,102,103, port,
The information such as user name, the password of target database 101,102,103.B, task are also stored in data aggregate configuration center 106
With the relationship of message queue, including but not limited to：Mission number, regular expression, message subject, message-oriented middleware server
Location, message-oriented middleware server user name, password, message send the information such as overtime duration.In data aggregate configuration center 106 also
C, task and database file relationship are stored, including but not limited to：Mission number, target database 101,102,103 store
Filename, the file currently read position.Server 104 selects after obtaining mission bit stream from task scheduling center 105
Unappropriated task obtains the mission number by the mission number of the unappropriated task from data aggregate configuration center 106
Corresponding target database address and the regular expression for filtering data in file, then to target database 101,102,
103 transmission data synchronization requests.
Target database 101,102,103 actively can transmit file after receiving data synchronization request to server 104.
Server 104, which receives document analysis and goes out the data contents of needs, is then forwarded to destination server 107.Server 104 can receive
File from different target database.
Destination server 107 can be message-oriented middleware server, for search engine server (for example,
ElasticSearch provides the full-text search engine of a distributed multi-user ability.Elasticsearch is to use Java
Exploitation, and be Enterprise search engine currently popular as the open source code publication under Apache license terms) it subscribes to and disappears
Breath.Destination server 107 can screen the data received, and required data are stored.Destination server 107 can
Data of the polymerization from different target database are inquired for user by search engine server.
It should be noted that the method for data synchronization that the embodiment of the present application is provided generally is executed by server 104, accordingly
Ground, data synchronization unit are generally positioned in server 104.
It should be understood that the target database, server, task scheduling center, data aggregate configuration center in Fig. 1 and target
The number of server is only schematical.According to realize needs, can have any number of target database, server,
Task scheduling center, data aggregate configuration center and destination server.In server, task scheduling center, data aggregate configuration
The heart can be integrated into a server.
With continued reference to Fig. 2, the flow 200 of one embodiment of the method for data synchronization according to the application is shown.The number
According to synchronous method, include the following steps：
Step 201, in response to receive from target database obtain data request, obtain target database address and
Regular expression for the file that Filtration Goal database is sent.
In the present embodiment, the electronic equipment (such as server 104 shown in FIG. 1) of method of data synchronization operation thereon
It can be received from task scheduling center 105 from target database by wired connection mode or radio connection and obtain number
According to request.Task scheduling center 105 is used to manage the task that data synchronize that executes, that is, thread or process, each task is to appoint
Business number is distinguished.Task scheduling center 105 can distribute the server address for executing the task (for example, Fig. 1 institutes for each task
The address of the server 104 shown), and task status can be obtained in real time.Just whether 105 periodic detection task of task scheduling center
Often operation needs to restart, restarts unsuccessfully, can redistribute task if task current state is abnormal end.Therefore availability
The configuration of detection need to include：Mission number, detection time interval attempt to restart number.Task scheduling center 105 receives target
After the data update message that database is sent, unappropriated task is obtained, and to the server for executing the unappropriated task
Location sends the request that data are obtained from target database.Exist in multiple target databases and multiple destination server aggregated applications
When, pass through Task Scheduling Mechanism so that the load of server 104 is almost the same.If task scheduling center 105 and server
104 when on same computer, can not configure the server address for executing the task, directly execute acquisition by the computer
The address of target database and for Filtration Goal database send file regular expression operation, otherwise by execute should
The server 104 of the server address instruction of task executes aforesaid operations.It can in advance (can in data aggregate configuration center 106
To be located in server 104) in the corresponding target database of each mission number of storage address, target data can also be stored
The username and password in library.Target database can be distributed relation database (for example, MySQL database).
Step 202, according to the address of target database, to target database transmission data synchronization request.
In the present embodiment, the address of the target database obtained based on step 201, it is same to target database transmission data
Step request.If target database needs to authenticate, the use that target database is obtained from data aggregate configuration center 106 is also needed to
Name in an account book and password are simultaneously sent.For example, for the case where target database is MySQL database, DUMP order (MySQL numbers are sent
According to backup command in library), for by the data of MySQL database with the SQL of standard (Structured Query Language,
Structured query language) mode of language exports, and is saved in file.
Step 203, the file that is sent from target database is received, and is obtained according to regular expression filtering file to be synchronized
Data.
In the present embodiment, this document is generated according to the data synchronization request received by the target database.
It can be the file of incremental data composition, can also be the file of full dose data composition.It is obtained according to step 201 after receiving this document
The regular expression filtering file obtained obtains data to be synchronized.The regular expression be used for involved in file database name,
The information such as data table name are screened.
In some optional realization methods of the present embodiment, this document is the binary log text of relevant database
Part.For example, the binary log file of MySQL database records user couple for recording all users to database manipulation
The SQL statement of database manipulation.If there is this file, when accident occurs for database, this Fileview to user can be passed through
The operation that user is done within the period of this file record, then be used cooperatively with DB Backup, you can user's operation is reproduced,
Make database recovery.And leader follower replication may be implemented, that is, the number between target database 101,102,103 and server 104
According to synchronization.Essence is exactly by the binary log file copy of target database 101,102,103 to server 104, to make
Proper target database 101,102,103 when something goes wrong, can be quickly switched into 104 offer service of server.It can also take
Inquiry operation is executed on business device 104, reduces the access pressure of target database 101,102,103.It can also be on the server 104
Backup is executed, to avoid the service of target database 101,102,103 is influenced during backup.
In some optional realization methods of the present embodiment, file is filtered according to regular expression and obtains number to be synchronized
According to, including：Resolution file obtains data structure information set, wherein data structure information includes：Database name, table name sum number
According to content；The data structure that database name and table name and regular expression matching are found out from data structure information set is believed
Breath；The corresponding data content of the data structure information found out is combined into data to be synchronized.It can be wrapped in the file received
Multirow data are included, each row of data is formed by three database name, table name and data content fields.Regular expression includes
Database name and table name, need to parse the data content with regular expression matching from file.For example, regular expression
For " content of database 1, table 1 ", file is：
" database 1, table 1, aaa
Database 1, table 2, bbb
Database 1, table 1, ccc "
The data content obtained after then filtering is " aaa " and " ccc ", is combined into data to be synchronized " aaaccc ".
Can also retain a data library name section and table name field " database 1, table 1 ", then data to be synchronized be " database 1, table 1,
aaaccc”。
Step 204, data to be synchronized are sent to destination server.
In the present embodiment, the data to be synchronized that step 203 obtains are sent to destination server 107.Destination server
Address can be stored in advance in data aggregate configuration center 106 (can be located at server 104 in).And in data aggregate configuration
User name, password, the message that the heart 106 can also store destination server send the information such as overtime duration.In the mesh authenticated to needs
When marking server transmission data to be synchronized, the user name, the password that send destination server are needed.And it may also be arranged to sending
It is retransmitted when failure.Successful message is sent if do not received in overtime duration period, then it is assumed that is sent failure, can be carried out
It retransmits.It can also be pre-configured with maximum retransmission, in order to avoid ceaselessly retransmitted.Can also detect send failure when,
Other unappropriated task execution above-mentioned steps 201-204 are obtained to be retransmitted.
In some optional realization methods of the present embodiment, destination server is message-oriented middleware server；And to
Destination server sends data to be synchronized, including：Data to be synchronized are packaged；It will be waited for together after encapsulation by message queue
Step data is distributed in message-oriented middleware server in scheduled theme, wherein scheduled theme is ordered by distributed search engine
It reads.Message-oriented middleware is applicable to the distributed environment of reliable data transmission.In system using message-oriented middleware mechanism,
The event of other side is activated by transmitting message between different objects, completes corresponding operation.Sender transmits the message to
Message-oriented middleware server, message-oriented middleware server by message deposit in several queues, again by message when suitable
It is transmitted to recipient.Message-oriented middleware can communicate between different platform, it be often used to mask various platforms and agreement it
Between characteristic, realize the collaboration between application program, the advantage is that can be provided between client and server it is synchronous with it is different
The connection of step, and can message be transmitted or be stored at any time forwarding, this is also it than remote process tune
With further reason.Message refers to the base unit of communication.By the message producer (producer) publication about certain word
The message of (topic) is inscribed, it means that message has been sent to using a kind of physics mode as agency's (broker)
Server (may be an other machine).Several message users (consumer) subscribes to (subscribe) some topic,
Then every message that the producer is issued can all be sent to all users.In the present embodiment, message refers to sealing
Data to be synchronized after dress, topic can be the titles of target database, and proxy server refers to that destination server, message make
User is distributed search engine (for example, ElasticSearch).Distributed search engine unseals the data to be synchronized received
After dress, it would be desirable to data be stored for using when aggregate query.Real-time and efficiently data aggregate is stored in
In ElasticSearch, solves dispersion data aggregate inquiry so that developer and operation maintenance personnel are more convenient neatly to use
Data.
The method that above-described embodiment of the application provides by the way that the data of target database are synchronized in server 104,
Be then forwarded to destination server, realize the data of hash in real time, efficiently, accurately transmit and be aggregated to destination server
In, and provide good failure transfer scheme.In the case of high concurrent, big data quantity, data still can be with real-time synchronization
And it is aggregated in destination server.
With further reference to Fig. 3, it illustrates the flows 300 of another embodiment of method of data synchronization.The data synchronize
The flow 300 of method, includes the following steps：
Step 301, in response to receive from target database obtain data request, obtain target database address and
Regular expression for the file that Filtration Goal database is sent.
Step 302, according to the address of target database, to target database transmission data synchronization request.
Step 303, the file that is sent from target database is received, and is obtained according to regular expression filtering file to be synchronized
Data.
Step 304, data to be synchronized are sent to destination server.
Step 301-304 and step 201-204 are essentially identical, therefore repeat no more.
Step 305, the location information of the filename and data to be synchronized of file hereof is recorded, and by filename and position
Confidence breath is as sync bit information.
In the present embodiment, after the file for receiving target database transmission every time, filename is recorded, and be recorded in basis
Regular expression filtering file obtains the position of the data to be synchronized obtained when data to be synchronized in this document, i.e., apart from file
The offset of initial position.The data of synchronized mistake can be determined according to filename and location information.It can will synchronize
Location information is stored in data aggregate configuration center 106 (can be located in server 104), in case follow-up transmission data synchronization is asked
In data synchronization request include the sync bit information when asking.
Step 306, sync bit information is sent to target database.
In the present embodiment, sync bit confidence can be sent when next time is to target database transmission data synchronization request
Breath, or can individually send a sync bit information.The position of synchronized data is indicated to target database,
So that target database determines the data not synchronized, generates file further according to not synchronous data and be sent to server 104.
From figure 3, it can be seen that compared with the corresponding embodiments of Fig. 2, the flow of the method for data synchronization in the present embodiment
300 the step of simply requesting incremental data when highlighting transmission data synchronization request.Without full dose synchrodata, incremental number is only synchronized
According to, it is possible to reduce network flow can also be arranged and carry out full dose data synchronization in the predetermined time, remaining time uses increment
Data synchronize, and can both ensure that the accuracy that data synchronize can also reduce network pressure.
It synchronizes and fills this application provides a kind of data as the realization to method shown in above-mentioned each figure with further reference to Fig. 4
The one embodiment set, the device embodiment is corresponding with embodiment of the method shown in Fig. 2, which specifically can be applied to respectively
In kind electronic equipment.
As shown in figure 4, the data synchronization unit 400 of the present embodiment includes：Acquiring unit 401, receives request unit 402
Unit 403 and transmission unit 404.Wherein, acquiring unit 401 is used to obtain asking for data in response to receiving from target database
It asks, obtains the regular expression of the address and the file sent for Filtration Goal database of target database；Request unit 402
For the address according to target database, to target database transmission data synchronization request；Receiving unit 403 is for receiving from mesh
The file that database is sent is marked, and file is filtered according to regular expression and obtains data to be synchronized, wherein file is by number of targets
It is generated according to the data synchronization request received according to library；Transmission unit 404 is used to send data to be synchronized to destination server.
In the present embodiment, the acquiring unit 401 of data synchronization unit 400, request unit 402, receiving unit 403 and hair
Send the specific processing of unit 404 can be with step 201, step 202, step 203, the step 204 in 2 corresponding embodiment of reference chart.
In some optional realization methods of the present embodiment, receiving unit 403 is further used for：Resolution file number
According to structural information set, wherein data structure information includes：Database name, table name and data content；From data structure information collection
The data structure information of database name and table name and regular expression matching is found out in conjunction；The data structure information that will be found out
Corresponding data content is combined into data to be synchronized.
In some optional realization methods of the present embodiment, destination server is message-oriented middleware server；And hair
Unit is sent to be further used for：Data to be synchronized are packaged；By message queue by the data publication to be synchronized after encapsulation extremely
In message-oriented middleware server in scheduled theme, wherein scheduled theme is subscribed to by distributed search engine.
In some optional realization methods of the present embodiment, device 400 further includes：Recording unit, for target
After server sends data to be synchronized, the location information of the filename and data to be synchronized of file hereof is recorded, and will
Filename and location information is as sync bit information.
In some optional realization methods of the present embodiment, device 400 further includes：Sync bit information transmitting unit,
For sending sync bit information to target database.
In some optional realization methods of the present embodiment, file is the binary log file of relevant database.
Below with reference to Fig. 5, it illustrates the computers suitable for terminal device/server for realizing the embodiment of the present application
The structural schematic diagram of system 500.Terminal device/server shown in Fig. 5 is only an example, should not be to the embodiment of the present application
Function and use scope bring any restrictions.
As shown in figure 5, computer system 500 includes central processing unit (CPU) 501, it can be read-only according to being stored in
Program in memory (ROM) 502 or be loaded into the program in random access storage device (RAM) 503 from storage section 508 and
Execute various actions appropriate and processing.In RAM 503, also it is stored with system 500 and operates required various programs and data.
CPU 501, ROM 502 and RAM 503 are connected with each other by bus 504.Input/output (I/O) interface 505 is also connected to always
Line 504.
It is connected to I/O interfaces 505 with lower component：Importation 506 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 507 of spool (CRT), liquid crystal display (LCD) etc. and loud speaker etc.；Storage section 508 including hard disk etc.；
And the communications portion 509 of the network interface card including LAN card, modem etc..Communications portion 509 via such as because
The network of spy's net executes communication process.Driver 510 is also according to needing to be connected to I/O interfaces 505.Detachable media 511, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on driver 510, as needed in order to be read from thereon
Computer program be mounted into storage section 508 as needed.
Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description
Software program.For example, embodiment of the disclosure includes a kind of computer program product comprising be carried on computer-readable medium
On computer program, which includes the program code for method shown in execution flow chart.In such reality
It applies in example, which can be downloaded and installed by communications portion 509 from network, and/or from detachable media
511 are mounted.When the computer program is executed by central processing unit (CPU) 501, limited in execution the present processes
Above-mentioned function.It should be noted that computer-readable medium described herein can be computer-readable signal media or
Computer readable storage medium either the two arbitrarily combines.Computer readable storage medium for example can be --- but
Be not limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or arbitrary above combination.
The more specific example of computer readable storage medium can include but is not limited to：Electrical connection with one or more conducting wires,
Portable computer diskette, hard disk, random access storage device (RAM), read-only memory (ROM), erasable type may be programmed read-only deposit
Reservoir (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory
Part or above-mentioned any appropriate combination.In this application, computer readable storage medium can any be included or store
The tangible medium of program, the program can be commanded the either device use or in connection of execution system, device.And
In the application, computer-readable signal media may include the data letter propagated in a base band or as a carrier wave part
Number, wherein carrying computer-readable program code.Diversified forms may be used in the data-signal of this propagation, including but not
It is limited to electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be computer
Any computer-readable medium other than readable storage medium storing program for executing, the computer-readable medium can send, propagate or transmit use
In by instruction execution system, device either device use or program in connection.Include on computer-readable medium
Program code can transmit with any suitable medium, including but not limited to：Wirelessly, electric wire, optical cable, RF etc., Huo Zheshang
Any appropriate combination stated.
Flow chart in attached drawing and block diagram, it is illustrated that according to the system of the various embodiments of the application, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part for a part for one module, program segment, or code of table, the module, program segment, or code includes one or more uses
The executable instruction of the logic function as defined in realization.It should also be noted that in some implementations as replacements, being marked in box
The function of note can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are actually
It can be basically executed in parallel, they can also be executed in the opposite order sometimes, this is depended on the functions involved.Also it to note
Meaning, the combination of each box in block diagram and or flow chart and the box in block diagram and or flow chart can be with holding
The dedicated hardware based system of functions or operations as defined in row is realized, or can use specialized hardware and computer instruction
Combination realize.
Being described in unit involved in the embodiment of the present application can be realized by way of software, can also be by hard
The mode of part is realized.Described unit can also be arranged in the processor, for example, can be described as：A kind of processor packet
Include acquiring unit, request unit, receiving unit and transmission unit.Wherein, the title of these units not structure under certain conditions
The restriction of the pairs of unit itself, for example, acquiring unit is also described as " obtaining in response to receiving from target database
The request of data obtains the address of the target database and the canonical table for filtering the file that the target database is sent
Up to the unit of formula ".
As on the other hand, present invention also provides a kind of computer-readable medium, which can be
Included in device described in above-described embodiment；Can also be individualism, and without be incorporated the device in.Above-mentioned calculating
Machine readable medium carries one or more program, when said one or multiple programs are executed by the device so that should
Device：The request for obtaining data from target database in response to receiving, obtains the address of target database and for filtering mesh
Mark the regular expression for the file that database is sent；According to the address of target database, synchronized to target database transmission data
Request；The file sent from target database is received, and file is filtered according to regular expression and obtains data to be synchronized, wherein
File is generated according to the data synchronization request received by target database；Data to be synchronized are sent to destination server.
Above description is only the preferred embodiment of the application and the explanation to institute's application technology principle.People in the art
Member should be appreciated that invention scope involved in the application, however it is not limited to technology made of the specific combination of above-mentioned technical characteristic
Scheme, while should also cover in the case where not departing from the inventive concept, it is carried out by above-mentioned technical characteristic or its equivalent feature
Other technical solutions of arbitrary combination and formation.Such as features described above has similar work(with (but not limited to) disclosed herein
Can technical characteristic replaced mutually and the technical solution that is formed.
Claims (14)
1. a kind of method of data synchronization, which is characterized in that the method includes：
The request for obtaining data from target database in response to receiving, obtains the address of the target database and for filtering
The regular expression for the file that the target database is sent；
According to the address of the target database, to the target database transmission data synchronization request；
The file that is sent from the target database is received, and is obtained according to the regular expression filtering file to be synchronized
Data, wherein the file is generated according to the data synchronization request received by the target database；
The data to be synchronized are sent to destination server.
2. according to the method described in claim 1, it is characterized in that, described obtain according to the regular expression filtering file
To data to be synchronized, including：
It parses the file and obtains data structure information set, wherein data structure information includes：Database name, table name sum number
According to content；
The data knot of database name and table name and the regular expression matching is found out from the data structure information set
Structure information；
The corresponding data content of the data structure information found out is combined into data to be synchronized.
3. according to the method described in claim 1, it is characterized in that, the destination server is message-oriented middleware server；With
And
It is described to send the data to be synchronized to destination server, including：
The data to be synchronized are packaged；
By message queue by scheduled theme in the data publication to be synchronized after encapsulation to the message-oriented middleware server,
Wherein, the scheduled theme is subscribed to by distributed search engine.
4. according to the method described in claim 1, it is characterized in that, sending the data to be synchronized to destination server described
Later, the method further includes：
Record the location information of the filename and the data to be synchronized of the file in the file, and by the filename
With the location information as sync bit information.
5. according to the method described in claim 4, it is characterized in that, the method further includes：
To the target database send described in sync bit information.
6. according to claim 1-5 any one of them methods, which is characterized in that the file be relevant database two into
Journal file processed.
7. a kind of data synchronization unit, which is characterized in that described device includes：
Acquiring unit, the request for obtaining data from target database in response to receiving, obtains the target database
Address and regular expression for filtering the file that the target database is sent；
Request unit, for the address according to the target database, to the target database transmission data synchronization request；
Receiving unit, for receiving the file sent from the target database, and according to described in regular expression filtering
File obtains data to be synchronized, wherein the file is asked according to the data synchronization received by the target database
It seeks survival；
Transmission unit, for sending the data to be synchronized to destination server.
8. device according to claim 7, which is characterized in that the receiving unit is further used for：
It parses the file and obtains data structure information set, wherein data structure information includes：Database name, table name sum number
According to content；
The data knot of database name and table name and the regular expression matching is found out from the data structure information set
Structure information；
The corresponding data content of the data structure information found out is combined into data to be synchronized.
9. device according to claim 7, which is characterized in that the destination server is message-oriented middleware server；With
And
The transmission unit is further used for：
The data to be synchronized are packaged；
By message queue by scheduled theme in the data publication to be synchronized after encapsulation to the message-oriented middleware server,
Wherein, the scheduled theme is subscribed to by distributed search engine.
10. device according to claim 7, which is characterized in that described device further includes：
Recording unit, for after the data to be synchronized to destination server transmission, recording the file of the file
Name and location information of the data to be synchronized in the file, and using the filename and the location information as same
Walk location information.
11. device according to claim 10, which is characterized in that described device further includes：
Sync bit information transmitting unit, for sync bit information described in being sent to the target database.
12. according to claim 7-11 any one of them devices, which is characterized in that the file is the two of relevant database
System journal file.
13. a kind of equipment, including：
One or more processors；
Storage device, for storing one or more programs；
When one or more of programs are executed by one or more of processors so that one or more of processors are real
The now method as described in any in claim 1-6.
14. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor
The method as described in any in claim 1-6 is realized when execution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710164627.4A CN108628881A (en) | 2017-03-20 | 2017-03-20 | Method of data synchronization and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710164627.4A CN108628881A (en) | 2017-03-20 | 2017-03-20 | Method of data synchronization and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108628881A true CN108628881A (en) | 2018-10-09 |
Family
ID=63687321
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710164627.4A Pending CN108628881A (en) | 2017-03-20 | 2017-03-20 | Method of data synchronization and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108628881A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109614440A (en) * | 2018-10-25 | 2019-04-12 | 深圳壹账通智能科技有限公司 | Method of data synchronization and relevant device based on big data |
| CN109785129A (en) * | 2018-12-14 | 2019-05-21 | 平安证券股份有限公司 | Data capture method, device, equipment and computer readable storage medium |
| CN110083579A (en) * | 2019-03-21 | 2019-08-02 | 深圳壹账通智能科技有限公司 | Incremental data synchronous method, apparatus, computer equipment and computer storage medium |
| CN111309747A (en) * | 2020-02-18 | 2020-06-19 | 京东数字科技控股有限公司 | Data synchronization method, system and device |
| CN111460038A (en) * | 2020-04-07 | 2020-07-28 | 中国建设银行股份有限公司 | Quasi-real-time data synchronization method and device |
| CN111475309A (en) * | 2019-01-24 | 2020-07-31 | 北京京东尚科信息技术有限公司 | Data processing method, device, block chain service system and storage medium |
| CN112925841A (en) * | 2021-03-26 | 2021-06-08 | 瀚高基础软件股份有限公司 | Distributed JDBC implementation method, device and computer-readable storage medium |
| CN113010609A (en) * | 2020-12-23 | 2021-06-22 | 上海海鼎信息工程股份有限公司 | Differentiated synchronization method and system applied to store operation |
| CN114374668A (en) * | 2021-12-30 | 2022-04-19 | 北京北信源软件股份有限公司 | Information processing method, information processing device, electronic equipment and storage medium |
| CN114500645A (en) * | 2021-12-30 | 2022-05-13 | 兰州乐智教育科技有限责任公司 | Data acquisition system and data acquisition method |
| CN114756628A (en) * | 2022-06-16 | 2022-07-15 | 天津七一二移动通信有限公司 | High-multiplexing extensible data and file synchronization method |
| CN116737678A (en) * | 2023-08-14 | 2023-09-12 | 浙江同信企业征信服务有限公司 | Data synchronization method, device, equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101557380A (en) * | 2008-04-10 | 2009-10-14 | 华为技术有限公司 | Data synchronization method, client terminal and server |
| US20130091105A1 (en) * | 2011-10-05 | 2013-04-11 | Ajit Bhave | System for organizing and fast searching of massive amounts of data |
| CN105574109A (en) * | 2015-12-10 | 2016-05-11 | 青岛海信传媒网络技术有限公司 | Database synchronizing method and equipment and system |
| CN106407288A (en) * | 2016-08-29 | 2017-02-15 | 上海掌门科技有限公司 | Method and system for synchronously updating information |
-
2017
- 2017-03-20 CN CN201710164627.4A patent/CN108628881A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101557380A (en) * | 2008-04-10 | 2009-10-14 | 华为技术有限公司 | Data synchronization method, client terminal and server |
| US20130091105A1 (en) * | 2011-10-05 | 2013-04-11 | Ajit Bhave | System for organizing and fast searching of massive amounts of data |
| CN105574109A (en) * | 2015-12-10 | 2016-05-11 | 青岛海信传媒网络技术有限公司 | Database synchronizing method and equipment and system |
| CN106407288A (en) * | 2016-08-29 | 2017-02-15 | 上海掌门科技有限公司 | Method and system for synchronously updating information |
Non-Patent Citations (1)
| Title |
|---|
| Y.勒帕热等: "《UNIX***管理员大全》", 31 May 2000 * |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109614440A (en) * | 2018-10-25 | 2019-04-12 | 深圳壹账通智能科技有限公司 | Method of data synchronization and relevant device based on big data |
| CN109785129A (en) * | 2018-12-14 | 2019-05-21 | 平安证券股份有限公司 | Data capture method, device, equipment and computer readable storage medium |
| CN109785129B (en) * | 2018-12-14 | 2024-05-28 | 平安证券股份有限公司 | Data acquisition method, device, equipment and computer readable storage medium |
| CN111475309A (en) * | 2019-01-24 | 2020-07-31 | 北京京东尚科信息技术有限公司 | Data processing method, device, block chain service system and storage medium |
| CN110083579A (en) * | 2019-03-21 | 2019-08-02 | 深圳壹账通智能科技有限公司 | Incremental data synchronous method, apparatus, computer equipment and computer storage medium |
| CN111309747A (en) * | 2020-02-18 | 2020-06-19 | 京东数字科技控股有限公司 | Data synchronization method, system and device |
| CN111460038A (en) * | 2020-04-07 | 2020-07-28 | 中国建设银行股份有限公司 | Quasi-real-time data synchronization method and device |
| CN113010609B (en) * | 2020-12-23 | 2023-05-16 | 上海海鼎信息工程股份有限公司 | Differentiated synchronization method and system applied to store operation |
| CN113010609A (en) * | 2020-12-23 | 2021-06-22 | 上海海鼎信息工程股份有限公司 | Differentiated synchronization method and system applied to store operation |
| CN112925841A (en) * | 2021-03-26 | 2021-06-08 | 瀚高基础软件股份有限公司 | Distributed JDBC implementation method, device and computer-readable storage medium |
| CN114374668A (en) * | 2021-12-30 | 2022-04-19 | 北京北信源软件股份有限公司 | Information processing method, information processing device, electronic equipment and storage medium |
| CN114500645A (en) * | 2021-12-30 | 2022-05-13 | 兰州乐智教育科技有限责任公司 | Data acquisition system and data acquisition method |
| CN114374668B (en) * | 2021-12-30 | 2024-07-23 | 北京北信源软件股份有限公司 | Information processing method, information processing device, electronic equipment and storage medium |
| CN114756628A (en) * | 2022-06-16 | 2022-07-15 | 天津七一二移动通信有限公司 | High-multiplexing extensible data and file synchronization method |
| CN114756628B (en) * | 2022-06-16 | 2022-10-28 | 天津七一二移动通信有限公司 | High-multiplexing extensible data and file synchronization method |
| CN116737678A (en) * | 2023-08-14 | 2023-09-12 | 浙江同信企业征信服务有限公司 | Data synchronization method, device, equipment and storage medium |
| CN116737678B (en) * | 2023-08-14 | 2024-04-19 | 浙江同信企业征信服务有限公司 | Data synchronization method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108628881A (en) | Method of data synchronization and device | |
| US11019128B2 (en) | Identity service management in limited connectivity environments | |
| CN107924411B (en) | Method and system for recovering UI state in transaction system | |
| US9460405B2 (en) | Systems and methods for cloud data loss prevention integration | |
| JP2022529967A (en) | Extracting data from the blockchain network | |
| US9354998B2 (en) | Automated conformance and interoperability test lab | |
| US9460307B2 (en) | Managing sensitive data in cloud computing environments | |
| US8732258B2 (en) | Method and system for transporting telemetry data across a network | |
| Calcote et al. | Istio: Up and running: Using a service mesh to connect, secure, control, and observe | |
| CN111736775A (en) | Multi-source storage method, device, computer system and storage medium | |
| EP3436935A1 (en) | Pre-formed instructions for a mobile cloud service | |
| CN111078745A (en) | Data uplink method and device based on block chain technology | |
| CA2843284C (en) | Computer system, computer-implemented method and computer program product for sequencing incoming messages for processing at an application | |
| CN109614439A (en) | Method of data synchronization, device, electronic equipment and storage medium | |
| CN111915301B (en) | Data processing method and device based on block chain, electronic equipment and readable medium | |
| US20200118120A1 (en) | Methods and systems for providing a customized network | |
| DE602005005435T2 (en) | System and method for communication management of component applications | |
| CN112883011A (en) | Real-time data processing method and device | |
| US9128886B2 (en) | Computer implemented method, computer system, electronic interface, mobile computing device and computer readable medium | |
| US9104742B1 (en) | Method and system for optimized parallel queuing | |
| CN112689020B (en) | Message transmission method, message middleware, electronic equipment and storage medium | |
| CN112825525B (en) | Method and apparatus for processing transactions | |
| CN108512889A (en) | A kind of application response method for pushing and proxy server based on HTTP | |
| US20240152504A1 (en) | Data interaction method, apparatus, and electronic device | |
| CN110324373A (en) | Sharing files method and device, file syn chronizing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181009 |
|
| RJ01 | Rejection of invention patent application after publication |