CN110334147A - A kind of method of data synchronization and device - Google Patents
A kind of method of data synchronization and device Download PDFInfo
- Publication number
- CN110334147A CN110334147A CN201910417898.5A CN201910417898A CN110334147A CN 110334147 A CN110334147 A CN 110334147A CN 201910417898 A CN201910417898 A CN 201910417898A CN 110334147 A CN110334147 A CN 110334147A
- Authority
- CN
- China
- Prior art keywords
- data
- declaration form
- database
- synchronized
- form data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 230000001360 synchronised effect Effects 0.000 claims abstract description 133
- 230000009466 transformation Effects 0.000 claims abstract description 39
- 238000013135 deep learning Methods 0.000 claims abstract description 34
- 238000012549 training Methods 0.000 claims description 36
- 230000015654 memory Effects 0.000 claims description 20
- 230000004048 modification Effects 0.000 claims description 19
- 238000012986 modification Methods 0.000 claims description 19
- 238000006243 chemical reaction Methods 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 11
- 230000005540 biological transmission Effects 0.000 claims description 10
- 238000012544 monitoring process Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 238000010801 machine learning Methods 0.000 claims description 7
- 108010001267 Protein Subunits Proteins 0.000 claims description 4
- 238000001514 detection method Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000032696 parturition Effects 0.000 description 1
- 230000001172 regenerating effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Marketing (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Finance (AREA)
- Accounting & Taxation (AREA)
- General Engineering & Computer Science (AREA)
- Development Economics (AREA)
- Technology Law (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a kind of method of data synchronization and devices, and the present invention relates to big data technical field, method includes: to monitor the declaration form data write information of first database, wherein first database is the storing data library of declaration form data；After listening to any declaration form data write-in successful information, generates a corresponding data synchronous task and simultaneously store to preset task list；According to the default frequency, the data synchronous task in preset task list is pulled；Declaration form data to be synchronized in first database are read based on data synchronous task；The declaration form data to be synchronized read are inputted to deep learning transformation model, obtain data to be imported；Data to be imported are synchronized in the second database based on Media Gateway Control Protocol.Technical solution provided in an embodiment of the present invention is able to solve the problem that data synchronization efficiency is low between service databases different in the prior art.
Description
[technical field]
The present invention relates to big data technical field more particularly to a kind of method of data synchronization and device.
[background technique]
In the declaration form generation system under micro services framework, existing insurance policy data can be stored after generating to single library out
In (the first micro services), different data are handled respectively from two different servers for correcting library due to going out single library, number
It to be split according to service according to source, therefore, it is necessary to the declaration form data Fast synchronization that will be increased newly backups, so that correcting in library
Declaration form data being capable of basis during follow-up service as declaration form data modification.Currently, different under micro services framework
Cannot be synchronous by direct-connected realization data between service database, cause data synchronization efficiency low.
[summary of the invention]
In view of this, the embodiment of the invention provides a kind of method of data synchronization, device, storage medium and computers to set
It is standby, to solve cannot be same by direct-connected realization data between service database different under micro services framework in the prior art
Step, leads to the problem that data synchronization efficiency is low.
To achieve the goals above, according to an aspect of the invention, there is provided a kind of method of data synchronization, the method
It include: the declaration form data write information for monitoring first database, wherein the first database is the storage of the declaration form data
Database；After listening to any declaration form data write-in successful information, a corresponding data synchronous task is generated simultaneously
It stores to preset task list；According to the default frequency, the data synchronous task in the preset task list is pulled；
Synchronous task reads declaration form data to be synchronized in the first database based on the data；It is defeated to deep learning transformation model
Enter the declaration form data to be synchronized read, so that the deep learning transformation model is to the declaration form number to be synchronized
According to type of data structure converted, obtain data to be imported, wherein the data to be imported be the list of SQL character string；Base
The data to be imported are synchronized in the second database in Media Gateway Control Protocol.
Further, it is described to the declaration form data to be synchronized that read of deep learning transformation model input it
Before, the method also includes: obtain the input sample data and described second that there is same data structure with the declaration form data
The template data of database；It is trained respectively using the input sample data and the template data as input training set and output
Collection, and analyzing and training is carried out to initial machine learning model based on the input training set and the output training set, obtain institute
State deep learning transformation model, wherein the deep learning transformation model can be according to conversion logic by the input sample number
According to being converted to the template data.
Further, the data to be imported are synchronized in the second database based on Media Gateway Control Protocol described
Later, the method also includes: when the declaration form data in second database are modified, based on declaration form data modification believe
Breath generates real-time informing；The real-time informing is sent to control data corporation, so that the control data corporation is to default people
Member's account sends the declaration form data modification information indicated in the real-time informing.
Further, the synchronous task based on the data reads declaration form number to be synchronized in the first database
According to, comprising: synchronous task writes synchronization request based on the data；The synchronization request is sent to the first database,
Wherein, the first database receives and parses through the synchronization request, and verifies the synchronization request according to preset allocation list；
When being verified, traverses the declaration form data respective field and splice generation sync message；The sync message is confirmed as
The declaration form data to be synchronized.
Further, the data to be imported are synchronized in the second database based on Media Gateway Control Protocol described
Later, the method also includes: be updated according to data directory file of the data to be imported to second database.
To achieve the goals above, according to an aspect of the invention, there is provided a kind of data synchronization unit, described device
It include: monitoring unit, for monitoring the declaration form data write information of first database, wherein the first database is described
The storing data library of declaration form data；Generation unit, for giving birth to after listening to any declaration form data write-in successful information
At a corresponding data synchronous task and store to preset task list；Unit is pulled, for pulling according to the frequency is preset
The data synchronous task in the preset task list；Reading unit is read for synchronous task based on the data
Declaration form data to be synchronized in the first database；Processing unit, for what is read to the input of deep learning transformation model
The declaration form data to be synchronized, so that data knot of the deep learning transformation model to the declaration form data to be synchronized
Structure type is converted, and data to be imported are obtained, wherein the data to be imported are the list of SQL character string；Synchronization unit is used
The data to be imported are synchronized in the second database in being based on Media Gateway Control Protocol.
Further, described device further include: acquiring unit has identical data knot with the declaration form data for obtaining
The template data of the input sample data of structure and second database；Training unit, for respectively by the input sample number
It is trained according to the template data as input training set and output training set, and based on the input training set and the output
Collection carries out analyzing and training to initial machine learning model, obtains the deep learning transformation model, wherein the deep learning turns
The input sample data can be converted to the template data according to conversion logic by mold changing type.
Further, the reading unit includes: to write subelement, writes synchronization for synchronous task based on the data
Request；Transmission sub-unit, for the synchronization request to be sent to the first database, wherein the first database connects
The synchronization request is received and parsed, and the synchronization request is verified according to preset allocation list；Subelement is generated, for when verifying
By when, traverse the declaration form data respective field and splice generation sync message；Subelement is confirmed, for reporting the synchronization
Text is confirmed as the declaration form data to be synchronized.
To achieve the goals above, according to an aspect of the invention, there is provided a kind of computer non-volatile memories are situated between
Matter, the storage medium include the program of storage, control equipment where the storage medium in described program operation and execute
The method of data synchronization stated.
To achieve the goals above, according to an aspect of the invention, there is provided a kind of computer equipment, including storage
Device, processor and storage in the memory and the computer program that can run on the processor, the computer
Program realizes the step of above-mentioned method of data synchronization when being executed by the processor.
Batch conversion is being carried out by deep learning transformation model in the present solution, multiple data synchronous tasks may be implemented,
Declaration form data are quickly converted for data type needed for the second database, then batch synchronization to the second database
In, the synchronous warehouse-in efficiency of data can be effectively improved.
[Detailed description of the invention]
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below will be to needed in the embodiment attached
Figure is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the invention, for this field
For those of ordinary skill, without any creative labor, it can also be obtained according to these attached drawings other attached
Figure.
Fig. 1 is a kind of flow chart of method of data synchronization according to an embodiment of the present invention；
Fig. 2 is a kind of schematic diagram of data synchronization unit according to an embodiment of the present invention；
Fig. 3 is a kind of schematic diagram of computer equipment according to an embodiment of the present invention.
[specific embodiment]
For a better understanding of the technical solution of the present invention, being retouched in detail to the embodiment of the present invention with reference to the accompanying drawing
It states.
It will be appreciated that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.Base
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts it is all its
Its embodiment, shall fall within the protection scope of the present invention.
The term used in embodiments of the present invention is only to be not intended to be limiting merely for for the purpose of describing particular embodiments
The present invention.In the embodiment of the present invention and the "an" of singular used in the attached claims, " described " and "the"
It is also intended to including most forms, unless the context clearly indicates other meaning.
It should be appreciated that term "and/or" used herein is only a kind of incidence relation for describing affiliated partner, indicate
There may be three kinds of relationships, for example, A and/or B, can indicate: individualism A, exist simultaneously A and B, individualism B these three
Situation.In addition, character "/" herein, typicallys represent the relationship that forward-backward correlation object is a kind of "or".
It will be appreciated that though terminal may be described using term first, second, third, etc. in embodiments of the present invention,
But these terminals should not necessarily be limited by these terms.These terms are only used to for terminal being distinguished from each other out.For example, not departing from the present invention
In the case where scope of embodiments, first terminal can also be referred to as second terminal, and similarly, second terminal can also be referred to as
One terminal.
Depending on context, word as used in this " if " can be construed to " ... when " or " when ...
When " or " in response to determination " or " in response to detection ".Similarly, depend on context, phrase " if it is determined that " or " if detection
(condition or event of statement) " can be construed to " when determining " or " in response to determination " or " when the detection (condition of statement
Or event) when " or " in response to detection (condition or event of statement) ".
Fig. 1 is a kind of flow chart of method of data synchronization according to an embodiment of the present invention, as shown in Figure 1, this method comprises:
Step S101 monitors the declaration form data write information of first database, wherein first database is declaration form data
Storing data library；
It is synchronous to generate a corresponding data after listening to any declaration form data write-in successful information by step S102
Task is simultaneously stored to preset task list；
Step S103 pulls the data synchronous task in preset task list according to the default frequency；
Step S104 reads declaration form data to be synchronized in first database based on data synchronous task；
Step S105 inputs the declaration form data to be synchronized read to deep learning transformation model, so that depth
It practises transformation model to convert the type of data structure of declaration form data to be synchronized, obtains data to be imported, wherein to be imported
Data are the list of SQL character string；
Data to be imported are synchronized in the second database by step S106 based on Media Gateway Control Protocol.
It is to be appreciated that each server is independently disposed under micro services frame, and there are oneself independent data
Library and be relevant database, such as: declaration form generates server, Claims Resolution processing server, declaration form correct server etc..Therefore,
When declaration form generates server one declaration form data of generation, data pass through preset interface and are stored in (i.e. first in a corresponding database
Database).At this time, the corresponding database of processing server of settling a claim cannot be synchronized directly and obtain newly-generated declaration form number
According to.Therefore an asynchronous and synchronous component is needed to realize that the data between each database are synchronous, so that the declaration form service of correcting
Device can read the declaration form data of the backup in corresponding database (i.e. the second database), using the basis as modification.
Batch conversion is being carried out by deep learning transformation model in the present solution, multiple data synchronous tasks may be implemented,
Declaration form data are quickly converted for data type needed for the second database, then batch synchronization to the second database
In, the synchronous warehouse-in efficiency of data can be effectively improved.
In one embodiment, the declaration form data that declaration form generates that server generates are written to first database, and first
Database will feed back write-in successful information and generate server to declaration form, when listening to the successful information of write-in, be located at first
Asynchronous and synchronous component between database and the second database generates data synchronous task.Specifically, data synchronous task is deposited
Storage is in preset task list.Optionally, the default frequency can be the time frequency or count the frequency, for example, every five seconds, asynchronous
Synchronization Component can go in task list to pull a data synchronous task, at this point, data synchronous task may be one or more.
When the default frequency is to count the frequency, such as when the data synchronous task in task list reaches 5, it is same to trigger a data
Step.
Optionally, declaration form data to be synchronized in first database are read based on data synchronous task, comprising: be based on data
Synchronous task writes synchronization request；Synchronization request is sent to first database, wherein first database receives and parses through synchronization
Request, and synchronization request is verified according to preset allocation list；When being verified, declaration form data respective field to be synchronized is traversed
And splice generation sync message；Sync message is confirmed as declaration form data to be synchronized.
Wherein, synchronization request carries the unique identification of declaration form data, such as number of policy.First database is synchronized in parsing and is asked
When asking, analytic parameter can be obtained, analytic parameter includes the unique identification of declaration form data to be synchronized.According to preset allocation list
Synchronization request is verified, can be whether there is according to preset allocation list inquiry declaration form tables of data to be synchronized, if it does, i.e.
Expression is verified.Further, the SQL format string in synchronization request is disassembled, with the corresponding word of SQL format string traversal declaration form data
Section, splicing generate sync message.
When the data synchronous task in task list has multiple, multiple data synchronous tasks can be write to one together
In step request, greatly reduce the quantity of synchronization request, reduce the time in network transmission, improves synchronous efficiency.
Optionally, declaration form data are txt format, excel format, any one in xml format.
Optionally, before the declaration form data to be synchronized read to the input input of deep learning transformation model, method
Further include: obtain the template data of the input sample data and the second database that there is same data structure with declaration form data；Point
Not using input sample data and template data as input training set and output training set, and based on input training set and output instruction
Practice collection and analyzing and training is carried out to initial machine learning model, obtains deep learning transformation model, wherein deep learning transformation model
Input sample data can be converted into template data according to conversion logic.
Wherein, the data format of the second database is the list of SQL character string.Such as: the data format of input sample data
For excel format, the field that transformation model passes through in training study input sample data is corresponding with the field in template data
Relationship parses the association attributes of field；Then according to the corresponding relationship of field and association attributes, transformation rule is determined；Using true
The value of field respective column in sample data is converted to field in template data according to the corresponding relationship of field by fixed transformation rule
The value of respective column, then the data being converted to are inserted into SQL statement, to realize data conversion.The wherein attribute packet of field
Include field name, field type, field length etc..
When the field type of date field in sample data is char, and the field of date field is corresponded in the second database
Type is date, this is the field of two different types of data, then transformation rule are as follows: by the field respective column in sample data
The field type of data is converted to date by char.In this way, the date data in sample data can just be imported into the second number
According to library.
Optionally, the step being synchronized to data to be imported based on Media Gateway Control Protocol in the second database, comprising:
Data to be imported are sent to the second database, wherein the second database disassembles the same step number of data acquisition to be imported
According to；The synchrodata that will acquire is written into the corresponding data table of the second database.Wherein, data to be imported are sql character string
List.It is to be appreciated that the data in the second database are facilitated the inquiry of data and are repaired with the storage of sql character string tabular form
Change.
Optionally, after data to be imported are synchronized in the second database based on Media Gateway Control Protocol, method
Further include: when the declaration form data in the second database are modified, real-time informing is generated based on declaration form data modification information；It will
Real-time informing is sent to control data corporation, so that control data corporation was indicated into default personnel's account transmission real-time informing
Declaration form data modification information.Such as: the declaration form data (beneficiary for being 123456 when the declaration form odd numbers listened in the second database
Title change) when modifying, real-time notification message will be generated.
It is to be appreciated that when in the second database data generate modification when, can the declaration form data based on modification it is timely
Notify contact staff, sales force or relevant person in charge.
Optionally, after data to be imported are synchronized in the second database based on Media Gateway Control Protocol, method
Further include: it is updated according to data directory file of the data to be imported to the second database.
Optionally, after data to be imported being synchronized in the second database based on Media Gateway Control Protocol, method is also
It include: the feedback information of error in data or synchronization failure that the second database is monitored in synchronizing process；Parsing feedback information obtains
To the unique identification of the declaration form data of synchronization failure；Data synchronous task is regenerated according to unique identification and is stored to preset
Task list.It is to be appreciated that can regenerate a synchronous task if synchronization failure for the declaration form to fail and go to hold
Row.
The embodiment of the invention provides a kind of data synchronization unit, the data synchronization unit is synchronous for executing above-mentioned data
Method, as shown in Fig. 2, the device includes: monitoring unit 10, generation unit 20, pulls unit 30, reading unit 40, processing list
First 50, synchronization unit 60.
Monitoring unit 10, for monitoring the declaration form data write information of first database, wherein first database is declaration form
The storing data library of data；
Generation unit 20, for generating a corresponding number after listening to any declaration form data write-in successful information
According to synchronous task and store to preset task list；
Unit 30 is pulled, for pulling the data synchronous task in preset task list according to the frequency is preset；
Reading unit 40, for reading declaration form data to be synchronized in first database based on data synchronous task；
Processing unit 50, for inputting the declaration form data to be synchronized read to deep learning transformation model, so that
Deep learning transformation model converts the type of data structure of declaration form data to be synchronized, obtains data to be imported, wherein
Data to be imported are the list of SQL character string；
Synchronization unit 60, for data to be imported to be synchronized in the second database based on Media Gateway Control Protocol.
It is to be appreciated that each server is independently disposed under micro services frame, and there are oneself independent data
Library and be relevant database, such as: declaration form generates server, Claims Resolution processing server, declaration form correct server etc..Therefore,
When declaration form generates server one declaration form data of generation, data pass through preset interface and are stored in (i.e. first in a corresponding database
Database).At this time, the corresponding database of processing server of settling a claim cannot be synchronized directly and obtain newly-generated declaration form number
According to.Therefore an asynchronous and synchronous component is needed to realize that the data between each database are synchronous, so that the declaration form service of correcting
Device can read the declaration form data of the backup in corresponding database (i.e. the second database), using the basis as modification.
Batch conversion is being carried out by deep learning transformation model in the present solution, multiple data synchronous tasks may be implemented,
Declaration form data are quickly converted for data type needed for the second database, then batch synchronization to the second database
In, the synchronous warehouse-in efficiency of data can be effectively improved.
In one embodiment, the declaration form data that declaration form generates that server generates are written to first database, and first
Database will feed back write-in successful information, when listening to the successful information of write-in, be located at first database and the second data
Asynchronous and synchronous component between library generates data synchronous task.Specifically, data synchronous task is stored in preset task column
In table.Optionally, the default frequency can be the time frequency or count the frequency, and for example, every five seconds, asynchronous and synchronous component can go task
A data synchronous task is pulled in list, at this point, data synchronous task may be one or more.When the default frequency is to count
The frequency, such as when the data synchronous task in task list reaches 5, it is synchronous to trigger a data.
Optionally, reading unit 40 includes: to write subelement, transmission sub-unit, generate subelement, confirmation subelement.
Subelement is write, for writing synchronization request based on data synchronous task；Transmission sub-unit is used for synchronization request
It is sent to first database, wherein first database receives and parses through synchronization request, and verifies and synchronize according to preset allocation list
Request；Subelement is generated, for when being verified, traversing declaration form data respective field to be synchronized and splicing the synchronous report of generation
Text；Confirm subelement, sync message is confirmed as declaration form data to be synchronized.
Wherein, synchronization request carries the unique identification of declaration form data, such as number of policy.First database is synchronized in parsing and is asked
When asking, analytic parameter can be obtained, analytic parameter includes the unique identification of declaration form data to be synchronized.According to preset allocation list
Synchronization request is verified, can be whether there is according to preset allocation list inquiry declaration form tables of data to be synchronized, if it does, i.e.
Expression is verified.Further, the SQL format string in synchronization request is disassembled, with the corresponding word of SQL format string traversal declaration form data
Section, splicing generate sync message.
When the data synchronous task in task list has multiple, multiple data synchronous tasks can be write to one together
In step request, greatly reduce the quantity of synchronization request, reduce the time in network transmission, improves synchronous efficiency.
Optionally, declaration form data are txt format, excel format, any one in xml format.
Optionally, device further includes acquiring unit, training unit.
Acquiring unit, for obtaining the input sample data and the second database that there is same data structure with declaration form data
Template data；Training unit, for being trained respectively using input sample data and template data as input training set and output
Collection, and analyzing and training is carried out to initial machine learning model based on input training set and output training set, it obtains deep learning and turns
Mold changing type, wherein input sample data can be converted to template data according to conversion logic by deep learning transformation model.
Wherein, the data format of the second database is the list of SQL character string.
Such as: the data format of input sample data is excel format, and transformation model passes through training study input sample number
The corresponding relationship of the field in field and template data in, parses the association attributes of field；Then according to the correspondence of field
Relationship and association attributes, determine transformation rule；It, will be in sample data according to the corresponding relationship of field using determining transformation rule
The value of field respective column is converted to the value of field respective column in template data, then the data being converted to are inserted into SQL statement,
To realize data conversion.Wherein the attribute of field includes field name, field type, field length etc..
When the field type of date field in sample data is char, and the field of date field is corresponded in the second database
Type is date, this is the field of two different types of data, then transformation rule are as follows: by the field respective column in sample data
The field type of data is converted to date by char.In this way, the date data in sample data can just be imported into the second number
According to library.
Optionally, synchronization unit 50 include: second transmit into subelement, write-in subelement.
Second transmission sub-unit, for data to be imported to be sent to the second database, wherein the second database dismantling to
Import data acquisition synchrodata；Subelement is written, the synchrodata for will acquire is written to the correspondence number of the second database
According in table.Wherein, data to be imported are the list of sql character string.It is to be appreciated that the data in the second database are with sql character
The storage of tandem sheet form, facilitates the inquiry and modification of data.
Optionally, device further includes the second generation unit, the second transmission unit.
Second generation unit, for being based on declaration form data modification when the declaration form data in the second database are modified
Information generates real-time informing；Second transmission unit, for real-time informing to be sent to control data corporation, so that in data management
The heart sends the declaration form data modification information indicated in real-time informing to default personnel's account.Such as: when listening to the second database
In declaration form odd numbers be 123456 declaration form data (change of beneficiary's title) when modifying, real-time informing will be generated and disappeared
Breath.
It is to be appreciated that after the data of the second database synchronize, when the data in the second database generate modification, energy
Enough contents based on modification notify contact staff, sales force or relevant person in charge in time.
Optionally, device further includes updating unit, for the data directory text according to data to be imported to the second database
Part is updated.It is to be appreciated that passing through the importing state real-time update data directory file of data, user can be facilitated to look into
When asking or modifying the declaration form data in the second database, it can be inquired according to index file, quick obtaining declaration form.
Optionally, device further includes the second monitoring unit, resolution unit, third generation unit.
Second monitoring unit monitors the error in data or synchronization failure of the second database in synchronizing process
Feedback information；Resolution unit, the unique identification for obtaining the declaration form data of synchronization failure for parsing feedback information；Third generates single
Member, for regenerating data synchronous task according to unique identification and storing to preset task list.It is to be appreciated that if
Synchronization failure goes to execute then can regenerate a synchronous task for the declaration form to fail.
The embodiment of the invention provides a kind of computer non-volatile memory medium, storage medium includes the program of storage,
Wherein, when program is run, equipment where control storage medium executes following steps:
Monitor the declaration form data write information of first database, wherein first database is the storing data of declaration form data
Library；After listening to any declaration form data write-in successful information, generates a corresponding data synchronous task and simultaneously store to pre-
If task list；According to the default frequency, the data synchronous task in preset task list is pulled；Based on data synchronous task
Read declaration form data to be synchronized in first database；The declaration form number to be synchronized read is inputted to deep learning transformation model
According to obtaining to be imported so that deep learning transformation model converts the type of data structure of declaration form data to be synchronized
Data, wherein data to be imported are the list of SQL character string；Data to be imported are synchronized to based on Media Gateway Control Protocol
In two databases.
Optionally, when program is run, equipment where control storage medium also executes following steps: obtaining and declaration form data
The template data of input sample data and the second database with same data structure；Respectively by input sample data and template
Data are based on input training set and output training set to initial machine learning model as input training set and output training set
Analyzing and training is carried out, deep learning transformation model is obtained, wherein deep learning transformation model can will be inputted according to conversion logic
Sample data is converted to template data.
Optionally, when program is run, equipment where control storage medium also executes following steps: when in the second database
Declaration form data when modifying, real-time informing is generated based on declaration form data modification information；Real-time informing is sent to data pipe
Reason center, so that control data corporation sends the declaration form data modification information indicated in real-time informing to default personnel's account.
Optionally, when program is run, equipment where control storage medium also executes following steps: based on data synchronous
Synchronization request is write in business；Synchronization request is sent to first database, wherein first database receives and parses through synchronization request,
And synchronization request is verified according to preset allocation list；When being verified, traverses declaration form data respective field and splice generation together
Walk message；Sync message is confirmed as declaration form data to be synchronized.
Optionally, when program is run, equipment where control storage medium also executes following steps: according to data to be imported
The data directory file of second database is updated.
Fig. 3 is a kind of schematic diagram of computer equipment provided in an embodiment of the present invention.As shown in figure 3, the meter of the embodiment
Machine equipment 100 is calculated to include: processor 101, memory 102 and storage in the memory 102 and can run on processor 101
Computer program 103, the computer program 103 by processor 101 execute when realize embodiment in method of data synchronization, be
It avoids repeating, not repeat one by one herein.Alternatively, realizing that data are same in embodiment when the computer program is executed by processor 101
The function of each model/unit does not repeat one by one herein in step device to avoid repeating.
Computer equipment 100 can be the calculating such as desktop PC, notebook, palm PC and cloud server and set
It is standby.Computer equipment may include, but be not limited only to, processor 101, memory 102.It will be understood by those skilled in the art that Fig. 3
The only example of computer equipment 100 does not constitute the restriction to computer equipment 100, may include than illustrate it is more or
Less component perhaps combines certain components or different components, such as computer equipment can also be set including input and output
Standby, network access equipment, bus etc..
Alleged processor 101 can be central processing unit (Central Processing Unit, CPU), can also be
Other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor
Deng.
Memory 102 can be the internal storage unit of computer equipment 100, for example, computer equipment 100 hard disk or
Memory.What memory 102 was also possible to be equipped on the External memory equipment of computer equipment 100, such as computer equipment 100 inserts
Connect formula hard disk, intelligent memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash memory
Block (Flash Card) etc..Further, memory 102 can also both including computer equipment 100 internal storage unit or
Including External memory equipment.Memory 102 is for storing other program sum numbers needed for computer program and computer equipment
According to.Memory 102 can be also used for temporarily storing the data that has exported or will export.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In several embodiments provided by the present invention, it should be understood that disclosed system, device and method can be with
It realizes by another way.For example, the apparatus embodiments described above are merely exemplary, for example, the division of unit,
Only a kind of logical function partition, there may be another division manner in actual implementation, for example, multiple units or components can be with
In conjunction with or be desirably integrated into another system, or some features can be ignored or not executed.Another point, it is shown or discussed
Mutual coupling, direct-coupling or communication connection can be through some interfaces, the INDIRECT COUPLING of device or unit or
Communication connection can be electrical property, mechanical or other forms.
Unit may or may not be physically separated as illustrated by the separation member, shown as a unit
Component may or may not be physical unit, it can and it is in one place, or may be distributed over multiple networks
On unit.It can some or all of the units may be selected to achieve the purpose of the solution of this embodiment according to the actual needs.
It, can also be in addition, the functional units in various embodiments of the present invention may be integrated into one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of hardware adds SFU software functional unit.
The above-mentioned integrated unit being realized in the form of SFU software functional unit can store and computer-readable deposit at one
In storage media.Above-mentioned SFU software functional unit is stored in a storage medium, including some instructions are used so that a computer
It is each that device (can be personal computer, server or network equipment etc.) or processor (Processor) execute the present invention
The part steps of embodiment method.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory (Read-Only
Memory, ROM), random access memory (Random Access Memory, RAM), magnetic or disk etc. is various to deposit
Store up the medium of program code.
The above is merely preferred embodiments of the present invention, be not intended to limit the invention, it is all in spirit of the invention and
Within principle, any modification, equivalent substitution, improvement and etc. done be should be included within the scope of the present invention.
Claims (10)
1. a kind of method of data synchronization, which is characterized in that the described method includes:
Monitor the declaration form data write information of first database, wherein the first database is the storage of the declaration form data
Database；
After listening to any declaration form data write-in successful information, generates a corresponding data synchronous task and simultaneously store
To preset task list；
According to the default frequency, the data synchronous task in the preset task list is pulled；
Synchronous task reads declaration form data to be synchronized in the first database based on the data；
The declaration form data to be synchronized read are inputted to deep learning transformation model, so that the deep learning is converted
Model converts the type of data structure of the declaration form data to be synchronized, obtains data to be imported, wherein described wait lead
Entering data is the list of SQL character string；
The data to be imported are synchronized in the second database based on Media Gateway Control Protocol.
2. the method according to claim 1, wherein being read described to the input of deep learning transformation model
Before the declaration form data to be synchronized, the method also includes:
Obtain the template number of the input sample data and second database that there is same data structure with the declaration form data
According to；
Respectively using the input sample data and the template data as input training set and output training set, and based on described
It inputs training set and the output training set and analyzing and training is carried out to initial machine learning model, obtain the deep learning conversion
Model, wherein the input sample data can be converted to the mould according to conversion logic by the deep learning transformation model
Plate data.
3. described in any item methods according to claim 1~2, which is characterized in that be based on Media Gateway Control Protocol described
After the data to be imported are synchronized in the second database, the method also includes:
When the declaration form data in second database are modified, real-time informing is generated based on declaration form data modification information；
The real-time informing is sent to control data corporation, so that the control data corporation sends institute to default personnel's account
State the declaration form data modification information indicated in real-time informing.
4. described in any item methods according to claim 1~2, which is characterized in that the synchronous task based on the data is read
Take declaration form data to be synchronized in the first database, comprising:
Synchronous task writes synchronization request based on the data；
The synchronization request is sent to the first database, wherein the first database receives and parses through the synchronization
Request, and the synchronization request is verified according to preset allocation list；
When being verified, traverses the declaration form data respective field and splice generation sync message；
The sync message is confirmed as the declaration form data to be synchronized.
5. the method according to claim 1, wherein being based on Media Gateway Control Protocol for described wait lead described
Enter after data are synchronized in the second database, the method also includes:
It is updated according to data directory file of the data to be imported to second database.
6. a kind of data synchronization unit, which is characterized in that described device includes:
Monitoring unit, for monitoring the declaration form data write information of first database, wherein the first database is the guarantor
The storing data library of forms data；
Generation unit, for generating a corresponding data after listening to any declaration form data write-in successful information
Synchronous task is simultaneously stored to preset task list；
Unit is pulled, for pulling the data synchronous task in the preset task list according to the frequency is preset；
Reading unit reads declaration form data to be synchronized in the first database for synchronous task based on the data；
Processing unit, for inputting the declaration form data to be synchronized read to deep learning transformation model, so that institute
It states deep learning transformation model to convert the type of data structure of the declaration form data to be synchronized, obtains number to be imported
According to, wherein the data to be imported are the list of SQL character string；
Synchronization unit, for the data to be imported to be synchronized in the second database based on Media Gateway Control Protocol.
7. device according to claim 6, which is characterized in that described device further include:
Acquiring unit, for obtaining the input sample data that there is same data structure with the declaration form data and second number
According to the template data in library；
Training unit, for being trained respectively using the input sample data and the template data as input training set and output
Collection, and analyzing and training is carried out to initial machine learning model based on the input training set and the output training set, obtain institute
State deep learning transformation model, wherein the deep learning transformation model can be according to conversion logic by the input sample number
According to being converted to the template data.
8. according to the described in any item devices of claim 6~7, which is characterized in that the reading unit includes:
Subelement is write, writes synchronization request for synchronous task based on the data；
Transmission sub-unit, for the synchronization request to be sent to the first database, wherein the first database receives
And the synchronization request is parsed, and the synchronization request is verified according to preset allocation list；
Subelement is generated, for when being verified, traversing the declaration form data respective field and splicing generation sync message；
Subelement is confirmed, for the sync message to be confirmed as the declaration form data to be synchronized.
9. a kind of computer non-volatile memory medium, the storage medium includes the program of storage, which is characterized in that described
Equipment perform claim requires method of data synchronization described in 1 to 5 any one program controls the storage medium when running where.
10. a kind of computer equipment, including memory, processor and storage are in the memory and can be in the processor
The computer program of upper operation, which is characterized in that when the computer program is executed by the processor realize claim 1 to
The step of method of data synchronization described in 5 any one.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910417898.5A CN110334147A (en) | 2019-05-20 | 2019-05-20 | A kind of method of data synchronization and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910417898.5A CN110334147A (en) | 2019-05-20 | 2019-05-20 | A kind of method of data synchronization and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110334147A true CN110334147A (en) | 2019-10-15 |
Family
ID=68139460
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910417898.5A Pending CN110334147A (en) | 2019-05-20 | 2019-05-20 | A kind of method of data synchronization and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110334147A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110866217A (en) * | 2019-10-24 | 2020-03-06 | 长城计算机软件与***有限公司 | Cross report recognition method and device, storage medium and electronic equipment |
| CN112306994A (en) * | 2020-11-10 | 2021-02-02 | 北京沃东天骏信息技术有限公司 | Database data migration method and device and storage medium |
| CN112817990A (en) * | 2021-01-28 | 2021-05-18 | 北京百度网讯科技有限公司 | Data processing method and device, electronic equipment and readable storage medium |
| CN116150280A (en) * | 2023-04-04 | 2023-05-23 | 之江实验室 | Mimicry redis database synchronization method, system, equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108920670A (en) * | 2018-07-06 | 2018-11-30 | 深圳市小牛在线互联网信息咨询有限公司 | Cache synchronization method, device, system and storage medium |
| CN109377383A (en) * | 2018-08-31 | 2019-02-22 | 平安科技（深圳）有限公司 | Product data synchronous method, device, computer equipment and storage medium |
-
2019
- 2019-05-20 CN CN201910417898.5A patent/CN110334147A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108920670A (en) * | 2018-07-06 | 2018-11-30 | 深圳市小牛在线互联网信息咨询有限公司 | Cache synchronization method, device, system and storage medium |
| CN109377383A (en) * | 2018-08-31 | 2019-02-22 | 平安科技（深圳）有限公司 | Product data synchronous method, device, computer equipment and storage medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110866217A (en) * | 2019-10-24 | 2020-03-06 | 长城计算机软件与***有限公司 | Cross report recognition method and device, storage medium and electronic equipment |
| CN112306994A (en) * | 2020-11-10 | 2021-02-02 | 北京沃东天骏信息技术有限公司 | Database data migration method and device and storage medium |
| CN112817990A (en) * | 2021-01-28 | 2021-05-18 | 北京百度网讯科技有限公司 | Data processing method and device, electronic equipment and readable storage medium |
| CN112817990B (en) * | 2021-01-28 | 2024-03-08 | 北京百度网讯科技有限公司 | Data processing method, device, electronic equipment and readable storage medium |
| CN116150280A (en) * | 2023-04-04 | 2023-05-23 | 之江实验室 | Mimicry redis database synchronization method, system, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110334147A (en) | A kind of method of data synchronization and device | |
| CN110032575A (en) | Data query method, apparatus, equipment and storage medium | |
| CN109344170B (en) | Stream data processing method, system, electronic device and readable storage medium | |
| CN104246767A (en) | Telemetry system for a cloud synchronization system | |
| CN104156832A (en) | Intersystem data verification method and device | |
| US20140289243A1 (en) | Method and system for centralized issue tracking | |
| CN112039701B (en) | Interface call monitoring method, device, equipment and storage medium | |
| CN103927314B (en) | A kind of method and apparatus of batch data processing | |
| US9998450B2 (en) | Automatically generating certification documents | |
| CN110222028A (en) | A kind of data managing method, device, equipment and storage medium | |
| US11913811B2 (en) | Enhanced meter management solution | |
| CN113505520A (en) | Method, device and system for supporting heterogeneous federated learning | |
| CN113282611A (en) | Method and device for synchronizing stream data, computer equipment and storage medium | |
| CN110781197A (en) | Hive offline synchronous verification method and device and electronic equipment | |
| CN105335466A (en) | Audio data retrieval method and apparatus | |
| WO2024103898A1 (en) | Database cluster management method and apparatus | |
| CN110825538A (en) | MQ-based method for dynamically interacting data by self-defining data types | |
| CN111078764A (en) | Data processing method and device, computer readable storage medium and electronic equipment | |
| CN114756301A (en) | Log processing method, device and system | |
| Rukkas et al. | Distributed datastores: Towards probabilistic approach for estimation of reliability | |
| CN113268553A (en) | Data auditing method, system, electronic equipment and storage medium | |
| CN113051135A (en) | Application thread pool monitoring method and thread pool monitoring device | |
| TW201610907A (en) | Methodology for synchronizing heterogeneous data with telecom orders | |
| CN116915870A (en) | Task creation request processing method, device, electronic equipment and readable medium | |
| CN118349607A (en) | Data synchronization method, system, device, equipment, storage medium and product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |