CN106776876A - Data migration method and data mover system - Google Patents
Data migration method and data mover system Download PDFInfo
- Publication number
- CN106776876A CN106776876A CN201611074474.6A CN201611074474A CN106776876A CN 106776876 A CN106776876 A CN 106776876A CN 201611074474 A CN201611074474 A CN 201611074474A CN 106776876 A CN106776876 A CN 106776876A
- Authority
- CN
- China
- Prior art keywords
- data
- migration
- source system
- system data
- goal systems
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a kind of data migration method and data mover system.Wherein, a kind of data migration method, including:Migrated to before goal systems by source system data, check that rule carries out validity checking to source system data according to default, source system data is processed according to inspection result;And source system data structure is carried out into control configuration with goal systems data structure according to configuration mode is compareed;When source system data is by after validity checking, source system data being migrated to goal systems;When source system data does not pass through validity checking, source system data is repaired and adjusted, until source system data passes through validity checking;The log information produced in record validity checking, control configuration and transition process.By technical scheme, logic is migrated using configurable mode processing data, Data Migration is worked becomes more flexible, easy-to-use, heterogeneous system Data Migration is become simpler, efficiently.
Description
Technical field
The present invention relates to a kind of Data Transference Technology field, in particular to a kind of data migration method, one is further related to
Plant data mover system.
Background technology
Enterprise resource planning (Enterprise Resource Planning, ERP) refers to set up in information technology
On the basis of, with the management thought of systematization, for business decision layer and employee provide the management platform that decision-making runs means.In information
Change all-round developing today, increasing enterprise have selected ERP system to manage each side resource and business of enterprise.
But it is so swift and violent in economic development, ecommerce enters development climax, existing under the background of Intranet surprise attack
Some ERP systems tend not to meet the demand of all business of client, and substantial amounts of enterprise all does on the basis of ERP system is existing
Different degrees of customization development, the thing followed is the demand that ERP system updates.And ERP system has its intrinsic
Complexity, update not a duck soup, as the development of big data technology is increasingly faster, data are that client's highest value is produced
Thing, so model change will certainly produce the demand to Data Migration, perhaps problem can't be too big for isomorphism system upgrade,
But it is related to the upgrading of heterogeneous system, just into one piece of worry of enterprise, either the user of ERP is still for Data Migration
The provider of ERP, heterogeneous system data liter is moved the capital to another place and will become not only headache but also a thing for active demand.
Correlation technique will do data structure control between legacy system and new system from code aspect, then by origin system
Data query does certain treatment in internal memory in internal memory, processes data into the requirement for meeting goal systems, preserves data
Storehouse.Although correlation technique completes the promotion of heterogeneous system data, there is following some deficiency:1st, before Data Migration
There is no source system data to check rule, it is impossible to be assessed before migration and avoided risk;2nd, origin system and goal systems data structure
Control can not accomplish flexible configuration, if run into field control need modification, can only by change source code solve run into ask
Topic;3rd, the data of project customization cannot be solved by existing capability configuration mode, can only be developed by customized project;4th, visitor
Family business rule cannot dynamic expansion in existing product, can only be developed by customized project and solved;5th, target is moved to
The later data of system do not have origin system to be processed with goal systems reconciliation, it is impossible to the report of automatically generated data verification of correctness, item
It is too big that mesh pays error in data potential risk.
Therefore, a kind of data migration method how is provided to solve the above problems, and is asked as technology urgently to be resolved hurrily at present
Topic
The content of the invention
It is contemplated that at least solving one of technical problem present in prior art or correlation technique.
Therefore, it is an object of the present invention to propose a kind of data migration method.
It is another object of the present invention to propose a kind of data mover system.
In view of this, the present invention proposes a kind of data migration method, including:Migrated to target system by source system data
Before system, check that rule carries out validity checking to source system data according to default, source system data is entered according to inspection result
Row treatment;And source system data structure is carried out into control configuration with goal systems data structure according to configuration mode is compareed;When
Source system data is by after validity checking, source system data being migrated to goal systems;When source system data is not by legal
Property check when, source system data is repaired and adjusted, until source system data pass through validity checking;Record legitimacy inspection
Look into, compare the log information produced in configuration and transition process.
Data migration method of the invention, moves by by each traffic table or business paper in origin system
Before moving to goal systems, validity checking is first carried out, only check that the data for passing through can just carry out follow-up Data Migration, such as
Fruit inspection does not pass through, and source system data is repaired and adjusted, and rechecking is then carried out again, until passing through validity checking
Untill, can to greatest extent find and avoid risk, so that target system after improving the stability of data migration process and migrating
The correctness of data of uniting;Compareed according to control configuration mode by by source system data structure and goal systems data structure
Configuration, realizes flexibly control configuration data structure, supports the Data Migration between various heterogeneous systems, supports customization development number
According to structural transfer, so as to reduce the development cost of data migration process, autgmentability, stability and the safety of migratory system are improved
Property;By recording the log information produced in validity checking, control configuration and data migration process, can in time check and move
The problem run into during shifting, analyzes illegal data and is repaired and adjustment so as to follow-up, and then the data after reparation are moved again
Move, be prevented effectively from Data Migration interruption, goal systems data after the further stability for improving data migration process and migration
Correctness.
In addition, according to the above-mentioned data migration method of the present invention, can also have following additional technical characteristic:
In the above-mentioned technical solutions, it is preferable that when source system data does not pass through validity checking, source system data is entered
Row is repaired and adjustment is specifically included:Generate illegal data report;Fault-tolerant checking is carried out to source system data;Work as source system data
During by fault-tolerant checking, source system data is migrated to goal systems;When source system data is not by fault-tolerant checking, analysis is not
Valid data is reported, source system data is repaired and adjusted according to analysis result, until source system data passes through legitimacy
Check;Delete illegal data report.
In the technical scheme, if source system data is not over validity checking, illegal data report is generated,
Illegal data with Update log information, and are carried out fault-tolerant checking by insertion backstage log sheet, when illegal data are by fault-tolerant
After checking, by the Data Migration after checking to goal systems, the stability that system is continuously run is greatly strengthen;When illegal number
During according to not over fault-tolerant checking, by analyzing illegal data report, the illegal reason of data is found out, with to illegal number
According to being repaired and being adjusted, rechecking is then carried out again, untill inspection passes through, and the illegal data in synchronization removal backstage
Log information, can to greatest extent find and avoid risk, so that mesh after improving the stability of data migration process and migrating
The correctness of mark system data.
In the above-mentioned technical solutions, it is preferable that when source system data is by after validity checking, will be by validity checking
Source system data migrate to goal systems and specifically include:Judge whether to need Extension migratory rule;If the determination result is YES, insert
Enter new migration rules, replace original migration rules or increase rule newly on the basis of original;Otherwise selection migration model and right
The migration pattern answered;Source system data is carried out into data conversion treatment according to default transformation rule;After data conversion treatment
Data are saved in goal systems.
In the technical scheme, Extension migratory rule be in order to be directed to the personalisation process of customization development data structure,
When client carries out customization development, it is possible to special assignment treatment logic can be done to the field of customization development, is led to
Cross and insert new migration rules, replace original migration rules or increase rule newly on the basis of original, effectively solve client and melt
The problem of hair, realizes that minimum cost supports the data structure migration of customer personalized exploitation, can fast and effectively support isomery
Data Migration between system, while can also be used as Data Migration or data transfer tool between isomorphism system;If client
The personalisation process of customization development data structure need not be carried out, then by selecting suitable migration model and corresponding moving
Shifting mode, is migrated to source system data, and the data of origin system are processed as meeting goal systems by default transformation rule
The data of rule, the data after data conversion treatment are saved in the corresponding storehouse table of goal systems.
In the above-mentioned technical solutions, it is preferable that reconciliation treatment is carried out to goal systems data and source system data.
In the technical scheme, after the migration of completion source system data to goal systems data, by goal systems
Data carry out reconciliation treatment with source system data, can more quickly generate data verification report, free by manpower checking
Labor problem and operating efficiency problem, there is provided more comprehensive data verification report, shortening project is reached the standard grade time of switching.
In the above-mentioned technical solutions, it is preferable that goal systems data and source system data are carried out with the specific bag of reconciliation treatment
Include:Selection needs the document and default reconciliation dimension or self-defined reconciliation dimension of reconciliation;Document reconciliation is performed, data verification is generated
Report.
In the technical scheme, document reconciliation is carried out by selecting document and reconciliation dimension, generation data verification report is
The reconciliation dimension of preset some standards of system, if client has individual demand, can be with oneself configuration reconciliation dimension, it is right then to perform
Account, generation checking report, flexible configuration reconciliation rule meets the demand that different clients are reported data verification of correctness.
In the above-mentioned technical solutions, it is preferable that default to check that rule has scalability.
In the technical scheme, according to different systems, the default check gauge of migration demand dynamic expansion of different clients
Then, do not influence standard check rule on the premise of flexible expansion different system different clients personalized demands on examination, in source
Find to greatest extent and avoid risk before system data migration, target after improving the stability of data migration process and migrating
The correctness of system data.
In the above-mentioned technical solutions, it is preferable that control configuration mode includes following at least any one:Excel configurations are imported
Pattern, interfaceization configuration normal control mode.
In the technical scheme, source system data structure can accomplish with goal systems data structure in data migration process
Flexible configuration, configuration mode includes that excel configures introduction model and interfaceization configuration normal control mode.By flexible configuration data knot
Structure is compareed, and accomplishes to support the Data Migration between various heterogeneous systems, supports the migration of customization development data structure.This area skill
Art personnel are it should be understood that control configuration mode includes but is not limited to following at least any one:Excel configurations introduction model, interface
Change configuration normal control mode.
In the above-mentioned technical solutions, it is preferable that migration model includes following at least any one:Smooth migration, control migration;
Migration pattern includes following at least any one:Static data migration, incremental data migration.
In the technical scheme, smooth migration is exactly as its name suggests that the data of origin system are moved to according to certain rule
Goal systems;Control migration refer in goal systems some data, it is necessary to by source system data and goal systems
Data do contrast relationship, migration are then performed again, for example:There is an organization name in origin system " A001 ", goal systems has
One tissue is " B001 ", and " A001 " tissue is compareed into migration with " B001 " tissue, then the attribute of origin system " A001 " tissue
Can compare in moving to goal systems " B001 " tissue.It should be appreciated by those skilled in the art that migration model includes but does not limit
In smooth migration with compare migration both of which.
Migration pattern supports that static data migration migrates both of which with incremental data, and static data migration refers to user
Data are no longer done and increase, delete, changing the data of operation and migrated;Incremental data migration refers to the source system of the user during migration
System is also being continuing with, and also in lasting change, incremental data migration is in order to user is doing system upgrade Data Migration to data
During, do not interrupt the current business of user as far as possible, the influence to user is minimized.Those skilled in the art should manage
Solution, migration pattern includes but is not limited to static data migration and incremental data migrates two ways.
In any of the above-described technical scheme, it is preferable that log information includes following at least any one or its combination:It is illegal
Data, migrating data amount statistics, the time-consuming statistics of migration, the treatment of illegal data exception, increment migration, incremental data contrast.
In the technical scheme, it should be appreciated by those skilled in the art that log information includes but is not limited to following at least appoint
One or its combination:Illegal data, migrating data amount statistics, the time-consuming statistics of migration, the treatment of illegal data exception, increment are moved
Move, incremental data is contrasted.By log information, the problem run into transition process can be in time checked, analyzed illegal
Data are repaired and adjustment so as to follow-up, and then the data after reparation are migrated again, are prevented effectively from Data Migration interruption, further
The correctness of goal systems data after improving the stability of data migration process and migrating.
The present invention also proposes a kind of data mover system, including:Data validity checking module, for by origin system number
According to migrating to before goal systems, check that rule carries out validity checking to source system data according to default, according to inspection result
Source system data is processed;And data structure control configuration module, for by source system data structure and goal systems
Data structure carries out control configuration according to control configuration mode;Data Migration core processing module, when source system data is by closing
After method is checked, source system data is migrated to goal systems;Data validity checking module, is additionally operable to work as source system data not
During by validity checking, source system data is repaired and adjusted, until source system data passes through validity checking;Daily record
Management module, for recording the log information produced in validity checking, control configuration and transition process.
Data mover system of the invention, moves by by each traffic table or business paper in origin system
Before moving to goal systems, validity checking is first carried out, only check that the data for passing through can just carry out follow-up Data Migration, such as
Fruit inspection does not pass through, and source system data is repaired and adjusted, and rechecking is then carried out again, until passing through validity checking
Untill, can to greatest extent find and avoid risk, so that target system after improving the stability of data migration process and migrating
The correctness of data of uniting;Compareed according to control configuration mode by by source system data structure and goal systems data structure
Configuration, realizes flexibly control configuration data structure, supports the Data Migration between various heterogeneous systems, supports customization development number
According to structural transfer, so as to reduce the development cost of data migration process, autgmentability, stability and the safety of migratory system are improved
Property;By recording the log information produced in validity checking, control configuration and data migration process, can in time check and move
The problem run into during shifting, analyzes illegal data and is repaired and adjustment so as to follow-up, and then the data after reparation are moved again
Move, be prevented effectively from Data Migration interruption, goal systems data after the further stability for improving data migration process and migration
Correctness.
In addition, according to the above-mentioned data mover system of the present invention, can also have following additional technical characteristic:
In the above-mentioned technical solutions, it is preferable that when source system data does not pass through validity checking, source system data is entered
Row is repaired and adjustment is specifically included:Generation unit, for generating illegal data report;Fault-tolerant unit, for origin system number
According to carrying out fault-tolerant checking;Data Migration core processing module, is additionally operable to when source system data is by fault-tolerant checking, by origin system
Data Migration is to goal systems;Log management module, is additionally operable to, when source system data is not by fault-tolerant checking, analyze illegal
Data report;Adjustment unit is repaired, for source system data to be repaired and adjusted according to analysis result, until origin system number
According to by validity checking;Log management module, is additionally operable to delete illegal data report.
In the technical scheme, if source system data is not over validity checking, illegal data report is generated,
Illegal data with Update log information, and are carried out fault-tolerant checking by insertion backstage log sheet, when illegal data are by fault-tolerant
After checking, by the Data Migration after checking to goal systems, the stability that system is continuously run is greatly strengthen;When illegal number
During according to not over fault-tolerant checking, by analyzing illegal data report, the illegal reason of data is found out, with to illegal number
According to being repaired and being adjusted, rechecking is then carried out again, untill inspection passes through, and the illegal data in synchronization removal backstage
Log information, can to greatest extent find and avoid risk, so that mesh after improving the stability of data migration process and migrating
The correctness of mark system data.
In the above-mentioned technical solutions, it is preferable that when source system data is by after validity checking, will be by validity checking
Source system data migrate to goal systems and specifically include:Judging unit, for judging whether to need Extension migratory rule;Rule
Expanding element, for if the determination result is YES, inserting new migration rules, replaces original migration rules or on the basis of original
Newly-increased rule;Select unit is otherwise migrated, for selecting migration model and corresponding migration pattern;Date Conversion Unit, is used for
Source system data is carried out into data conversion treatment according to default transformation rule;Memory cell, for by after data conversion treatment
Data are saved in goal systems.
In the technical scheme, Extension migratory rule be in order to be directed to the personalisation process of customization development data structure,
When client carries out customization development, it is possible to special assignment treatment logic can be done to the field of customization development, is led to
Cross and insert new migration rules, replace original migration rules or increase rule newly on the basis of original, effectively solve client and melt
The problem of hair, realizes that minimum cost supports the data structure migration of customer personalized exploitation, can fast and effectively support isomery
Data Migration between system, while can also be used as Data Migration or data transfer tool between isomorphism system;If client
The personalisation process of customization development data structure need not be carried out, then by selecting suitable migration model and corresponding moving
Shifting mode, is migrated to source system data, and the data of origin system are processed as meeting goal systems by default transformation rule
The data of rule, the data after data conversion treatment are saved in the corresponding storehouse table of goal systems.
In the above-mentioned technical solutions, it is preferable that data reconciliation processing module, for goal systems data and origin system number
According to carrying out reconciliation treatment.
In the technical scheme, after the migration of completion source system data to goal systems data, by goal systems
Data carry out reconciliation treatment with source system data, can more quickly generate data verification report, free by manpower checking
Labor problem and operating efficiency problem, there is provided more comprehensive data verification report, shortening project is reached the standard grade time of switching.
In the above-mentioned technical solutions, it is preferable that goal systems data and source system data are carried out with the specific bag of reconciliation treatment
Include:Reconciliation select unit, the document and default reconciliation dimension or self-defined reconciliation dimension of reconciliation are needed for selection;Perform list
Unit, for performing document reconciliation, and generates data verification report.
In the technical scheme, document reconciliation is carried out by selecting document and reconciliation dimension, generation data verification report is
The reconciliation dimension of preset some standards of system, if client has individual demand, can be with oneself configuration reconciliation dimension, it is right then to perform
Account, generation checking report, flexible configuration reconciliation rule meets the demand that different clients are reported data verification of correctness.
In the above-mentioned technical solutions, it is preferable that default to check that rule has scalability.
In the technical scheme, according to different systems, the default check gauge of migration demand dynamic expansion of different clients
Then, do not influence standard check rule on the premise of flexible expansion different system different clients personalized demands on examination, in source
Find to greatest extent and avoid risk before system data migration, target after improving the stability of data migration process and migrating
The correctness of system data.
In the above-mentioned technical solutions, it is preferable that control configuration mode includes following at least any one:Excel configurations are imported
Pattern, interfaceization configuration normal control mode.
In the technical scheme, source system data structure can accomplish with goal systems data structure in data migration process
Flexible configuration, configuration mode includes that excel configures introduction model and interfaceization configuration normal control mode.By flexible configuration data knot
Structure is compareed, and accomplishes to support the Data Migration between various heterogeneous systems, supports the migration of customization development data structure.This area skill
Art personnel are it should be understood that control configuration mode includes but is not limited to following at least any one:Excel configurations introduction model, interface
Change configuration normal control mode.
In the above-mentioned technical solutions, it is preferable that migration model includes following at least any one:Smooth migration, control migration;
Migration pattern includes following at least any one:Static data migration, incremental data migration.
In the technical scheme, smooth migration is exactly as its name suggests that the data of origin system are moved to according to certain rule
Goal systems;Control migration refer in goal systems some data, it is necessary to by source system data and goal systems
Data do contrast relationship, migration are then performed again, for example:There is an organization name in origin system " A001 ", goal systems has
One tissue is " B001 ", and " A001 " tissue is compareed into migration with " B001 " tissue, then the attribute of origin system " A001 " tissue
Can compare in moving to goal systems " B001 " tissue.It should be appreciated by those skilled in the art that migration model includes but does not limit
In smooth migration with compare migration both of which.
Migration pattern supports that static data migration migrates both of which with incremental data, and static data migration refers to user
Data are no longer done and increase, delete, changing the data of operation and migrated;Incremental data migration refers to the source system of the user during migration
System is also being continuing with, and also in lasting change, incremental data migration is in order to user is doing system upgrade Data Migration to data
During, do not interrupt the current business of user as far as possible, the influence to user is minimized.Those skilled in the art should manage
Solution, migration pattern includes but is not limited to static data migration and incremental data migrates two ways.
In any of the above-described technical scheme, it is preferable that log information includes following at least any one or its combination:It is illegal
Data, migrating data amount statistics, the time-consuming statistics of migration, the treatment of illegal data exception, increment migration, incremental data contrast.
In the technical scheme, it should be appreciated by those skilled in the art that log information includes but is not limited to following at least appoint
One or its combination:Illegal data, migrating data amount statistics, the time-consuming statistics of migration, the treatment of illegal data exception, increment are moved
Move, incremental data is contrasted.By log information, the problem run into transition process can be in time checked, analyzed illegal
Data are repaired and adjustment so as to follow-up, and then the data after reparation are migrated again, are prevented effectively from Data Migration interruption, further
The correctness of goal systems data after improving the stability of data migration process and migrating.
Additional aspect of the invention and advantage will become obvious in following description section, or by practice of the invention
Recognize.
Brief description of the drawings
Of the invention above-mentioned and/or additional aspect and advantage will become from description of the accompanying drawings below to embodiment is combined
Substantially and be readily appreciated that, wherein:
Fig. 1 shows the schematic flow sheet of data migration method according to an embodiment of the invention;
Fig. 2 shows the schematic flow sheet of data validity checking according to an embodiment of the invention;
Fig. 3 a show that Data Migration according to an embodiment of the invention processes the schematic flow sheet of logic;
Fig. 3 b show the schematic flow sheet of Data Migration treatment logic according to still a further embodiment;
Fig. 4 shows the schematic flow sheet of data migration method according to still a further embodiment;
Fig. 5 shows the schematic flow sheet of reconciliation treatment according to an embodiment of the invention;
Fig. 6 shows the schematic block diagram of data mover system according to an embodiment of the invention;
Fig. 7 shows the schematic block diagram of data mover system according to still a further embodiment;
Fig. 8 shows the schematic block diagram of Data Migration core processing module according to an embodiment of the invention;
Fig. 9 shows the schematic block diagram of data mover system according to another embodiment of the invention;
Figure 10 shows the schematic block diagram of data reconciliation processing module according to an embodiment of the invention;
Figure 11 shows the configuration diagram of data mover system according to an embodiment of the invention.
Specific embodiment
It is below in conjunction with the accompanying drawings and specific real in order to be more clearly understood that the above objects, features and advantages of the present invention
Mode is applied to be further described in detail the present invention.It should be noted that in the case where not conflicting, the implementation of the application
Feature in example and embodiment can be mutually combined.
Many details are elaborated in the following description in order to fully understand the present invention, but, the present invention may be used also
Implemented with being different from other modes described here using other, therefore, protection scope of the present invention is not by described below
Specific embodiment limitation.
As shown in figure 1, the schematic flow sheet of data migration method according to an embodiment of the invention:
Step 102, migrates to before goal systems by source system data, checks rule to source system data according to default
Validity checking is carried out, source system data is processed according to inspection result;
Step 104, source system data structure and goal systems data structure are matched somebody with somebody according to compareing configuration mode and carry out control
Put;
Step 106, when source system data is by after validity checking, source system data being migrated to goal systems;
Step 108, source system data when source system data does not pass through validity checking, repaired and adjusted, directly
Pass through validity checking to source system data;
Step 110, the log information produced in record validity checking, control configuration and transition process.
In this embodiment, migrated to goal systems by by each traffic table or business paper in origin system
Before, validity checking is first carried out, only checks that the data for passing through can just carry out follow-up Data Migration, if checked obstructed
Cross, source system data is repaired and adjusted, rechecking is then carried out again, untill by validity checking, can
Find to greatest extent and avoid risk, thus improve data migration process stability and migration after goal systems data just
True property;Control configuration is carried out according to control configuration mode by by source system data structure and goal systems data structure, is realized
Flexibly control configuration data structure, supports the Data Migration between various heterogeneous systems, supports that customization development data structure is moved
Move, so as to reduce the development cost of data migration process, improve autgmentability, stability and the security of migratory system;By note
The log information produced in record validity checking, control configuration and data migration process, in can in time checking transition process
The problem for running into, analyzes illegal data and is repaired and adjustment so as to follow-up, and then the data after reparation are migrated again, effectively keeps away
Exempt from Data Migration interruption, the correctness of goal systems data after further improving the stability of data migration process and migrating.
As shown in Fig. 2 the schematic flow sheet of data validity checking according to an embodiment of the invention:
Step 202, validity checking is carried out to source system data;When source system data does not pass through validity checking, hold
Row step 204;When source system data passes through validity checking, step 208 is performed;
Step 204, generates illegal data report;
Step 206, fault-tolerant checking is carried out to source system data;When source system data is by fault-tolerant checking, step is performed
208;When source system data is not by fault-tolerant checking, step 210 is performed;
Step 208, source system data is migrated to goal systems;
Step 210, analyzes illegal data report, and source system data is repaired and adjusted according to analysis result, weight
Multiple step 202, until source system data passes through validity checking
Step 212, the log information produced in record validity checking and transition process;
Step 214, deletes illegal data report.
In this embodiment, if source system data is not over validity checking, illegal data report is generated, is inserted
Enter backstage log sheet, with Update log information, and fault-tolerant checking is carried out to illegal data, when illegal data are tested by fault-tolerant
After card, by the Data Migration after checking to goal systems, the stability that system is continuously run is greatly strengthen;When illegal data
During not over fault-tolerant checking, by analyzing illegal data report, the illegal reason of data is found out, with to illegal data
Repaired and adjusted, rechecking is then carried out again, untill inspection passes through, and the illegal data day in synchronization removal backstage
Will information, can to greatest extent find and avoid risk, so that target after improving the stability of data migration process and migrating
The correctness of system data.Data validity checking demonstration effect such as table 1:
The data validity checking demonstration effect of table 1
As shown in Figure 3 a, Data Migration according to an embodiment of the invention processes the schematic flow sheet of logic:
Step 302, judges whether to need Extension migratory rule;If the determination result is YES, step 304 is performed;Otherwise perform
Step 306;
Step 304, inserts new migration rules, replaces original migration rules or increases rule newly on the basis of original;
Step 306, selects migration model and corresponding migration pattern;
Step 308, data conversion treatment is carried out by source system data according to default transformation rule;
Data after data conversion treatment are saved in goal systems by step 310
Step 312, the log information produced in record transition process.
In this embodiment, Extension migratory rule be in order to be directed to the personalisation process of customization development data structure, when
When client carries out customization development, it is possible to special assignment treatment logic can be done to the field of customization development, is passed through
The new migration rules of insertion, replace original migration rules or increase rule newly on the basis of original, effectively solve customization development
Problem, realize that minimum cost supports the data structure migration of customer personalized exploitation, can fast and effectively support isomery system
Data Migration between system, while can also be used as Data Migration or data transfer tool between isomorphism system;If client is not
Need to carry out the personalisation process of customization development data structure, then by selecting suitable migration model and corresponding migration
Mode, is migrated to source system data, and the data of origin system are processed as by default transformation rule to meet goal systems rule
Data then, the data after data conversion treatment are saved in the corresponding storehouse table of goal systems.
As shown in Figure 3 b, Data Migration according to still a further embodiment processes the schematic flow sheet of logic:
Step 302, judges whether to need Extension migratory rule;If the determination result is YES, step 304 is performed;Otherwise perform
Step 306;
Step 304, inserts new migration rules, replaces original migration rules or increases rule newly on the basis of original;
Step 306, selects migration model and corresponding migration pattern;
Step 308, source data backup;
Step 310, inquires about backup source data;
Step 312, data conversion treatment is carried out by source system data according to default transformation rule;
Data after data conversion treatment are saved in goal systems by step 314
Step 316, the log information produced in record transition process.
In this embodiment, Data Migration treatment logic includes:Selection migration model, migration rules extension, source data are standby
Part (choosing is done), inquiry source data, data conversion treatment, preservation database, wherein source data backup, data conversion treatment, preserve
The traveling locus of generation all recorded backstage daily record by the process of database, subsequently through analysis log information, process and solve
Migration problem.Source data backup be the database table of origin system is first backed up it is a in goal systems database, then from standby
Read data in part table to move in goal systems in corresponding storehouse table, this migration model is migrated suitable for static data.When
Complete to avoid the need for operating source system data storehouse again after backup source database table, data and the target database table of backup exist
Same database, it is convenient to be processed followed by Data Migration error analysis and reconciliation.
As shown in figure 4, the schematic flow sheet of data migration method according to still a further embodiment:
Step 402, migrates to before goal systems by source system data, checks rule to source system data according to default
Validity checking is carried out, source system data is processed according to inspection result;
Step 404, source system data structure and goal systems data structure are matched somebody with somebody according to compareing configuration mode and carry out control
Put;
Step 406, when source system data is by after validity checking, source system data being migrated to goal systems;
Step 408, source system data when source system data does not pass through validity checking, repaired and adjusted, directly
Pass through validity checking to source system data;
Step 410, the log information produced in record validity checking, control configuration and transition process;
Goal systems data and source system data are carried out reconciliation treatment by step 412.
In this embodiment, after the migration of completion source system data to goal systems data, by goal systems number
Reconciliation treatment is carried out according to source system data, data verification report can be more quickly generated, freed by manpower checking
Labor problem and operating efficiency problem, there is provided more comprehensive data verification report, shortening project are reached the standard grade time of switching.
As shown in figure 5, the schematic flow sheet of reconciliation treatment according to an embodiment of the invention:
Step 502, selection needs the document of reconciliation;
Step 504, the default reconciliation dimension of selection;Or
Step 506, self-defined reconciliation dimension;
Step 508, performs document reconciliation, generation data verification report.
In this embodiment, document reconciliation, generation data verification report, system are carried out by selecting document and reconciliation dimension
The reconciliation dimension of preset some standards, if client has individual demand, can be with oneself configuration reconciliation dimension, it is right then to perform
Account, generation checking report, flexible configuration reconciliation rule meets the demand that different clients are reported data verification of correctness.
In the above embodiment, it is preferable that default check that rule has scalability.
In this embodiment, according to different systems, the migration demand dynamic expansion of different clients is default to check rule,
The personalized demands on examination of flexible expansion different system different clients on the premise of not influenceing standard to check rule, in origin system
Found to greatest extent before Data Migration and avoided risk, goal systems after improving the stability of data migration process and migrating
The correctness of data.
In the above embodiment, it is preferable that control configuration mode includes following at least any one:Excel configurations import mould
Formula, interfaceization configuration normal control mode.
In this embodiment, source system data structure can accomplish spirit with goal systems data structure in data migration process
Configuration living, configuration mode includes that excel configures introduction model and interfaceization configuration normal control mode.By flexibly control configuration data
Structure, accomplishes to support the Data Migration between various heterogeneous systems, the migration of customization development data structure is supported, to meet difference
The demand of customization development.It should be appreciated by those skilled in the art that control configuration mode include but is not limited to it is following at least any
:Excel configurations introduction model, interfaceization configuration normal control mode.
Specific embodiment one, the data structure contrast relationship of origin system and goal systems is configured by excel, and configuration is completed
Source system data is imported into goal systems via foreground import feature later, is closed according to control in follow-up migration logic
Source system data storehouse literary name segment value is moved to the database table field value of corresponding goal systems for system.Excel configurations import mould
Formula contrast template such as table 2:
Table 2excel configures introduction model contrast template
Specific embodiment two, foreground interfaceization configuration normal control mode, the left side is goal systems database table structure (data dictionary),
The right is origin system database table structure (data dictionary), and the database table structure of origin system and goal systems is all by storehouse table name from source
Load out in system library and target database.Target database database table structure is sorted alphabetically and is set forth in interface, source system
System correspondence document database table structure is designed to reference pattern and manually enters both of which, is selected to use according to different rule types
Reference pattern or pattern is manually entered, if rule type is assignment, using manual typing pattern, if other are regular
Type, then by referring to the database table structure for enumerating correspondence document table, then field name corresponding with goal systems is referred to as control,
Advantage of this is that avoid manually entering causing mistake.Origin system can more flexibly be realized by the configuration of foreground interface
Field between goal systems is compareed, and is capable of the data structure migration of minimum cost support customization development, is reduced data and is moved
Customization development cost is moved, autgmentability, stability and the security of migratory system is improved.Interfaceization configuration normal control mode template is such as
Table 3:
The interfaceization of table 3 configures normal control mode template
Object table Property Name | Target Table Properties | Target data type | Rule type | Source Table Properties title | Source Table Properties | Source data type | Remarks |
Buying storage bill master meter ID | cbillid | char | Directly compare | Document is identified | cbillid | char | Directly compare |
Printing times | iprintcount | int | Directly compare | Printing times | iprintcount | int | Conversion int types |
Founder | creator | varchar | PK is translated | The single mark of system | coperatorid | varchar | Translation |
Warehouse | cstordocid | varchar | PK is translated | Warehouse is identified | cwarehouse | varchar | Translation |
Inventory organization latest edition | cstockorgid | varchar | PK is translated | Inventory organization is identified | crdcenterid | varchar | Translation |
Document date | dbilldate | char | Control | Document date | dbilldate | char | Date conversion is directly compareed |
Accounting period | caccountperiod | char | Splicing | Accounting month | caccountmd | varchar | Fiscal year+accounting month |
Accounting period | caccountperiod | char | Splicing | Fiscal year | caccountye | varchar | Fiscal year+accounting month |
In the above embodiment, it is preferable that migration model includes following at least any one:Smooth migration, control migration;Move
Shifting mode includes following at least any one:Static data migration, incremental data migration.
In this embodiment, smooth migration is exactly as its name suggests that the data of origin system are moved into mesh according to certain rule
Mark system;Control migration refer in goal systems some data, it is necessary to by source system data and goal systems number
According to contrast relationship is done, migration is then performed again, for example:There is an organization name in origin system " A001 ", goal systems has one
Individual tissue is " B001 ", and " A001 " tissue is compareed into migration with " B001 " tissue, then the attribute meeting of origin system " A001 " tissue
Control is moved in " B001 " tissue of goal systems.It should be appreciated by those skilled in the art that migration model is included but is not limited to
Smooth migration with compare migration both of which.
Migration pattern supports that static data migration migrates both of which with incremental data, and static data migration refers to user
Data are no longer done and increase, delete, changing the data of operation and migrated;Incremental data migration refers to the source system of the user during migration
System is also being continuing with, and also in lasting change, incremental data migration is in order to user is doing system upgrade Data Migration to data
During, do not interrupt the current business of user as far as possible, the influence to user is minimized.Those skilled in the art should manage
Solution, migration pattern includes but is not limited to static data migration and incremental data migrates two ways.
In any of the above-described embodiment, it is preferable that log information includes following at least any one or its combination:Illegal number
Contrasted according to, migrating data amount statistics, the time-consuming statistics of migration, the treatment of illegal data exception, increment migration, incremental data.
In this embodiment, it should be appreciated by those skilled in the art that log information include but is not limited to it is following at least any
Item or its combination:Illegal data, migrating data amount statistics, the time-consuming statistics of migration, the treatment of illegal data exception, increment are moved
Move, incremental data is contrasted.By log information, the problem run into transition process can be in time checked, analyzed illegal
Data are repaired and adjustment so as to follow-up, and then the data after reparation are migrated again, are prevented effectively from Data Migration interruption, further
The correctness of goal systems data after improving the stability of data migration process and migrating.
As shown in fig. 6, the schematic block diagram of data mover system according to an embodiment of the invention:
Data validity checking module 602, for being migrated to before goal systems by source system data, according to default inspection
Look into rule carries out validity checking to source system data, and source system data is processed according to inspection result;And
Data structure compare configuration module 604, for by source system data structure with goal systems data structure according to right
Control configuration is carried out according to configuration mode;
Data Migration core processing module 606, when source system data is by after validity checking, source system data being migrated
To goal systems;
Data validity checking module 602, is additionally operable to when source system data does not pass through validity checking, by origin system number
According to being repaired and being adjusted, until source system data passes through validity checking;
Log management module 608, for recording the daily record letter produced in validity checking, control configuration and transition process
Breath.
In this embodiment, migrated to goal systems by by each traffic table or business paper in origin system
Before, validity checking is first carried out, only checks that the data for passing through can just carry out follow-up Data Migration, if checked obstructed
Cross, source system data is repaired and adjusted, rechecking is then carried out again, untill by validity checking, can
Find to greatest extent and avoid risk, thus improve data migration process stability and migration after goal systems data just
True property;Control configuration is carried out according to control configuration mode by by source system data structure and goal systems data structure, is realized
Flexibly control configuration data structure, supports the Data Migration between various heterogeneous systems, supports that customization development data structure is moved
Move, so as to reduce the development cost of data migration process, improve autgmentability, stability and the security of migratory system;By note
The log information produced in record validity checking, control configuration and data migration process, in can in time checking transition process
The problem for running into, analyzes illegal data and is repaired and adjustment so as to follow-up, and then the data after reparation are migrated again, effectively keeps away
Exempt from Data Migration interruption, the correctness of goal systems data after further improving the stability of data migration process and migrating.
As shown in fig. 7, the schematic block diagram of data mover system according to still a further embodiment:
Data validity checking module 702 includes:Generation unit 7022, for generating illegal data report;
Data structure compare configuration module 704, for by source system data structure with goal systems data structure according to right
Control configuration is carried out according to configuration mode;
Data Migration core processing module 706, is additionally operable to when source system data is by fault-tolerant checking, by source system data
Migrate to goal systems;
Log management module 708, is additionally operable to, when source system data is not by fault-tolerant checking, analyze illegal datagram
Accuse;
Data validity checking module 702 also includes:Repair adjustment unit 7024, for according to analysis result to origin system
Data are repaired and adjusted, until source system data passes through validity checking;
Log management module 708, is additionally operable to delete illegal data report.
In this embodiment, if source system data is not over validity checking, illegal data report is generated, is inserted
Enter backstage log sheet, with Update log information, and fault-tolerant checking is carried out to illegal data, when illegal data are tested by fault-tolerant
After card, by the Data Migration after checking to goal systems, the stability that system is continuously run is greatly strengthen;When illegal data
During not over fault-tolerant checking, by analyzing illegal data report, the illegal reason of data is found out, with to illegal data
Repaired and adjusted, rechecking is then carried out again, untill inspection passes through, and the illegal data day in synchronization removal backstage
Will information, can to greatest extent find and avoid risk, so that target after improving the stability of data migration process and migrating
The correctness of system data.
As shown in figure 8, the schematic block diagram of Data Migration core processing module according to an embodiment of the invention:
Data Migration core processing module 800 includes:Judging unit 802, for judging whether to need Extension migratory rule;
Rule Extended unit 804, for if the determination result is YES, inserting new migration rules, replaces original migration rules;
Migration select unit 806, for selecting migration model and corresponding migration pattern;
Date Conversion Unit 808, for source system data to be carried out into data conversion treatment according to default transformation rule;
Memory cell 810, for the data after data conversion treatment to be saved in into goal systems.
In this embodiment, Extension migratory rule be in order to be directed to the personalisation process of customization development data structure, when
When client carries out customization development, it is possible to special assignment treatment logic can be done to the field of customization development, is passed through
The new migration rules of insertion, replace original migration rules, effectively solve the problems, such as customization development, realize that minimum cost supports visitor
The data structure migration of the personalized exploitation in family, can fast and effectively support the Data Migration between heterogeneous system, while also may be used
As Data Migration or data transfer tool between isomorphism system;If client need not carry out customization development data structure
Personalisation process, then by selecting suitable migration model and corresponding migration pattern, source system data is migrated,
The data of origin system are processed as meeting the data of goal systems rule by default transformation rule, after data conversion treatment
Data are saved in the corresponding storehouse table of goal systems.
As Fig. 9 shows, the schematic block diagram of data mover system according to another embodiment of the invention:
Data validity checking module 902, for being migrated to before goal systems by source system data, according to default inspection
Look into rule carries out validity checking to source system data, and source system data is processed according to inspection result;Wherein, data are closed
Method checks that module 902 includes:Generation unit 9022, for generating illegal data report;Adjustment unit 9024 is repaired, is used for
Source system data is repaired and adjusted according to analysis result, until source system data passes through validity checking;
Data structure compare configuration module 904, for by source system data structure with goal systems data structure according to right
Control configuration is carried out according to configuration mode;
Data Migration core processing module 906, when source system data is by after validity checking, source system data being migrated
To goal systems;Wherein, Data Migration core processing module 906 includes:Judging unit 9062, for judging whether to need extension
Migration rules;Rule Extended unit 9064, for if the determination result is YES, inserting new migration rules, replaces original migration rule
Then;Migration select unit 9066, for selecting migration model and corresponding migration pattern;Date Conversion Unit 9068, for inciting somebody to action
Source system data carries out data conversion treatment according to default transformation rule;Memory cell 9070, for by after data conversion treatment
Data be saved in goal systems;
Data validity checking module 902, is additionally operable to when source system data does not pass through validity checking, by origin system number
According to being repaired and being adjusted, until source system data passes through validity checking;
Log management module 908, for recording the daily record letter produced in validity checking, control configuration and transition process
Breath;
Data reconciliation processing module 910, for carrying out reconciliation treatment to goal systems data and source system data.
In this embodiment, after the migration of completion source system data to goal systems data, by goal systems number
Reconciliation treatment is carried out according to source system data, data verification report can be more quickly generated, freed by manpower checking
Labor problem and operating efficiency problem, there is provided more comprehensive data verification report, shortening project are reached the standard grade time of switching.
As shown in Figure 10, the schematic block diagram of data reconciliation processing module according to an embodiment of the invention:
Data reconciliation processing module 1000 includes:Reconciliation select unit 1002, for selecting to need the document of reconciliation and pre-
If reconciliation dimension or self-defined reconciliation dimension;Execution unit 1004, for performing document reconciliation, and generates data verification report.
In this embodiment, document reconciliation, generation data verification report, system are carried out by selecting document and reconciliation dimension
The reconciliation dimension of preset some standards, if client has individual demand, can be with oneself configuration reconciliation dimension, it is right then to perform
Account, generation checking report, flexible configuration reconciliation rule meets the demand that different clients are reported data verification of correctness.
In the above embodiment, it is preferable that default check that rule has scalability.
In this embodiment, according to different systems, the migration demand dynamic expansion of different clients is default to check rule,
The personalized demands on examination of flexible expansion different system different clients on the premise of not influenceing standard to check rule, in origin system
Found to greatest extent before Data Migration and avoided risk, goal systems after improving the stability of data migration process and migrating
The correctness of data.
In the above embodiment, it is preferable that control configuration mode includes following at least any one:Excel configurations import mould
Formula, interfaceization configuration normal control mode.
In this embodiment, source system data structure can accomplish spirit with goal systems data structure in data migration process
Configuration living, configuration mode includes that excel configures introduction model and interfaceization configuration normal control mode.By flexible configuration data structure
Control, accomplishes to support the Data Migration between various heterogeneous systems, supports the migration of customization development data structure.Art technology
Personnel are it should be understood that control configuration mode includes but is not limited to following at least any one:Excel configurations introduction model, interface
Configuration normal control mode.
In the above embodiment, it is preferable that migration model includes following at least any one:Smooth migration, control migration;Move
Shifting mode includes following at least any one:Static data migration, incremental data migration.
In this embodiment, smooth migration is exactly as its name suggests that the data of origin system are moved into mesh according to certain rule
Mark system;Control migration refer in goal systems some data, it is necessary to by source system data and goal systems number
According to contrast relationship is done, migration is then performed again, for example:There is an organization name in origin system " A001 ", goal systems has one
Individual tissue is " B001 ", and " A001 " tissue is compareed into migration with " B001 " tissue, then the attribute meeting of origin system " A001 " tissue
Control is moved in " B001 " tissue of goal systems.It should be appreciated by those skilled in the art that migration model is included but is not limited to
Smooth migration with compare migration both of which.
Migration pattern supports that static data migration migrates both of which with incremental data, and static data migration refers to user
Data are no longer done and increase, delete, changing the data of operation and migrated;Incremental data migration refers to the source system of the user during migration
System is also being continuing with, and also in lasting change, incremental data migration is in order to user is doing system upgrade Data Migration to data
During, do not interrupt the current business of user as far as possible, the influence to user is minimized.Those skilled in the art should manage
Solution, migration pattern includes but is not limited to static data migration and incremental data migrates two ways.
In any of the above-described embodiment, it is preferable that log information includes following at least any one or its combination:Illegal number
Contrasted according to, migrating data amount statistics, the time-consuming statistics of migration, the treatment of illegal data exception, increment migration, incremental data.
In this embodiment, it should be appreciated by those skilled in the art that log information include but is not limited to it is following at least any
Item or its combination:Illegal data, migrating data amount statistics, the time-consuming statistics of migration, the treatment of illegal data exception, increment are moved
Move, incremental data is contrasted.By log information, the problem run into transition process can be in time checked, analyzed illegal
Data are repaired and adjustment so as to follow-up, and then the data after reparation are migrated again, are prevented effectively from Data Migration interruption, further
The correctness of goal systems data after improving the stability of data migration process and migrating.
As shown in figure 11, the configuration diagram of data mover system according to an embodiment of the invention:Data Migration
System 1100 is included comprising 5 big modules:Data validity checking module 1102, data conjunction was carried out to origin system DB before migration
Method checks, analyzes and avoid risk before system data migration is done as far as possible;Data structure compares configuration module 1104, makes source
System DB structures can accomplish flexible configuration with goal systems DB structures, by flexible configuration data structure, realize supporting various
Data Migration between heterogeneous system, supports the migration of customization development data structure;Data Migration core processing module 1106, leads to
Dynamic expansion Data Migration rule is crossed, the Data Migration between heterogeneous system can be fast and effectively supported, and effectively solves visitor
Family melts the problem of hair, while can also be used as Data Migration or data transfer tool between isomorphism system;Log management
Module 1108, by recording the log information produced in validity checking, control configuration and data migration process, can be timely
The problem that runs into examination transition process, analyzes illegal data and repairs and adjust so as to follow-up, and then by the data after reparation
Again migrate, be prevented effectively from Data Migration interruption, goal systems after further improving the stability of data migration process and migrating
The correctness of data;Data reconciliation processing module 1110, can be with flexible configuration reconciliation rule, to meet different clients to data just
The demand of true property checking report, performs the data reconciliation of origin system DB and goal systems DB, generation after Data Migration completion
Data verification report, frees the labour by manpower checking and efficiency, there is provided more comprehensive data verification report, contracting
Short project is reached the standard grade time of switching.
Data mover system 1100, logic is migrated using configurable mode processing data, and Data Migration is worked becomes
It is more flexible, easy-to-use, heterogeneous system Data Migration is become simpler, efficiently;Meanwhile, the Data Migration system of present invention design
System can also more flexibly support the Data Migration between isomorphism system, it is also possible to support the Data Migration of same system to not
With in the database of version.
The preferred embodiments of the present invention are the foregoing is only, is not intended to limit the invention, for the skill of this area
For art personnel, the present invention can have various modifications and variations.It is all within the spirit and principles in the present invention, made any repair
Change, equivalent, improvement etc., should be included within the scope of the present invention.
Claims (18)
1. a kind of data migration method, it is characterised in that including:
Migrated to before goal systems by source system data, the source system data is carried out according to default inspection rule legal
Property check, the source system data is processed according to inspection result;And
Source system data structure is carried out into control configuration with goal systems data structure according to configuration mode is compareed;
When the source system data is by after the validity checking, the source system data being migrated to the goal systems;
When the source system data is not by the validity checking, the source system data repaired and adjusted, directly
To the source system data by the validity checking;
Record the log information produced in the validity checking, control configuration and the transition process.
2. data migration method according to claim 1, it is characterised in that described when the source system data does not pass through institute
When stating validity checking, the source system data is repaired and adjustment is specifically included:
Generate illegal data report;
Fault-tolerant checking is carried out to the source system data;
When the source system data is by the fault-tolerant checking, the source system data is migrated to the goal systems;
When the source system data is not by the fault-tolerant checking, the illegal data report is analyzed, according to analysis result
The source system data is repaired and adjusted, until the source system data is by the validity checking;
Delete the illegal data report.
3. data migration method according to claim 1, it is characterised in that described when the source system data is by described
After validity checking, specifically included being migrated to the goal systems by the source system data of the validity checking:
Judge whether to need Extension migratory rule;
If the determination result is YES, new migration rules are inserted, original migration rules is replaced or is increased rule newly on the basis of original;
Otherwise
Selection migration model and corresponding migration pattern;
The source system data is carried out into data conversion treatment according to default transformation rule;
Data after data conversion treatment are saved in the goal systems.
4. data migration method according to claim 1, it is characterised in that also include:
Reconciliation treatment is carried out to the goal systems data and the source system data.
5. data migration method according to claim 4, it is characterised in that it is described to the goal systems data with it is described
Source system data carries out reconciliation treatment and specifically includes:
Selection needs the document and default reconciliation dimension or self-defined reconciliation dimension of reconciliation;
Perform document reconciliation, generation data verification report.
6. data migration method according to claim 1, it is characterised in that
The default inspection rule has scalability.
7. data migration method according to claim 1, it is characterised in that
The control configuration mode includes following at least any one:Excel configurations introduction model, interfaceization configuration normal control mode.
8. data migration method according to claim 3, it is characterised in that
The migration model includes following at least any one:Smooth migration, control migration;
The migration pattern includes following at least any one:Static data migration, incremental data migration.
9. the data migration method according to any one of claim 1 to 8, it is characterised in that
The log information includes following at least any one or its combination:Illegal data, migrating data amount statistics, migration are time-consuming
Statistics, the treatment of illegal data exception, increment migration, incremental data contrast.
10. a kind of data mover system, it is characterised in that including:
Data validity checking module, for being migrated to before goal systems by source system data, rule is checked according to default
Validity checking is carried out to the source system data, the source system data is processed according to inspection result;And
Data structure compares configuration module, for source system data structure and goal systems data structure to be configured into mould according to compareing
Formula carries out control configuration;
Data Migration core processing module, when the source system data is by after the validity checking, by the origin system number
According to migrating to the goal systems;
The data validity checking module, is additionally operable to when the source system data is not by the validity checking, by institute
State source system data to be repaired and adjusted, until the source system data is by the validity checking;
Log management module, for recording the validity checking, control configuration and generation in the transition process
Log information.
11. data mover systems according to claim 10, it is characterised in that described when the source system data does not pass through
During the validity checking, the source system data is repaired and adjustment is specifically included:
Generation unit, for generating illegal data report;
Fault-tolerant unit, for carrying out fault-tolerant checking to the source system data;
Data Migration core processing module, is additionally operable to when the source system data is by the fault-tolerant checking, by the source system
System Data Migration is to the goal systems;
The log management module, is additionally operable to when the source system data is not by the fault-tolerant checking, does not conform to described in analysis
Method data report;
Adjustment unit is repaired, for the source system data to be repaired and adjusted according to analysis result, until the source system
System data are by the validity checking;
The log management module, is additionally operable to delete the illegal data report.
12. data mover systems according to claim 10, it is characterised in that described when the source system data passes through institute
After stating validity checking, specifically included being migrated to the goal systems by the source system data of the validity checking:
Judging unit, for judging whether to need Extension migratory rule;
Rule Extended unit, for if the determination result is YES, inserting new migration rules, replaces original migration rules or original
On the basis of increase newly rule;Otherwise
Migration select unit, for selecting migration model and corresponding migration pattern;
Date Conversion Unit, for the source system data to be carried out into data conversion treatment according to default transformation rule;
Memory cell, for the data after data conversion treatment to be saved in into the goal systems.
13. data mover systems according to claim 10, it is characterised in that also include:
Data reconciliation processing module, for carrying out reconciliation treatment to the goal systems data and the source system data.
14. data mover systems according to claim 13, it is characterised in that described to the goal systems data and institute
State source system data carry out reconciliation treatment specifically include:
Reconciliation select unit, the document and default reconciliation dimension or self-defined reconciliation dimension of reconciliation are needed for selection;
Execution unit, for performing document reconciliation, and generates data verification report.
15. data mover systems according to claim 10, it is characterised in that
The default inspection rule has scalability.
16. data mover systems according to claim 10, it is characterised in that
The control configuration mode includes following at least any one:Excel configurations introduction model, interfaceization configuration normal control mode.
17. data mover systems according to claim 12, it is characterised in that
The migration model includes following at least any one:Smooth migration, control migration;
The migration pattern includes following at least any one:Static data migration, incremental data migration.
18. data mover system according to any one of claim 10 to 17, it is characterised in that
The log information includes following at least any one or its combination:Illegal data, migrating data amount statistics, migration are time-consuming
Statistics, the treatment of illegal data exception, increment migration, incremental data contrast.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201611074474.6A CN106776876A (en) | 2016-11-29 | 2016-11-29 | Data migration method and data mover system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201611074474.6A CN106776876A (en) | 2016-11-29 | 2016-11-29 | Data migration method and data mover system |
Publications (1)
Publication Number | Publication Date |
---|---|
CN106776876A true CN106776876A (en) | 2017-05-31 |
Family
ID=58900612
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201611074474.6A Pending CN106776876A (en) | 2016-11-29 | 2016-11-29 | Data migration method and data mover system |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN106776876A (en) |
Cited By (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107506451A (en) * | 2017-08-28 | 2017-12-22 | 泰康保险集团股份有限公司 | abnormal information monitoring method and device for data interaction |
CN108647270A (en) * | 2018-04-28 | 2018-10-12 | 尚谷科技(天津)有限公司 | A method of the Data Migration based on fault-tolerant time daily record |
CN108959205A (en) * | 2018-06-27 | 2018-12-07 | 成都泰盟软件有限公司 | A kind of method of ERP system export data fast transferring |
CN109639778A (en) * | 2018-11-29 | 2019-04-16 | 威富通科技有限公司 | A kind of data transmission method, private clound and server |
CN109639468A (en) * | 2018-11-29 | 2019-04-16 | 威富通科技有限公司 | A kind of data mutual-conductance method, public cloud and server |
CN109684298A (en) * | 2018-09-07 | 2019-04-26 | 平安科技(深圳)有限公司 | Database migration method, apparatus, equipment and storage medium |
CN109905384A (en) * | 2019-02-19 | 2019-06-18 | 北京深思数盾科技股份有限公司 | Data migration method and system |
CN110297813A (en) * | 2019-05-22 | 2019-10-01 | 平安银行股份有限公司 | Data migration method, device, computer equipment and storage medium |
CN111159150A (en) * | 2019-12-19 | 2020-05-15 | 北京文安智能技术股份有限公司 | Data expansion method and device |
CN111597197A (en) * | 2020-06-29 | 2020-08-28 | 腾讯科技(深圳)有限公司 | Data reconciliation method and device between databases, storage medium and electronic equipment |
CN112597128A (en) * | 2020-12-17 | 2021-04-02 | 广东南方通信建设有限公司 | Data migration method and device |
CN112860666A (en) * | 2021-02-08 | 2021-05-28 | 土巴兔集团股份有限公司 | Data migration method and device, computer equipment and storage medium |
CN113064906A (en) * | 2021-04-21 | 2021-07-02 | 杭州天谷信息科技有限公司 | Binlog log data adaptive migration method and system |
CN113239019A (en) * | 2021-06-03 | 2021-08-10 | 锋泾(中国)建材集团有限公司 | Data migration method and system between ERP systems, electronic device and medium |
CN113409015A (en) * | 2021-06-21 | 2021-09-17 | 浙江省能源集团有限公司 | ERP system and use method of ERP system in double-activity deployment of double data centers |
CN116107993A (en) * | 2022-12-26 | 2023-05-12 | 北京万里开源软件有限公司 | Data migration evaluation method and system in MySQL protocol database |
CN116756247A (en) * | 2023-08-21 | 2023-09-15 | 腾讯科技(深圳)有限公司 | Data restoration method, device, computer equipment and storage medium |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101706803A (en) * | 2009-11-27 | 2010-05-12 | 天津工业大学 | Method for migrating production data of weaving workshop |
CN103176988A (en) * | 2011-12-21 | 2013-06-26 | 上海博腾信息科技有限公司 | Data migration system based on software-as-a-service (SaaS) |
CN104424283A (en) * | 2013-08-30 | 2015-03-18 | 阿里巴巴集团控股有限公司 | Data migration system and data migration method |
CN104461516A (en) * | 2014-11-21 | 2015-03-25 | 北京锐安科技有限公司 | Data processing method and device |
CN105354314A (en) * | 2015-11-10 | 2016-02-24 | 中国建设银行股份有限公司 | Data migration method and device |
CN105739929A (en) * | 2016-01-29 | 2016-07-06 | 哈尔滨工业大学深圳研究生院 | Data center selection method for big data to migrate to cloud |
CN105740411A (en) * | 2016-01-30 | 2016-07-06 | 武汉大学 | SOA (Service-Oriented Architecture) and WebService based data migration method |
CN105868343A (en) * | 2016-03-28 | 2016-08-17 | 上海携程商务有限公司 | Database migration method and system |
-
2016
- 2016-11-29 CN CN201611074474.6A patent/CN106776876A/en active Pending
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101706803A (en) * | 2009-11-27 | 2010-05-12 | 天津工业大学 | Method for migrating production data of weaving workshop |
CN103176988A (en) * | 2011-12-21 | 2013-06-26 | 上海博腾信息科技有限公司 | Data migration system based on software-as-a-service (SaaS) |
CN104424283A (en) * | 2013-08-30 | 2015-03-18 | 阿里巴巴集团控股有限公司 | Data migration system and data migration method |
CN104461516A (en) * | 2014-11-21 | 2015-03-25 | 北京锐安科技有限公司 | Data processing method and device |
CN105354314A (en) * | 2015-11-10 | 2016-02-24 | 中国建设银行股份有限公司 | Data migration method and device |
CN105739929A (en) * | 2016-01-29 | 2016-07-06 | 哈尔滨工业大学深圳研究生院 | Data center selection method for big data to migrate to cloud |
CN105740411A (en) * | 2016-01-30 | 2016-07-06 | 武汉大学 | SOA (Service-Oriented Architecture) and WebService based data migration method |
CN105868343A (en) * | 2016-03-28 | 2016-08-17 | 上海携程商务有限公司 | Database migration method and system |
Non-Patent Citations (2)
Title |
---|
朱爱华: ""广电行业台内分布式异构网络化***的集成方法探讨和实践"", 《中国电影电视技市学会影视技术文集》 * |
秦学勇: ""数据仓库中数据集成的研究"", 《农业网络信息》 * |
Cited By (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107506451A (en) * | 2017-08-28 | 2017-12-22 | 泰康保险集团股份有限公司 | abnormal information monitoring method and device for data interaction |
CN107506451B (en) * | 2017-08-28 | 2020-11-03 | 泰康保险集团股份有限公司 | Abnormal information monitoring method and device for data interaction |
CN108647270A (en) * | 2018-04-28 | 2018-10-12 | 尚谷科技(天津)有限公司 | A method of the Data Migration based on fault-tolerant time daily record |
CN108959205A (en) * | 2018-06-27 | 2018-12-07 | 成都泰盟软件有限公司 | A kind of method of ERP system export data fast transferring |
CN109684298A (en) * | 2018-09-07 | 2019-04-26 | 平安科技(深圳)有限公司 | Database migration method, apparatus, equipment and storage medium |
CN109684298B (en) * | 2018-09-07 | 2024-02-20 | 平安科技(深圳)有限公司 | Database migration method, device, equipment and storage medium |
CN109639778A (en) * | 2018-11-29 | 2019-04-16 | 威富通科技有限公司 | A kind of data transmission method, private clound and server |
CN109639468A (en) * | 2018-11-29 | 2019-04-16 | 威富通科技有限公司 | A kind of data mutual-conductance method, public cloud and server |
CN109905384A (en) * | 2019-02-19 | 2019-06-18 | 北京深思数盾科技股份有限公司 | Data migration method and system |
CN109905384B (en) * | 2019-02-19 | 2020-11-20 | 北京纬百科技有限公司 | Data migration method and system |
CN110297813A (en) * | 2019-05-22 | 2019-10-01 | 平安银行股份有限公司 | Data migration method, device, computer equipment and storage medium |
CN111159150A (en) * | 2019-12-19 | 2020-05-15 | 北京文安智能技术股份有限公司 | Data expansion method and device |
CN111597197A (en) * | 2020-06-29 | 2020-08-28 | 腾讯科技(深圳)有限公司 | Data reconciliation method and device between databases, storage medium and electronic equipment |
CN112597128A (en) * | 2020-12-17 | 2021-04-02 | 广东南方通信建设有限公司 | Data migration method and device |
CN112860666A (en) * | 2021-02-08 | 2021-05-28 | 土巴兔集团股份有限公司 | Data migration method and device, computer equipment and storage medium |
CN113064906A (en) * | 2021-04-21 | 2021-07-02 | 杭州天谷信息科技有限公司 | Binlog log data adaptive migration method and system |
CN113239019A (en) * | 2021-06-03 | 2021-08-10 | 锋泾(中国)建材集团有限公司 | Data migration method and system between ERP systems, electronic device and medium |
CN113409015A (en) * | 2021-06-21 | 2021-09-17 | 浙江省能源集团有限公司 | ERP system and use method of ERP system in double-activity deployment of double data centers |
CN116107993A (en) * | 2022-12-26 | 2023-05-12 | 北京万里开源软件有限公司 | Data migration evaluation method and system in MySQL protocol database |
CN116107993B (en) * | 2022-12-26 | 2023-08-29 | 北京万里开源软件有限公司 | Data migration evaluation method and system in MySQL protocol database |
CN116756247A (en) * | 2023-08-21 | 2023-09-15 | 腾讯科技(深圳)有限公司 | Data restoration method, device, computer equipment and storage medium |
CN116756247B (en) * | 2023-08-21 | 2023-11-17 | 腾讯科技(深圳)有限公司 | Data restoration method, device, computer equipment and storage medium |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN106776876A (en) | Data migration method and data mover system | |
US10706221B2 (en) | Method and system operable to facilitate the reporting of information to a report reviewing entity | |
US11144308B2 (en) | System and method for predicting defects in a computer program | |
US20080193008A1 (en) | System and Method for Generating Magnetic Ink Character Recognition (MICR) Testing Documents | |
CN101523413A (en) | Automated generation of form definitions from hard-copy forms | |
CN107220757A (en) | A kind of system and method for rule configuration and parsing | |
CN101504672A (en) | Method and system for dynamically configuring physical data table | |
CN105426419B (en) | Data promotion system and method between heterogeneous system | |
CN106250319A (en) | Static code scanning result treating method and apparatus | |
CN103077019B (en) | A kind of graphical parameter batch maintenance method and device | |
CN110045981A (en) | A method of analytical technology is compared based on source code and establishes demand tracking relationship | |
CN101221564A (en) | Universal independent data acquisition method | |
CN104748757B (en) | A kind of data in navigation electronic map update method and device | |
CN107688593A (en) | Script file generation method and device | |
US20030110175A1 (en) | Deploying predefined data warehouse process models | |
CN103399744B (en) | A kind of method building business gate platform | |
CN104573053B (en) | A kind of configuration item template dynamic customization method based on XML | |
CN108133341A (en) | A kind of generation method and system of the material alternative condition based on ERP system | |
CN109426576B (en) | Fault-tolerant processing method and fault-tolerant assembly | |
KR101120989B1 (en) | Method and system for generating standardized source etl program, recording medium and data-etl-method using the method | |
CN106383778A (en) | Method for improving test case reuse rate | |
Wang et al. | Using workflow patterns to model and validate service requirements | |
CN113538626A (en) | Workflow generation method and device | |
Platonov et al. | Development of a methodology for cost optimization of software testing for the automatically tests generation | |
CN101262657A (en) | Base station single-board upgrade system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170531 |
|
RJ01 | Rejection of invention patent application after publication |