CN107122355A

CN107122355A - Data mover system and method

Info

Publication number: CN107122355A
Application number: CN201610101890.4A
Authority: CN
Inventors: 付大超
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2016-02-24
Filing date: 2016-02-24
Publication date: 2017-09-01
Anticipated expiration: 2036-02-24
Also published as: CN107122355B

Abstract

Data mover system general between isomorphism and heterogeneous database is supported there is provided a kind of this application discloses a kind of data mover system and method, can be in the complete data mover system shut down with full dose migration is carried out under non-stopped status；The full dose verifying function carried out after being migrated there is provided full dose, makes user find the problem of inconsistency in transition process in time；There is provided the increment migration under non-stopped status and increment verifying function, newly-increased data is obtained timely synchronization, and increment is verified in real time, the problem of inconsistency of incremental data is found in time.

Description

Data mover system and method

Technical field

The application belongs to Internet technical field, specifically, is related to a kind of data mover system and method.

Background technology

Typical data mover system is to complete Data Migration by full dose migration or increment migration at present. Such as Oracle GoldenGate, GoldenGate are that a kind of structural data based on daily record replicates soft Part, the increment that it obtains data by parsing the online daily record of source database or archive log changes, then by this A little change applications are to target database, so as to realize that source database is synchronous with target database.GoldenGate Can isomery IT foundation structures (including nearly all conventional operating system platform and database platform) Between realize the real-time duplication of mass data submicrosecond one-level so that can emergency system, online form, Should under multiple scenes such as Real-time Data Warehouse supply, transaction tracking, data syn-chronization, concentration/distribution, disaster tolerance With.GoldenGate Data Integration is implemented main including three processes (data pick-up process, biography It is input into journey and application process).

Online daily record or archive log are read in source database using data pick-up process, is then solved Analysis, the change for only extracting wherein data such as increases, deleted, changing operation, and relevant information is converted to self-defined Intermediate form be stored in queue file, recycle transmission process queue file is passed by TCP/IP It is sent to target database.Data variation of the data pick-up process in daily record is run through every time and in data transmission To after goal systems, it can write a self-criticism a little, the position of the current daily record for completing to extract of record, checkpoint is deposited Data pick-up process can be made to continue replication log from checkpoint location after termination and recovery.Target Database receives data variation and is cached in queue, and queue is a series of text of interim storage data variations Part, waits application process to read data.Application process reads data variation from queue and creates corresponding SQL statement, is performed by the local interface of database, and the inspection of oneself is updated after being submitted to database success Make an inventory of, record has completed the position replicated.

Also some similar databases, the SQL Server Migration Assistant of such as Microsoft are (referred to as SSMA it is) the help client that is issued by Microsoft more easily from Oracle/Sybase/MySQL/Access Move to SQL Server and SQL Azure a tool set.

The SQL Server Migration Assistant of Microsoft are not a kind of general data mover systems, and Oracle GoldenGate also only has increment migration, without full dose migration,

The content of the invention

In view of this, technical problems to be solved in this application there is provided a kind of data mover system and side Method, supports to carry out full dose migration between various isomorphisms and heterogeneous database to solve to lack in the prior art The technical problem of general data mover system.

In order to solve the above-mentioned technical problem, this application discloses a kind of data mover system, including：Entrance (Portal) module, application programming interface (API) module, distributed scheduling module, full dose is moved Shifting formwork block, increment transferring module, full dose correction verification module, increment correction verification module；

The inlet module is used for the entrance for providing management migration task, receives the migration created by client The data to be migrated of task and the first database, the second database and first database specified, institute Stating migration task includes full dose migration, increment migration, full dose verification, increment verification；

The application programming interface module is used to provide service interface to the client, with according to institute State the migration type in migration task and call corresponding full dose transferring module, increment transferring module, full dose school Test module, increment correction verification module；

The distributed scheduling module is used for the execution state for monitoring each migration task, and according to each equipment Present load migration task is scheduled, to realize the load balancing of each equipment；

The full dose transferring module, for by the data to be migrated from first database migration to institute The second database is stated, when first database and the second database isomery, passes through default middle layout Formula by the data type conversion of the data to be migrated be second database data type, according to turn The data to be migrated are write second database by the data type after changing；

The full dose correction verification module, for when the full dose transferring module completes Data Migration, for institute State the data to be migrated that full dose transferring module migrated and judge first database and the second data Whether the data in storehouse are consistent, and inconsistent data are stored in into the presets list；When in described the presets list When there are inconsistent data, duplication check is carried out to the inconsistent data every preset duration, , will be described inconsistent when the inconsistent data still have after the duplication check of preset times Data Ji Lu migration report in；

The increment transferring module, will on the basis of the first moment during for by full dose migration beginning The data to be migrated produce the Data Migration of change to second data after first moment Storehouse, when first database and the second database isomery, by default intermediate form by the production The data type conversion of the data for changing is the data type of second database, after conversion The data for producing change are write second database by data type；

The increment correction verification module；For when the increment transferring module migrated at first moment When the data for producing change afterwards have caught up with the data to be migrated current data variation, it is directed in real time The data that change is produced after first moment judge the number of first database and the second database According to whether consistent, and by inconsistent data Ji Lu the presets lists；Described in existing when described the presets list During inconsistent data, duplication check is carried out to the inconsistent data every preset duration, passed through When the inconsistent data still have after the duplication check of preset times, by the inconsistent number According to record in migration report.

Wherein, the distributed scheduling module is by performing in each equipment in statistics migratory system The quantity and memory usage of blocking carries out the scheduling of migration task, is born with realizing between each equipment Carry balanced.

Wherein, the full dose transferring module or the increment transferring module write to second database During data, the site information of written data is recorded, for breakpoint transmission.

Wherein, the full dose transferring module is further used for, the first database in full dose migration task During with the second database isomorphism, data to be migrated are migrated according to Ku Biaolie current structure.

Wherein, the full dose transferring module is further used for, the first database in full dose migration task During with the second database isomery, for the Data Migration between heterogeneous database, according to default middle layout Formula completes the conversion of data among heterogeneous databases type.

Wherein, the full dose transferring module is further used for, in full dose transition process, if running into first The library name of database, table name, row the name situation inconsistent with library name, table name, the row name of the second database, Then the mapping ruler according to default Ku Biaolie is mapped, and Ku Biaolie data are written into the second data The corresponding Ku Biaolie in storehouse.

Wherein, the full dose correction verification module is further used for, in the first database and the second database isomery When, read the data to be migrated in the first database and be converted to default intermediate form, in described Between form be converted to the type of the second database, then data corresponding with the second database are compared.

Wherein, the increment transferring module is further used for, using the daily record of first database, or The operation note that person is preserved using the trigger of first database was obtained at first moment The data of change are produced afterwards.

Wherein, the increment correction verification module is further used for：By the daily record data for parsing the first database Or the operation note of the trigger of the first database of parsing obtains the data of increment migration.On solving Technical problem is stated, disclosed herein as well is a kind of data migration method, including：

Receive the migration task that is created by client and the first database specified, the second database and described The data to be migrated of first database, wherein, the migration task is migrated including full dose, increment is migrated, Full dose verification, increment verification, first database are in open state；

Call full dose transferring module by the data to be migrated from first database migration to described Two databases, will by default intermediate form when first database and the second database isomery The data type conversion of the data to be migrated is the data type of second database, after conversion Data type the data to be migrated are write into second database；

After by the data to be migrated from first database migration to second database, adjust The data after the migration are verified with full dose correction verification module, to judge first database and Whether the corresponding data of two databases is consistent, and by inconsistent data Ji Lu the presets lists；When described When there are inconsistent data in the presets list, the inconsistent data are weighed every preset duration Resume classes and test, will when the inconsistent data still have after the duplication check Jing Guo preset times In the inconsistent data Ji Lu migration reports；

Increment transferring module is called, will be described on the basis of the first moment for migrating when starting by the full dose Data to be migrated produce the Data Migration of change to second database after first moment, when When first database and the second database isomery, change is produced by described by default intermediate form Data data type conversion be second database data type, according to the data class after conversion The data for producing change are write second database by type；

When the increment transferring module migrated at first moment after produced the data of change When catching up with the current data variation of the data to be migrated, call increment correction verification module in real time for described The data that change is produced after first moment judge that the data of first database and the second database are Data that are no consistent and being migrated for the migration task judge first database and the second number It is whether consistent according to the data in storehouse, by inconsistent data Ji Lu the presets lists；When described the presets list is deposited In the inconsistent data, duplication check is carried out to the inconsistent data every preset duration, , will be described inconsistent when data inconsistent described in the duplication check Jing Guo preset times still have In data Ji Lu migration reports.

Wherein, the full dose transferring module or the increment transferring module are called, by the data to be migrated During from first database migration to second database, further comprise：Record written data Site information, for breakpoint transmission.

Wherein, the full dose transferring module is called, the data to be migrated are write into second database, Further comprise,

When the first database in full dose migration task and the second database isomorphism, data to be migrated are pressed Current structure according to Ku Biaolie is migrated.

Wherein, call full dose transferring module by the data to be migrated from first database migration to institute The second database is stated, is further comprised,

When the first database and the second database isomery in full dose migration task, for heterogeneous database Between Data Migration, completed according to default intermediate form data among heterogeneous databases type turn Change.

In full dose transition process, if running into the library name of the first database, table name, row name and the second data The inconsistent situation of library name, table name, the row name in storehouse, then the mapping ruler according to default Ku Biaolie carry out Ku Biaolie data are written to the corresponding Ku Biaolie of the second database by mapping.

Wherein, call full dose correction verification module to verify the data after the migration, further comprise,

In the first database and the second database isomery, the data to be migrated in the first database of reading are simultaneously Default intermediate form is converted to, by the type that the intermediate format conversion is the second database, then with Corresponding data are compared in second database.

Wherein, increment transferring module is called, is specifically included, using the daily record of first database, or The operation note that person is preserved using the trigger of first database was obtained at first moment The data of change are produced afterwards.

Wherein, increment correction verification module is called, is specifically included：By the daily record data for parsing the first database Or the operation note of the trigger of the first database of parsing obtains the data of increment migration.

Wherein, methods described also includes：When the client to the write operation of first database by When being switched to second database, with the client to second database write data first when The second moment on the basis of, second database is produced to the data of change after second moment Move to first database；

The data that change is produced after being migrated at second moment have caught up with second data During the current data variation in storehouse, judge in real time for producing the data of change after second moment Whether the data of first database and the second database are consistent, and inconsistent data Ji Lu is preset List.

Wherein, methods described also includes：Occur in the client for the access of second database When abnormal, the write operation by the client for second database switches back into first data Storehouse.

Compared with prior art, the application can be obtained including following technique effect：

1) data mover system general between isomorphism and heterogeneous database is supported there is provided a kind of, can In the complete data mover system shut down with full dose migration is carried out under non-stopped status；

2) the structural transfer function before full dose migration is provided, when the data volume for needing to migrate is larger When, substantial amounts of time and manpower can be saved.

3) the full dose verifying function carried out after full dose is migrated is provided, makes user find to migrate in time Problem of inconsistency in journey.

4) increment migration and increment verifying function under non-stopped status are provided, newly-increased data are obtained Verified in real time to timely synchronization, and to increment, the problem of inconsistency of incremental data is found in time.

5) data mover system also has distributed scheduling function, and the data that can lift big data quantity are moved The treatment effeciency of shifting；

6) when carrying out Data Migration under non-stopped status, realized by data backflow between database Data syn-chronization, allows the business of operation to switch at any time.

Certainly, implementing any product of the application must be not necessarily required to while reaching all the above skill Art effect.

Brief description of the drawings

Accompanying drawing described herein is used for providing further understanding of the present application, constitutes one of the application Point, the schematic description and description of the application is used to explain the application, does not constitute to the application's It is improper to limit.In the accompanying drawings：

Fig. 1 is a kind of configuration diagram for data mover system that the embodiment of the present application is provided；

Fig. 1 a are configuration diagrams when full dose transferring module 131 is individually present；

Fig. 1 b are the framework signals in the presence of full dose transferring module 131 and the combination of structural transfer module 135 Figure；

Fig. 1 c are the framework signals in the presence of full dose transferring module 131 and the migration combination of evaluation module 136 Figure；

Fig. 1 d are full dose transferring module 131 and structural transfer module 135 and 136 groups of evaluation module of migration Configuration diagram in the presence of conjunction；

Fig. 1 e are full dose transferring module 131 and structural transfer module 135 and 133 groups of full dose correction verification module Configuration diagram in the presence of conjunction；

Fig. 1 f are that full dose transferring module 131 and migration evaluation module 136 and full dose correction verification module 133 are combined In the presence of configuration diagram；

Fig. 1 g are the framework signals in the presence of full dose transferring module 131 and the combination of full dose correction verification module 133 Figure；

Fig. 1 h are full dose transferring module 131 and structural transfer module and migration evaluation module 136 and full dose Configuration diagram in the presence of the combination of correction verification module 133；

Fig. 1 i are the framework signals in the presence of full dose transferring module 131 and the combination of increment transferring module 132 Figure；

Fig. 1 j are that full dose transferring module 131 and structural transfer module 135 and increment transferring module 132 are combined In the presence of configuration diagram；

Fig. 1 k are full dose transferring module 131 and 132 groups of migration evaluation module 136 and increment transferring module Configuration diagram in the presence of conjunction；

Fig. 1 l are full dose transferring module 131 and structural transfer module 135 and increment transferring module 132 and moved The configuration diagram moved in the presence of evaluation module 136 is combined；

Fig. 1 m are full dose transferring module 131 and structural transfer module 135 and the He of increment transferring module 132 Configuration diagram in the presence of the combination of full dose correction verification module 133；

Fig. 1 n are full dose transferring module 131 and migration evaluation module 136 and the He of increment transferring module 132 Configuration diagram in the presence of the combination of full dose correction verification module 133；

Fig. 1 o are full dose transferring module 131 and full dose correction verification module 133 and 132 groups of increment transferring module Configuration diagram in the presence of conjunction；

Fig. 1 p are full dose transferring module 131 and structural transfer module 135 and the migration He of evaluation module 136 Configuration diagram in the presence of full dose correction verification module 133 and the combination of increment transferring module 132；

Fig. 1 q are the framework signals in the presence of full dose transferring module 131 and the combination of increment transferring module 132 Configuration diagram in the presence of figure and the increasing combination of correction verification module 134；

Fig. 1 r are full dose transferring module 131 and structural transfer module 135 and increment transferring module 132 and increasing Configuration diagram in the presence of the combination of correction verification module 134；

Fig. 1 s are full dose transferring module 131 and migration evaluation module 136 and increment transferring module 132 and increased Configuration diagram in the presence of the combination of correction verification module 134；

Fig. 1 t are full dose transferring module 131 and structural transfer module 135 and increment transferring module 132 and moved Move evaluation module 136 and increase the configuration diagram in the presence of correction verification module 134 is combined；

Fig. 1 u are full dose transferring module 131 and structural transfer module 135 and the He of increment transferring module 132 Configuration diagram in the presence of full dose correction verification module 133 and the increasing combination of correction verification module 134；

Fig. 1 v are full dose transferring module 131 and migration evaluation module 136 and the He of increment transferring module 132 Configuration diagram in the presence of full dose correction verification module 133 and the increasing combination of correction verification module 134；

Fig. 1 w are full dose transferring module 131 and full dose correction verification module 133 and the He of increment transferring module 132 Increase the configuration diagram in the presence of correction verification module 134 is combined；

Fig. 1 x are full dose transferring module 131 and structural transfer module 135 and the migration He of evaluation module 136 Framework in the presence of full dose correction verification module 133 and increment transferring module 132 and the increasing combination of correction verification module 134 Schematic diagram；

Fig. 2 is the state for the state machine that the embodiment of the present application distributed scheduling module is provided for migration task Change schematic diagram；

Fig. 3 is the signal that the embodiment of the present application completes type conversion between heterogeneous database by intermediate form Figure；

Fig. 4 is a kind of schematic flow sheet for data migration method that the embodiment of the present application is provided；

Fig. 5 is a kind of schematic flow sheet for data migration method that the embodiment of the present application is provided；

Fig. 6 is a kind of schematic flow sheet for data migration method that the embodiment of the present application is provided；

Fig. 7 a are a kind of schematic flow sheets for data migration method that the embodiment of the present application is provided；

Fig. 7 b are a kind of schematic flow sheets for data migration method that the embodiment of the present application is provided；

Fig. 8 a are a kind of schematic flow sheets for data migration method that the embodiment of the present application is provided；

Fig. 8 b are a kind of schematic flow sheets for data migration method that the embodiment of the present application is provided；

Fig. 9 is a kind of schematic flow sheet for data migration method that the embodiment of the present application is provided；

Figure 10 is a kind of schematic flow sheet for data migration method that the embodiment of the present application is provided；

Figure 11 is a kind of schematic flow sheet for data migration method that the embodiment of the present application is provided.

Embodiment

Presently filed embodiment is described in detail below in conjunction with drawings and Examples, thereby to the application How application technology means can fully understand to solve technical problem and reach the implementation process of technology effect And implement according to this.

The embodiment of the present application proposes a kind of data mover system and migrated, entirely there is provided full dose migration, increment The migration service of the types such as amount verification, increment verification, migration assessment and structural transfer.Moved using the data Shifting system, can complete Data Migration between isomorphism or heterogeneous database, and to full dose migration Data carry out full dose verification, and increment verification is carried out to the data that increment is migrated.Before Data Migration starts Migration assessment and structural transfer can also be carried out, to analyze the feasibility of migration and improve the effect of Data Migration Rate., can no matter in the case where the business that source database is run allows to shut down or do not allow shutdown Business datum is moved into target database from source database using the data mover system, in source database In the case of not allowing shutdown, it is possible to achieve the data syn-chronization between source database and target database, from And the access for business datum is switched at any time.

Fig. 1 is a kind of configuration diagram for data mover system that the embodiment of the present application is provided, including entrance (Portal) module 10, application programming interface (Application Programming Interface, API) module 11, distributed scheduling module 12, full dose transferring module 131, increment transferring module 132, Full dose correction verification module 133, increment correction verification module 134, structural transfer module 135 and migration evaluation module 136。

Inlet module 10 provides the entrance of management migration task, can be complete by the client of inlet module 10 The configurations such as establishment, startup, management, progress queries into migration task.

Inlet module 10 can be realized by entrance (Portal) software increased income, such as Liferay Portal, Support asynchronous JavaScript and XML (Asynchronous Javascript And XML, AJAX) With Java code requirements (Java Specification Request, JSR) 286 standards, create data and move The management console application program of shifting system, the interaction of the management console application program is shown in client Interface.User can be created in interactive interface migration task and specify source database, target database and Data to be migrated in source database.

Before the management console of data mover system is entered, in addition it is also necessary to create mesh in cloud service console The second database in server is marked, data mover system can create second in destination server automatically Database, the title of the second database automatically created and the title phase of the first database of source server Together, can also self-defined second if user's title for wanting the second database is different from the first database The title of database.Then, the used account of source and target server is created, to ensure This migration task has the reading authority of source server and the write-in authority of destination server.Wherein, source Server and destination server can also be the examples being separately operable on two servers.

Into the management console of data mover system, start to create migration task, first moving for establishment Shifting task sets task names, for example, task names are " abc_ Beijing to Hangzhou ", " bcd_ is local To Qingdao " etc..

After task names are provided with, start to fill in the link information of source and target server, wrap The example types of source server are included, for example, " Self-built Database for having public network IP "；The of source server One type of database, for example, Oracle, Mysql etc.；The Hostname or IP address of source server with And port numbers, such as 201.165.1.112：3308；The account and password of source server, the account need With the digital independent authority for source server；The example types of destination server, for example, relationship type Database service (Relational Database Service, RDS) example；The RDS of destination server Instance identification, such as " rds2afnbt "；The account and password of destination server, the account need to have Data for destination server write authority.

The link information of source and target server is filled in finish after, selection migration and is waited to move type Move data.Verified in interactive interface there is provided full dose migration, increment migration, full dose verification, increment, The options for user selection of the migration types such as structural transfer, migration assessment, migration type can be defaulted as entirely Amount migration, and the other migration types of addition are needed according to user.Data to be migrated can select one or One or more tables inside multiple database collections (Schema) or some set, for example, Data to be migrated could be arranged to database collection " amptest1 ", or database collection " amptest1 " In a table " a00_full_type_table_m1b1 ".If user wants to create in destination server The second database title and the data to be migrated of selection title it is inconsistent, it is necessary to repository at this moment Name mapping relations, for example, the title mapping for the data to be migrated " amptest1 " chosen is configured to “liuy_amptest1”。

After the completion of above-mentioned configuration, start migration task in the management console of data mover system, at interface In it can be seen that the migration task current execution migration type and progress, for example, structural transfer is complete Into 80%, full dose migration has completed 100%.

Client to inlet module 10 initiate HTTP (HyperText Transfer Protocol, Http) ask, inlet module 10 receives and parses through out the migration task created by client, according to migration The open API of task call data mover system.

Application programming interface module 11 is used for the interface that the migration type that can be called is provided to client, Corresponding full dose transferring module 131, increment migration are called with the migration type in the migration task Module 132, full dose correction verification module 133, increment correction verification module 134, structural transfer module 135 and migration One or more of evaluation module 136.

Can be by Http or (the Hyper Text Transfer of the HTTP based on SSL Protocol over Secure Socket Layer, Https) send call request to API module 11.Branch Hold Http Get methods and send request, each request needs to formulate operation to be performed, i.e. Action parameters, For example, creating database, full dose migration, increment migration etc..

Distributed scheduling module 12 is used for the execution state for monitoring each migration task, and according to each equipment Present load migration task is scheduled, to realize the load balancing of each equipment.

Distributed scheduling module 12 provides state machine and is monitored come the running status to each migration task, The operation schematic diagram of state machine is as shown in Fig. 2 state machine is respectively altogether including seven kinds of states：Initialization (init) (running), is run, has been caught up with (catch), paused (paused) has stopped (finished), Failed (failed), succeeds (successful).Transfer process such as Fig. 2 of seven kinds of states of the above It is shown, wherein, the time (time) represents the continuity duration (such as 15 minutes) of increment transition process, Illustrate to have caught up with current data variation when the duration that increment is migrated is less than the preset time, reach (catch) state is caught up with；Pause (pause) represents client selection suspended task；Kill (kill) The task is represented to be killed；Abnormal (unusual) is represented and is captured abnormal conditions, for example, destination service Device example is locked, and causes the data write-in authority of the account of input to be recovered.If the migration of configuration is appointed Business includes full dose migration without being migrated including increment, then from running status at the end of full dose migration task Change turns to success status；If the migration task of configuration includes full dose migration and increment migration, increasing Become at the end of amount migration task from running status and turn to success status.

Distributed scheduling module 12 can be by the unit that is performed in each equipment in statistics migratory system The quantity and memory usage of change carries out the scheduling of migration task, with realize load between each equipment it is equal Weighing apparatus.

Full dose transferring module 131 is used for data to be migrated from the first database migration to the second database. If carrying out full dose migration when the first database is in stopped status, the first database is currently preserved Data Migration to be migrated is to the second database；If the first database is not shut down, task is migrated with full dose On the basis of the first moment when starting to perform, by the Data Migration to be migrated preserved at the first moment to second Database.

First database refers to that the structure of the table of two databases is identical with the second database isomorphism, the first number Refer to that the structure of the table of two databases is different from the second database isomery according to storehouse.

When the first database in full dose migration task and the second database isomorphism, data to be migrated are pressed Current structure according to Ku Biaolie is migrated.When the first database and the second data in full dose migration task During the isomery of storehouse, for the Data Migration between heterogeneous database, completed according to default intermediate form different The conversion of data type between structure database.As shown in figure 3, by the data type conversion of the first database For intermediate form, then from the data type that the intermediate format conversion is the second database.What is so handled is excellent Gesture is, it is not necessary to developing respectively for other kinds of database for each type of database Conversion method.For example, seven kinds of databases shown in Fig. 3, pin is developed if this seven kinds of databases respectively To the conversion method of other kinds of database, 42 kinds of conversion methods are needed altogether；And use this logical The mode that intermediate form is changed is crossed, a kind of read method is developed respectively for seven kinds of databases, by seven The digital independent for planting database is intermediate form, then develops a kind of write-in side respectively for seven kinds of databases Method, the data for being read as intermediate form is written to different databases.As such, it is desirable to develop 14 It is that the type conversion between seven kinds of databases can be achieved to plant conversion method, improves development efficiency, makes difference Data type conversion between the database of type is more prone to realize.

Above-mentioned default intermediate form includes character string (String) array and byte (Byte) of Java language Array.Wherein, when reading data from the first database, by the binary data types of the first database The form for being converted to byte arrays is preserved, and other data types are all converted to the shape of character string dimension Formula is preserved.Preserved when writing data to the second database, then by character string dimension and byte arrays Data be converted to the type corresponding with the row being written into.

For example, to exemplified by Mysql Data Migration, when reading Oracle data, being incited somebody to action by from Oracle Oracle binary large object (Binary Large Object, blob) type, binary_float and Byte (byte) array that the data of the binary floating point type such as binary_double are converted to Java is carried out Preserve, the other kinds of data such as Oracle varchar2, nvarchar2 are converted to Java character The form of string (string) array is preserved, so that the data to be migrated in oracle database be protected Save as intermediate form.It is when the data to be migrated of intermediate form are write into Mysql, blob types is corresponding Byte arrays are converted to Mysql longblob types, by the corresponding byte number of binary_float types Group is converted to Mysql decimal types, is Mysql by binary_double data type conversion Double types, and the corresponding character string dimension of varchar2 types is converted to Mysql varchar Type, the corresponding character string dimension of nvarchar2 types is converted to Mysql national varchar types, The corresponding character string dimension of other types is also converted in Mysql databases corresponding type simultaneously accordingly Write Mysql databases.

As shown in figure 3, Mysql, Oracle, SQLServer, Gao Xing can be realized by the intermediate form Can distributed data base system OceanBase, object --- Relational DBMS PostgreSQL, open architecture data, services OTS, the distributed relational data library management towards row The type conversion of seven kinds of databases such as system Hbase SQLServer, improves development efficiency.

Wherein, full dose transferring module 131 configure a white list, client selection full dose migration as When migrating type, if specifying needs the table of migration in the first database, by the title of specified table It is added to this white list；If necessary to which the first database is integrally migrated, then this white list is It is empty.When full dose migrates beginning, determined to need the table of migration or to the first database according to the white list It is overall to be migrated.

And in one embodiment, cutting can be carried out to the table for needing to migrate before being migrated, The cutting can be realized by existing table segmentation algorithm, it would be desirable to which the table of migration is cut into multiple subtasks point Other asynchronous process, to improve the efficiency of full dose migration.It is, for example, possible to use horizontal cutting, vertical cutting Or hash segmentation algorithm realizes the cutting to table；Wherein, horizontal cutting puts data according to some conditions Into two or more independent tables, i.e., cutting is carried out by record, different records can be retained separately, The columns of each sublist is identical, and can be reduced after cutting needs the page of the data and index read in inquiry Number, while also reducing the number of plies of index, accelerates inquiry velocity；Vertical cutting is major key and some row A table is put into, then major key and some other row are put into another table, original table is divided into many Individual include less row table so that row data diminish, and data block (Block) just can deposit more Data, inquiry when will reduce by I/O number；Cutting is hashed using hash (Hash) algorithm number According to being distributed in each point of table, such I/O is more balanced.

In full dose transition process, if running into the library name of the first database, table name, row name and the second number According to the inconsistent situation of the library name in storehouse, table name, row name, it can enter according to default Ku Biaolie mapping ruler Ku Biaolie data are written to Ku Biaolie corresponding in the second database by row mapping.Opened in migration task Before beginning, when selecting data to be migrated, the inconsistent situation of library name can be configured, herein no longer Repeat specification.And when table name and inconsistent row name, similarly need default table name and row name Mapping ruler.For example：

And in one embodiment, can write every time to the second database after data, record has been write The storage location information of the data to be migrated entered, for breakpoint transmission.

" mapperList=db_1；table_1；*=db_1；table_2；*|db_1；table_2；Usename=db_1；t able_2；uname”.Wherein " | " is used to separate two mapping rulers, db_1；table_1；*=db_1；table_2；* I.e. the title that expression database db_1 table table_1 is mapped as table_2 tables is changed. db_1；table_2；Usename=db_1；table_2；Uname represents database db_1 table table_2 word Section username is mapped as the table_2 uname that is classified as, i.e. column name is also changed accordingly.

And in one embodiment, can write every time to the second database after data, record has been write The site information of the data to be migrated entered, for breakpoint transmission.For example, being created in destination server One table increment_trx, table increment_trx are the site tables created in destination server, mainly For record full dose migration site, solve the problems, such as task abnormity restart after breakpoint transmission.

Structural transfer module 135, for before full dose transferring module 131 carries out Data Migration, will treat Ku Biaolie used in migrating data and object migration are to the second database, when the first database and second is counted During according to storehouse isomery, determine the Ku Biaolie and object in the second database by default Type mapping relation In type, create the Ku Biaolie and object corresponding with the type in the second database.

When needing the data for carrying out full dose migration very many, the process of full dose migration can be more slow, consumption Take excessive time and manpower, such as when being migrated to the data of a few thousand sheets tables of hundreds of databases, Problem above will be more obvious.Therefore, create larger data amount migration task when, can by with Family selects structural transfer in migration type, can also judge migration task institute by data mover system When the data volume of the data to be migrated of configuration exceedes pre-determined threshold, structure is carried out before full dose migration automatically Migration.

For example, the first database in source server is second in oracle database, destination server Database is Mysql databases, and specifies table name in oracle database to be " dbtest " " tabletest " Table includes following each row as data to be migrated, the structure of the table：

("C_ID"NUMBER(20,0),

"C_NUM_1"NUMBER,

"C_NUM_2"NUMBER(38,0),

"C_NUM_3"NUMBER(20,3),

"C_FLOAT_1"BINARY_FLOAT,

"C_FLOAT_2"BINARY_DOUBLE,

"C_DATE_1"DATE,

"C_DATE_2"TIMESTAMP(6),

"C_DATE_3"TIMESTAMP(6),

"C_DATE_4"TIMESTAMP(6)WITH TIME ZONE,

"C_DATE_5"TIMESTAMP(6)WITH LOCAL TIME ZONE,

"C_STRING_1"CHAR(40CHAR),

"C_STRING_2"CHAR(40BYTE),

"C_STRING_3"NCHAR(40),

"C_STRING_4"VARCHAR2(40BYTE),

"C_STRING_5"VARCHAR2(40CHAR),

"C_STRING_6"NVARCHAR2(40),

"C_STRING_7"CLOB,

"C_STRING_8"NCLOB,

"C_STRING_9"LONG,

"C_BLOB_1"BLOB,

"C_BLOB_2"RAW(200)

)。

If have selected structural transfer during user's selection migration type, then structural transfer module 135 is first Corresponding table is created in Mysql databases, if the mapping of user's Non-precondition table name, in Mysql The table `dbtest.`tabletest` with same names is created in database.The table `dbtest.`tabletest` of establishment Also possess corresponding row, the code for creating the table is as follows：

CREATE TABLE`dbtest.`tabletest`(

`C_ID`DECIMAL(20,0)NOT NULL,

`C_NUM_1`BIGINT NOT NULL,

`C_NUM_2`DECIMAL(38,0)NOT NULL,

`C_NUM_3`DECIMAL(20,3)NOT NULL,

`C_FLOAT_1`DECIMAL(65,8),

`C_FLOAT_2`DOUBLE,

`C_DATE_1`DATETIME,

`C_DATE_2`DATETIME,

`C_DATE_3`DATETIME,

`C_DATE_4`DATETIME,

`C_DATE_5`DATETIME,

`C_STRING_1`CHAR(40),

`C_STRING_2`CHAR(40),

`C_STRING_3`VARCHAR(40),

`C_STRING_4`VARCHAR(40),

`C_STRING_5`VARCHAR(40),

`C_STRING_6`VARCHAR(40),

`C_STRING_7`LONGTEXT,

`C_STRING_8`LONGTEXT,

`C_STRING_9`LONG,

`C_BLOB_1`LONGBLOB,

`C_BLOB_2`VARBINARY(200)

) engine=INNODB charset=UTF8COLLATE UTF8_bin；

So as to create corresponding table `dbtest.`tabletest`, the table and Oracle numbers in Mysql databases There are corresponding row according to table " dbtest " " tabletest " in storehouse, so that will before real data is migrated Table " dbtest " " tabletest " to be migrated structure takes the lead in having moved to Mysql databases.In migration During, the type to each row is completed according to the Type mapping relation of " from Oracle to Mysql " Oracle row " C_ID " type is Number in conversion, such as upper example, in Mysql databases wound In the table built, corresponding row " C_ID " type is Decimal, and other row are also changed accordingly, It is numerous to list herein.

User creates migration task, when selecting data to be migrated except the mapping of library name can be set to close System, can also set the mapping relations of table name and row name.When have selected the table in the first database as treating During migrating data, table name of the table in the second database can be preset.The table is being set in the second data In the interface of table name in storehouse, all row inside the table are shown, user, which can be set, each is listed in second Row name in the table of database.

For example, user have selected table CHINESE_KEY_TABLE in the first database as waiting to move Data are moved, an edit page can be opened by choosing after the table, table correspondence in the second database is set Table title CHINESE_KEY_TABLE_TEST；In the interface, while displaying is to be migrated Table CHINESE_KEY_TABLE in each arrange：C, ID, K, PAD etc. are arranged, and user Editor can be re-started to the row name of each row, for example by the row of each row be revised as C_TEST, ID_TEST、K_TEST、PAD_TEST。

Evaluation module 136 is migrated, the progress for treating migrating data in full dose transferring module 131 is migrated Before, or treat Ku Biaolie used in migrating data in structural transfer module 135 and object is moved Before shifting, assessment report is exported according to the first database and the second database, wherein, assessment report includes The space hold information of first database, Ku Biaolie and object information, the SQL of execution and processing Performance information, incompatible characteristic information and improvement cost letter of the assessment report also including the second database Breath.

For example, choosing database collection amptest1 and amptest2 as to be migrated from oracle database Data, move to the corresponding database collection liuy_amptest1 and liuy_amptest2 of Mysql databases. So before full dose migration or structural transfer is started, assessment report is exported, assessment report content includes： Oracle database current spatial takes 500GB, free space 1.5TB, including 200 storehouses, 5000 The average columns for opening table and every table is 35, supports the where conditions of SQL standard, current internal memory is accounted for With 25%；This migration amptest1 and amptest2 two storehouse, has 55 tables, every table it is flat Equal columns is 30, and target Mysql databases liuy_amptest1 and liuy_amptest2 do not support Oracle In bfile, interval year to month, interval day to second this several data type, move If containing these three types during shifting in amptest1 and amptest2 table, structural transfer can be caused to fail, Please according to business demand, by the corresponding data of these three types be previously written liuy_amptest1 and Liuy_amptest2, the transformation time takes around 24 hours.

Full dose correction verification module 133, for when full dose transferring module 131 completes Data Migration, for complete The data to be migrated that amount transferring module 131 is migrated judge that the data of the first database and the second database are It is no consistent, and inconsistent data are stored in the presets list；When there is inconsistent number in the presets list According to when, duplication check is carried out to inconsistent data every preset duration, in the repetition Jing Guo preset times When inconsistent data still have after verification, inconsistent data Ji Lu is migrated into assessment report It is interior.

Full dose correction verification module 133 can equally determine to need to carry out according to the white list of full dose transferring module 131 The table of full dose verification, when whole database collection is carried out into full dose migration, white list is sky, then to whole Data in individual database collection are verified.And in the first database and the second database isomery, together Sample needs first to read data to be migrated and is converted to default intermediate form, is by the intermediate format conversion The type of second database, then data corresponding with the second database are compared.

The first database be in shut down and non-stopped status under, complete full dose migration after may continue into Row full dose is verified.And under non-stopped status, enter on the basis of being the first moment migrated by full dose when starting Row Data Migration, during full dose is migrated, there may be new write operation to have modified data to be migrated, Make the database of write-in second data and modification after data to be migrated it is inconsistent.

For example, table " dbtest " " tabletest " data full dose moves to Mysql's in oracle database After table `dbtest.`tabletest`, corresponding row " C_DATE_1 " and `C_DATE_1` date value differ Cause, it may be possible to the time is changed in transition process, then by it is corresponding row " C_DATE_1 " with `C_DATE_1` is recorded in the presets list, is once repeated every preset duration (such as 3 seconds) Verify, if the inconsistent data of the duplication check by preset times (for example, 20 times) still have, Then record in migration report.Token record is from table " dbtest " " tabletest " to table in migration report The migration of `dbtest.`tabletest` full doses has been completed, but there are inconsistent data：Corresponding row " C_DATE_1 " and `C_DATE_1`.

Increment transferring module 132, on the basis of the first moment during for by full dose migration beginning, will wait to move Move data and the Data Migration of change produced after the first moment to the second database, when the first database with During the second database isomery, the data type conversion of the data of change will be produced by default intermediate form For the data type of the second database, the data write-in the of change will be produced according to the data type after conversion Two databases.

Increment migration is proceeded by the end of full dose is migrated, can be by parsing daily record data or parsing The operation note of trigger is realized, is illustrated separately below.

When being realized by parsing daily record data, increment transferring module 132 reads the daily record of the first database, For example, reading binary log (binlog) when the first database is Mysql, read when being Oracle Take and (redo log) redolog.

The log content read is parsed and the default intermediate form (word of Java language is converted into Joint number group and character string dimension) and queue (Queue) is write, read the number of the intermediate form in queue According to.The data for needing to read, the white name that the white list is created when being full dose migration are determined by white list It is single.Such as, white list includes table " dbtest " " tabletest ", then redoes log middle filtering from Oracle Go out to be directed to the data of this table, complete to write Mysql table after the operations such as type conversion `dbtest.`tabletest`.Data are being read from queue and during being written to the second database, Ke Yitong Cross the concurrent operations of multithreading to complete, to improve increment transport efficiency.By the side for parsing daily record data Businesslike write-in is supported in the increment migration that formula is realized, so-called transactional refers to as single logic working The sequence of operations that unit is performed, otherwise it is completely executed, otherwise do not perform fully.Issued transaction can To ensure, except all operations in non-transactional unit are all successfully completed, otherwise will not forever update towards number According to resource.One logical unit of work will turn into affairs, it is necessary to meet so-called ACID (atomicity, Uniformity, isolation and persistence) attribute.Affairs are the operating logicals unit of work (LUW) of database, By transaction management in data base management system (Database Management System, DBMS) System is responsible for the processing of affairs.Write that is, client is submitted in the first database with affairs once To the data in the first database, increment is also to be written to the second database with an affairs when migrating.This Most important, remittance of for example transferring accounts, goods during shopping at network for the application that can not receive data intermediateness Inventory information of thing etc., user client transfer operation do not complete or shopping process not at the end of, though Some data in right table have changed, but not carry out increment migration, until the behaviour that transfers accounts or do shopping Make to carry out increment migration to all data changed after terminating.

When being realized by trigger, increment transferring module 132 is remembered by the trigger function of the first database Record the operation of data manipulation language (Data Manipulation Language, DML) after the first moment And be written in increment list；The information and the anti-corresponding data looked into the first database in increment list are read, The data inquired are converted into default intermediate form (byte arrays and character string dimension of Java language) And be written in queue；Read and from queue by being written to the second database after type conversion process. Wherein, what increment list was preserved is major key information, is not initial data, it is therefore desirable to anti-according to major key information Discover and seize to obtain initial data.For example, after full dose migration starts, client writes the table of oracle database " dbtest " " tabletest ", trigger records major key information id=1 in increment list, after increment migration starts, It is anti-when reading increment list major key information id=1 to find corresponding write-in data

" C_STRING_1 "=fh2aje.

Increment correction verification module 134, for when increment transferring module 132 migrated at the first moment after When the data for producing change have caught up with data to be migrated current data variation, it was directed to for the first moment in real time The data for producing change afterwards judge whether the data of the first database and the second database are consistent, and will not Consistent data Ji Lu the presets lists；When the presets list has inconsistent data, when default Length carries out duplication check to inconsistent data, inconsistent after the duplication check Jing Guo preset times When data still have, by inconsistent data Ji Lu migration reports.

For example, carrying out Data Migration, full dose migration to table " dbtest " " tabletest " of oracle database After the first moment during beginning, four write operations are occurred in that again for table " dbtest " " tabletest " is current, So this four write operations are write Data Migration to Mysql table `dbtest.`tabletest` by increment migration When, described " having caught up with (Catch) " state is reached, " producing the equalizer " state is referred to as.Now, Start to carry out increment verification to the data that increment is migrated.

Increment verification also accordingly can by way of parsing daily record data or trigger mode come Realize.

When being realized by way of parsing daily record data, the daily record data and solution of the first database are read first Analysis, and the data corresponding to the involved storehouse of increment migration or table are filtered out by white list.Inquire phase The storehouse answered or table produce the row of data variation after the first moment；According to Ku Biaolie mapping relations, look into Ask Ku Biaolie corresponding to the second database；The data that change is produced after the first moment are passed through into type Conversion data corresponding with the second database are contrasted, and the presets list is write if inconsistent.

For example, white list includes table " dbtest " " tabletest " of oracle database, from Oracle data Storehouse redo log it is middle read for table " dbtest " " tabletest " data, and inquire the first moment it After have four write operations, including for row " C_STRING_1 " twice, for row " C_ID " one It is secondary, for row " C_DATE_1 " once.Read the row in table " dbtest " " tabletest " " C_STRING_1 ", " C_ID " and " C_DATE_1 " current data.Closed according to Ku Biaolie mapping System, row `C_STRING_1`, the `C_ID` inquired about in the table `dbtest.`tabletest` of Mysql databases With `C_DATE_1` current data.Will row " C_STRING_1 ", " C_ID " Current data with " C_DATE_1 " is after type is changed, with row `C_STRING_1`, `C_ID` It is compared with `C_DATE_1` current data, by inconsistent record in the presets list.

When being realized by way of trigger, the information in the first database increment list is read first, according to The major key information of increment list record is counter to look into corresponding real data in the first database, further according to white list mistake The real data for belonging to the table for having carried out increment migration is filtered out, the real data in the first database is passed through Ku Biaolie is mapped and type conversion from the corresponding data of the second database with being contrasted, if inconsistent Write default data list.

For example, increment list include in major key id=1 to 20, the anti-data found with white list Table " dbtest " " tabletest " of oracle database it is relevant have four, including for row " C_STRING_1's " has two, has one for row " C_ID ", for row " C_DATE_1 " There is one.According to Ku Biaolie mapping relations, in the table `dbtest.`tabletest` for reading Mysql databases Row `C_STRING_1`, `C_ID` and `C_DATE_1` current data.By the anti-Oracle found Table " dbtest " " tabletest " of database row " C_STRING_1 ", " C_ID " and " C_DATE_1 " Data carry out row `C_STRING_1`, `C_ID` in type conversion, with table `dbtest.`tabletest` Compare with `C_DATE_1` current data, by inconsistent write-in the presets list.

Increment verification can carry out multiple reinspection for inconsistent data.Due to being write to the second database The process of data is possible to that delay occurs, therefore increment correction verification module 134 is recorded not to the presets list Consistent data carry out a duplication check every preset duration (such as 3 seconds), if by default time The data that the duplication check of number (such as 20 times) is inconsistent still have, then record in migration report. The migration report in token record from table " dbtest " " tabletest " to table `dbtest.`tabletest` full dose migration with Increment migration has been completed, but there are inconsistent data, for example, corresponding row " C_DATE_1 " with `C_DATE_1`。

It should be noted that in the embodiment of the present application, the full dose transferring module 131, the increment is moved Shifting formwork block 132, the full dose correction verification module 133, the increment correction verification module 134, the structural transfer In module 135 and the migration evaluation module 136, part of module, which works independently, can realize specifically Function, at the same time, can carry out the combination between multiple modules, so as to be formed according to the selection of user System with difference in functionality.For example, the full dose transferring module 131 is as can be real during standalone module Distributed data migration is carried out to the multiple appearances in multiple storehouses existing multithreading；The full dose transferring module 131 with the migration evaluation module 136 when being combined, and can be taken a step forward carrying out full dose migration First database and the second database are estimated, so as to obtain the base of source database and purpose database This situation, realizes that one is estimated to full dose migration；The full dose transferring module 131 can also be with the structure Transferring module 135 is combined, when first database and the second database isomery, and data are moved When shifting amount is larger, structural transfer is carried out in advance can improve the efficiency of data full dose migration, save the time.

The full dose correction verification module 133 depends on the full dose transferring module 131 and existed, can be with The verification of source storehouse and purpose storehouse data consistency is carried out after data carry out full dose migration.Above-mentioned several feelings Condition is that the full dose correction verification module 133 assesses mould with the structural transfer module 135, the migration respectively The combination function explanation of block 136, the full dose correction verification module 133, the full dose correction verification module 133 is also Evaluation module 136, the full dose correction verification module 133, the structural transfer can be migrated with described simultaneously Any two or all three modules are combined in module 135, and eight kinds of optional combinations are obtained. This eight kinds of combinations as shown in Fig. 1 a~Fig. 1 h respectively

Wherein, 1a is configuration diagram when full dose transferring module 131 is individually present；Fig. 1 b are full doses Configuration diagram in the presence of transferring module 131 and the combination of structural transfer module 135；Fig. 1 c are full doses Configuration diagram in the presence of transferring module 131 and the migration combination of evaluation module 136；Fig. 1 d are full doses Framework in the presence of transferring module 131 and structural transfer module 135 and migration evaluation module 136 are combined shows It is intended to；Fig. 1 e are full dose transferring module 131 and structural transfer module 135 and full dose correction verification module 133 Configuration diagram in the presence of combination；Fig. 1 f are full dose transferring module 131 and migration evaluation module 136 Configuration diagram in the presence of being combined with full dose correction verification module 133；Fig. 1 g are full dose transferring modules 131 Configuration diagram in the presence of being combined with full dose correction verification module 133；Fig. 1 h are full dose transferring modules 131 Framework in the presence of being combined with structural transfer module and migration evaluation module 136 and full dose correction verification module 133 Schematic diagram；

Above-mentioned each block combiner is in use, each functions of modules is as described in the corresponding embodiments of Fig. 1, herein Repeat no more.

In addition, the increment transferring module 132 of the embodiment of the present application is also to depend on the full dose migration Module 131 and exist, increment migration generally full dose migration complete after operate.With reference to above-mentioned full dose The implementation of transferring module 131, it should be understood that the increment transferring module 132 is realizing it positioned at number During according to function in migratory system, there are following several implementations：The increment transferring module 132 and institute State full dose transferring module 131 combine, the increment transferring module 132 and the full dose transferring module 131 Combined with the full dose correction verification module 133, the increment transferring module 132 and the full dose transferring module 131 combine with the increasing correction verification module 134, the increment transferring module 132 migrates mould with the full dose Block 131 is combined with the increasing correction verification module 134 with the full dose correction verification module 133, wherein, above-mentioned four Plant in combination, every kind of combination can be commented with the structural transfer module 135 or the migration again Estimate one of module in module 136 or all two modules be combined, be obtained 16 kinds it is optional Combination.As shown in Fig. 1 i~Fig. 1 x：

Fig. 1 i are the framework signals in the presence of full dose transferring module 131 and the combination of increment transferring module 132 Figure；Fig. 1 j are that full dose transferring module 131 and structural transfer module 135 and increment transferring module 132 are combined In the presence of configuration diagram；Fig. 1 k are full dose transferring module 131 and migration evaluation module 136 and increased Measure the configuration diagram in the presence of transferring module 132 is combined；Fig. 1 l are full dose transferring module 131 and structure Framework in the presence of transferring module 135 and increment transferring module 132 and migration evaluation module 136 are combined shows It is intended to；Fig. 1 m are full dose transferring module 131 and structural transfer module 135 and increment transferring module 132 Configuration diagram in the presence of being combined with full dose correction verification module 133；Fig. 1 n are full dose transferring modules 131 In the presence of being combined with migration evaluation module 136 and increment transferring module 132 and full dose correction verification module 133 Configuration diagram；Fig. 1 o are full dose transferring module 131 and full dose correction verification module 133 and increment migration mould Configuration diagram in the presence of the combination of block 132；Fig. 1 p are full dose transferring module 131 and structural transfer mould Block 135 and migration evaluation module 136 and full dose correction verification module 133 and the combination of increment transferring module 132 are deposited When configuration diagram；Fig. 1 q are that full dose transferring module 131 and the combination of increment transferring module 132 are deposited When configuration diagram and increase correction verification module 134 combine in the presence of configuration diagram；Fig. 1 r are full doses Transferring module 131 and structural transfer module 135 and increment transferring module 132 and 134 groups of correction verification module of increasing Configuration diagram in the presence of conjunction；Fig. 1 s are full dose transferring module 131 and migration evaluation module 136 and increased Measure transferring module 132 and increase the configuration diagram in the presence of correction verification module 134 is combined；Fig. 1 t are that full dose is moved Shifting formwork block 131 and structural transfer module 135 and increment transferring module 132 and the migration He of evaluation module 136 Increase the configuration diagram in the presence of correction verification module 134 is combined；Fig. 1 u are full dose transferring module 131 and knot Structure transferring module 135 and increment transferring module 132 and full dose correction verification module 133 and increasing correction verification module 134 Configuration diagram in the presence of combination；Fig. 1 v are full dose transferring module 131 and migration evaluation module 136 Frame in the presence of being combined with increment transferring module 132 and full dose correction verification module 133 and increasing correction verification module 134 Structure schematic diagram；Fig. 1 w are full dose transferring module 131 and full dose correction verification module 133 and increment transferring module 132 and increase correction verification module 134 combine in the presence of configuration diagram；Fig. 1 x are full dose transferring modules 131 With structural transfer module 135 and migration evaluation module 136 and full dose correction verification module 133 and increment migration mould Configuration diagram in the presence of block 132 and the increasing combination of correction verification module 134；Above-mentioned each block combiner in use, Each functions of modules is as described in the corresponding embodiments of Fig. 1, and here is omitted.

The data mover system that the embodiment of the present application is provided, using the teaching of the invention it is possible to provide full dose migration, increment are migrated, entirely The migration service of the types such as amount verification, increment verification, migration assessment and structural transfer, and by user's root Above-mentioned migration type is selected according to demand.Number general between isomorphism and heterogeneous database is supported there is provided a kind of , can be in the complete Data Migration system for shutting down with full dose migration being carried out under non-stopped status according to migratory system System；There is provided the structural transfer function before full dose is migrated, when the data volume for needing to migrate is larger, Substantial amounts of time and manpower can be saved.The full dose verifying function carried out after being migrated there is provided full dose, makes User has found the problem of inconsistency in transition process in time.Moved there is provided the increment under non-stopped status Move and increment verifying function, newly-increased data is obtained timely synchronization, and increment is verified in real time, The problem of inconsistency of incremental data is found in time.Data mover system also has distributed scheduling function, The treatment effeciency of the Data Migration of big data quantity can be lifted.

The data mover system provided using the application, the business run in source database allows the feelings shut down Under condition, full dose migration can be carried out；Further, full dose verification is carried out after full dose migration；Enter one Step, carry out migration assessment and structural transfer before full dose migration according to actual conditions.With by Oracle Database collection amptest1 in database is moved to exemplified by the liuy_amptest1 of Mysql databases, Process is as shown in figure 4, comprise the following steps.

In step S210, the migration task of establishment is received, is moved to from amptest1 Liuy_amptest1, migration type includes migration assessment, structural transfer, full dose migration and full dose verification, Wherein Oracle storehouse amptest1 is in stopped status.

In step S220, this migration task is estimated, assessment report is exported.

Wherein, the assessment report includes amptest1 space hold information, Ku Biaolie and object information, The SQL and process performance information of execution, assessment report also include liuy_amptest1 incompatible spy Property information and improvement cost information.

In step S230, when receiving the instruction for continuing executing with this migration task, by amptest1 In table be created to liuy_amptest1, and be that every table creates corresponding row by Type mapping relation.

Optionally, step S230 further comprises S231, in step S231, according to what is be pre-configured with White list determines the table to be migrated in amptest1；Wherein, the white list is to be moved the need for user specifies The list of the table of shifting, if specifying needs the table of migration in oracle database, by specified table Title is added to this white list；The first database is integrally migrated if desired, then this white name Single is sky.

Optionally, step S230 further comprises S232, if needs are determined according to the white list The table of migration, then in step S232, cutting is carried out to the table for needing to migrate.

This step is not intended to limit specific segmentation algorithm, can be horizontal cutting, vertical cutting, hash and cut Divide any one of algorithm scheduling algorithm.Pass through segmentation algorithm, it would be desirable to which the table of migration is cut into many height and appointed Business difference asynchronous process, to improve the efficiency of full dose migration.

In step S240, the total data in amptest1 is moved into correspondence in liuy_amptest1 Table and row.Optionally, the data migration process of the present embodiment also includes step S241, in step S241 In, write every time into liuy_amptest1 after data, record the storage location information of data to be migrated, To breakpoint transmission.

Optionally, the data migration process of the present embodiment also includes step S242：Every time to Write in liuy_amptest1 after data, the site information of data to be migrated is recorded, to breakpoint transmission.

Specifically, a site table increment_trx can be created in destination server Mysql, use In the site of record full dose migration.If book server exception is restarted, site table can effectively solve task Breakpoint transmission problem after abnormal restarting.

In step s 250, amptest1 is compared with corresponding table in liuy_amptest1 and column data It is right.

The process compared described in this step is migrated to the number of purpose database to judge by source database It is reported that it is no consistent, if there are inconsistent data, then inconsistent data are stored in default list In.Wherein, when there are inconsistent data in the presets list, every preset duration to described inconsistent Data carry out duplication check, inconsistent data are still deposited after the duplication check Jing Guo preset times When, inconsistent data Ji Lu is migrated in assessment report, so as to find full dose migration in time In data inconsistence problems, realize the real-time verification of Data Migration, further ensured Data Migration Accuracy.

Specifically, the table for needing to carry out full dose verification is determined according to the white list being pre-configured with, as general When whole database collection carries out full dose migration, white list is sky, then to the number in whole database collection According to being verified.Especially, treated when when the first database is with the second database isomery, it is necessary to first read Migrating data is simultaneously converted to default intermediate form, passes through the class that the intermediate format conversion is the second database Type, then data corresponding with the second database are compared.For example, the Oracle numbers in the present embodiment According in storehouse with Mysql databases be heterogeneous database, then need first read oracle database in waiting move Move data and be translated into default intermediate form, be then converted to the corresponding data of Mysql databases Type, then data corresponding with Mysql databases are compared, so as to realize the migration of data full dose Verification in real time.

With reference to Fig. 5, in a kind of optional embodiment, the full dose transition process of the embodiment of the present application enters One step can be realized by following steps：

In step S501, determine that the table of needs migration or judgement are according to the white list being pre-configured with It is no that first database is integrally migrated.

In step S502, cutting is carried out to the table for needing to migrate and obtains multiple subtasks.

Big quantity table is cut into multiple subtasks and handled by this step, and for each subtask, point Cloth scheduler module can be dispatched accordingly, to realize load balancing, while improving the effect of Data Migration Rate.

In step S503, the first storehouse data that the distribution of scheduling system is read in each subtask are carried out Processing.

In step S504, when the first database and the second database isomery, by read first Storehouse data are converted into default intermediate form.

In step S505, by default Type mapping relation determine the first database Ku Biaolie and Type of the object to be migrated in the second database.

In step S506, the Ku Biaolie corresponding with the Ku Biaolie types is created in the second database And object.

In step s 507, by data to be migrated from the first database migration to the second database.

In step S508, write every time into the second database after data, record data to be migrated Site information, to breakpoint transmission.The data mover system provided using the application, in source database In the case that the business of operation does not allow shutdown, full dose migration and increment migration can be carried out；Further, Full dose verification is carried out after full dose migration, increment verification is carried out after increment migration；Further, According to actual conditions migration assessment and structural transfer are carried out before full dose migration.Still with by Oracle numbers Exemplified by the liuy_amptest1 of Mysql databases being moved to according to the database collection amptest1 in storehouse, mistake Journey is as shown in fig. 6, comprise the following steps.

In step s 601, the migration task of establishment is received, is moved to from amptest1 Liuy_amptest1, migration type is migrated including migration assessment, structural transfer, full dose, full dose is verified, Increment is migrated and increment verification, and wherein Oracle storehouse amptest1 is in non-stopped status.

In step S602, this migration task is estimated, assessment report is exported.

In step S603, when receiving the instruction for continuing executing with this migration task, by amptest1 Middle table is created to liuy_amptest1, and is that every table creates corresponding row by Type mapping relation.

In step s 604, the total data in amptest1 is moved into correspondence in liuy_amptest1 Table and row.

In step s 605, amptest1 is compared with corresponding table in liuy_amptest1 and column data It is right.

In step S606, the data changed after starting in full dose migration in amptest1 are moved Move on to liuy_amptest1 corresponding table and row.

In step S607, when increment migration reaches " having caught up with " state, in real time by amptest1 The data migrated with having carried out increment in liuy_amptest1 are compared.

In step S608, the access for amptest1 is switched to liuy_amptest1.

In step S609, the second moment of data is write to liuy_amptest1 first after switching Start, start reverse increment migration, by the Data Migration changed in liuy_amptest1 to amptest1 Corresponding table and row.

In step S610, when the reverse increment migration since the second moment reaches " having caught up with " shape During state, the data that reverse increment migration has been carried out in amptest1 and liuy_amptest1 are carried out in real time Compare.

In step s 611, when liuy_amptest1 appearance is abnormal, liuy_amptest1 will be directed to Access switch back into amptest1.

In the present embodiment, full dose migration and increment migration are completed under the non-stop-machine state of database, and in fact When full dose verification and increment carried out to the data after migration verified, with one after real-time verification Data Migration Cause property.The amount of storage that the reason for carrying out Data Migration generally includes the first database is excessive, it is necessary to migrate portion Divided data to the second database, or, partial data visit capacity is excessive in the first database, have impact on it The normal access of his data is, it is necessary to by the excessive Data Migration of visit capacity to the second database.Therefore, test Demonstrate,prove out the data after migration it is completely the same when, just can by client for data to be migrated access from the One database is switched to the second database.

And after the handover, because there is program error (BUG) or second in operation in application program Database processing ability goes wrong, it is possible to occurs that needs will be accessed and switches back into the first database again Situation.In order that data to be migrated can be consistent in real time in two databases.Therefore, wait to move The change that data are moved in the second database also will be back to the first number with reverse increment migration pattern in real time According to storehouse, and data to backflow in the way of increment is verified are verified in real time, and above-mentioned Data Migration may be used also To further comprise the steps.

For example, write operation is switched to after liuy_amptest1, write operation have modified table first The data of row " id_type " in " a00_full_type_table_m1b1 ", the then at the time of write operation As the second moment.Now start the reverse increment migration from liuy_amptest1 to amptest1.

25 seconds after the second moment, it have modified table " a00_full_type_table_m1b1 " totally four times Interior data, when the data changed for this four times are all migrated to amptest1, the pin come into effect Increment verification to above-mentioned reverse increment migration, to verify current amptest1 and liuy_amptest1 number According to whether being consistent.

, can be real-time due to having been carried out reverse increment migration when liuy_amptest1 occurs abnormal Write operation etc. is accessed and switches back into amptest1 again.

The non-stop-machine data migration method that the present embodiment is provided, is carried out real-time between two databases Increment is migrated and increment verification, is consistent two databases, the access for corresponding data can be Switch in real time between two databases.

Specifically, with reference to Fig. 7 a and 7b, in a kind of optional embodiment, in the embodiment of the present application, Increment transition process based on log mode can further be realized by following steps：

In step s 701：The daily record read and parse the first database obtains incremental data.

Generally, increment migration can be pre-created the resolution path of increment task, for example, the embodiment of the present invention Topic store can be used to identify a data parsing passage.

First database is that the log recording in source storehouse is incremental data, passes through the day for parsing first database Will can obtain the increased data of source storehouse data.

In step S702：Determine to need the table of migration in incremental data by white list.

Increment reach is carried out after full dose migration terminates, after incremental data is got from topic store, First pass through white list to filter incremental data, so as to obtain table to be migrated.Wherein, the white name The white list that list is pre-configured with when being migrated with full dose is identical, is specified by client.

In step S703：By data conversion to be migrated is into default intermediate form and writes queue Queue。

In step S704, the digital independent number of intermediate form is read from queue Queue multithreading According to and carry out the mapping of storehouse list.

In step S 705, data to be migrated are write into the second database, and record data to be migrated Site information, to breakpoint transmission.

Further, when the Major key of tables of data has renewal (update) or deletion action (delete), In step S706, according to the renewal result of the first database, write data into process and further split To write and deleting.

For example, the Major key for working as a certain tables of data in the first database fades to 2 by 1, then accordingly, It is corresponding with the table that the tables of data that the first database key value is 1 has mapping relations in second database Carry out deletion action (delete), at the same time, correspondingly in the second database with the first database key There is the write operation of the table progress data of mapping relations in the tables of data being worth for 2.Certainly, described above is only For citing, the embodiment of the present application is not construed as limiting.

With reference to Fig. 8, in a kind of optional embodiment, in the embodiment of the present application, based on trigger side The increment transition process of formula can further be realized by following steps：

In step S801, increment list and trigger (trigger) are created first.

In step S802, the information in the first database increment list is read, is recorded according to increment list Major key information is counter to look into corresponding initial data in the first database.

In step S803, the real data of the table of increment migration is filtered out according to white list.

In step S804, by data conversion to be migrated is into default intermediate form and carries out storehouse list Mapping.

In step S805, data to be migrated are write into the second database, and record has been handled Increment site information, to breakpoint transmission.

, wherein it is desired to simply major key information rather than the initial data that are preserved in explanation, increment list, because This writes data to be migrated before second database, need to be looked into according to the information in the increment list of acquisition is counter Source storehouse, i.e. the first database, so as to obtain initial data to be migrated, and the initial data are write Second database.

Further, if in the first database, the Major key of tables of data has renewal (update) or deletion During operation (delete), in step S806, according to the renewal result of the first database, data are write Enter process to be further broken into writing and delete.

The data mover system provided using the application, carries out full dose verification, specifically after increment migration Ground, the implementation process of full dose verification is as shown in figure 9, comprise the following steps：

In step S901, the table information of the first database is read, and according to the white list being pre-configured with Obtain the table information that need to be verified；

In step S902, the data in the first database are read, and the data read are converted into Default intermediate form simultaneously carries out storehouse table mapping；

In step S903, mapped, inquired about in the second database by the first data according to the storehouse table The data that storehouse migrates and come；

, wherein it is desired to which explanation, step S902 and step S903 are parallel steps, the application is real Example is applied not limit its order.

In step S904, the data obtained in step S902 and step S903 are mapped according to storehouse table Relation is contrasted, if result is consistent, is performed step S905, is otherwise performed step S906.

In step S905, terminate checking procedure.

In step S906, inconsistent data are stored in the presets list.

When there are inconsistent data in the presets list, inconsistent data are carried out every preset duration Duplication check, will not when data inconsistent after the duplication check Jing Guo preset times still have In consistent data Ji Lu migration assessment reports.

The data mover system provided using the application, increment verification is carried out after increment migration.Wherein, The implementation process of increment verification based on daily record as shown in Figure 10, comprises the following steps：

In step S1001, read the daily record of the first database and the content of the daily record is parsed；

In step S1002, the first database is inquired about according to the result of the parsing, is obtained Incremental data to be migrated in one data.

In the step s 1003, the filtering of white list is carried out to the incremental data inquired in the first database Obtain the storehouse table and data increased involved by migration and be converted into default intermediate form.

In step S1004, the result queries second for carrying out storehouse table mapping and being mapped according to the storehouse table are counted According to the storehouse table and data increased in storehouse involved by migration.

In step S1005, according to storehouse table mapping relations, it will be obtained in step S1003 and step S1004 The storehouse table or data that the increment got moves forward involved are contrasted, if unanimously, performing step S1006, if it has not, then performing step S1007.

In step S1006, the site information of increment verification is preserved.

In step S1007, inconsistent data are stored in the presets list, and jump to step S1006。

Specifically, in the embodiment of the present application, increment verification implementation process such as Figure 11 institutes based on trigger Show, comprise the following steps：

In step S1101, the increment list of the first database is read；

In step S1102, the first database is inquired about according to the increment list, the first number is obtained According to the middle storehouse table and data increased involved by migration.

In step S1103, the incremental data that is inquired in the first database is carried out white list filtering, And it is converted into default intermediate form.

In step S1104, the result queries second for carrying out storehouse table mapping and being mapped according to the storehouse table are counted According to the storehouse table and data increased in storehouse involved by migration.

In step S1105, according to storehouse table mapping relations, it will be obtained in step S1103 and step S1104 The data got are contrasted, if unanimously, performing step S1106, if it has not, then performing step S1107.

In step S1106, the site information of increment verification is preserved.

In step S1107, inconsistent data are stored in the presets list, and jump to step S1106。

In the present embodiment, verified, can found in time in data migration process by full dose verification and increment The problem of data of appearance are inconsistent, particularly with the data of core application, it is ensured that its migrating data Integrality and security.

In a typical configuration, computing device include one or more processors (CPU), input/ Output interface, network interface and internal memory.

Internal memory potentially includes the volatile memory in computer-readable medium, random access memory And/or the form, such as read-only storage (ROM) or flash memory (flash RAM) such as Nonvolatile memory (RAM). Internal memory is the example of computer-readable medium.

Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by Any method or technique come realize information store.Information can be computer-readable instruction, data structure, The module of program or other data.The example of the storage medium of computer includes, but are not limited to phase transition internal memory (PRAM), static RAM (SRAM), dynamic random access memory (DRAM), Other kinds of random access memory (RAM), read-only storage (ROM), electrically erasable Read-only storage (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc read-only storage (CD-ROM), digital versatile disc (DVD) or other optical storages, magnetic cassette tape, tape Magnetic rigid disk is stored or other magnetic storage apparatus or any other non-transmission medium, can be by available for storage The information that computing device is accessed.Defined according to herein, computer-readable medium does not include non-temporary electricity The data-signal and carrier wave of brain readable media (transitory media), such as modulation.

Some vocabulary have such as been used to censure specific components among specification and claim.This area skill Art personnel are, it is to be appreciated that hardware manufacturer may call same component with different nouns.This explanation Book and claim be not in the way of the difference of title is used as differentiation component, but with component in function On difference be used as the criterion of differentiation.Such as " the bag in specification in the whole text and claim mentioned in Containing " it is an open language, therefore " include but be not limited to " should be construed to." substantially " refer to receivable In error range, those skilled in the art can solve the technical problem, base in the range of certain error Originally the technique effect is reached.In addition, " coupling " one word is herein comprising any direct and indirect electrically coupling Take over section.Therefore, if a first device is coupled to a second device described in text, described first is represented Device can directly be electrically coupled to the second device, or indirectly electric by other devices or coupling means Property is coupled to the second device.Specification subsequent descriptions is implement the better embodiment of the application, so The description is for the purpose of the rule for illustrating the application, to be not limited to scope of the present application. The protection domain of the application is worked as to be defined depending on the appended claims person of defining.

It should also be noted that, term " comprising ", "comprising" or its any other variant be intended to it is non- It is exclusive to include, so that commodity or system including a series of key elements not only will including those Element, but also other key elements including being not expressly set out, or also include for this commodity or be The intrinsic key element of system.In the absence of more restrictions, limited by sentence "including a ..." Key element, it is not excluded that also there is other identical element in the commodity or system including the key element.

Some preferred embodiments of the application have shown and described in described above, but as it was previously stated, should manage Solution the application is not limited to form disclosed herein, is not to be taken as the exclusion to other embodiment, And available for it is various other combination, modification and environment, and can in invention contemplated scope described herein, It is modified by the technology or knowledge of above-mentioned teaching or association area.And those skilled in the art are carried out changes Dynamic and change does not depart from spirit and scope, then all should the application appended claims protection In the range of.

Claims

1. a kind of data mover system, it is characterised in that including：Entrance (Portal) module, application Program Interfaces (API) module, distributed scheduling module, full dose transferring module, full dose calibration mode Block；

The inlet module is used for the entrance for providing management migration task, receives the migration created by client The data to be migrated of task and the first database, the second database and first database specified, institute Stating migration task includes full dose migration, full dose verification；

The application programming interface module is used to provide service interface to the client, with according to institute State the migration type in migration task and call corresponding full dose transferring module, full dose correction verification module；

The full dose correction verification module, for when the full dose transferring module completes Data Migration, for institute State the data to be migrated that full dose transferring module migrated and judge first database and the second data Whether the data in storehouse are consistent, and inconsistent data are stored in into the presets list；When in described the presets list When there are inconsistent data, duplication check is carried out to the inconsistent data every preset duration, , will be described inconsistent when the inconsistent data still have after the duplication check of preset times Data Ji Lu migration report in.

2. data mover system as claimed in claim 1, it is characterised in that the system is further Including the first migration assessment unit,

When the migration task, which includes migration, to be assessed, the first migration assessment unit, receiving is described should With calling for Program Interfaces module, in the full dose transferring module to the data to be migrated Before being migrated, assessment report is exported according to first database and the second database, wherein, institute Stating assessment report includes the space hold information of first database, Ku Biaolie and object information, performs SQL and process performance information, the assessment report also includes the incompatible spy of second database Property information and improvement cost information.

3. data mover system as claimed in claim 1, it is characterised in that the system is further Including structural transfer module,

When the migration task includes structural transfer, the structural transfer module receives the application journey Sequence DLL module is called, for before the full dose transferring module carries out Data Migration, by institute Ku Biaolie used in data to be migrated and object migration are stated to second database, when the described first number During according to storehouse and the second database isomery, the Ku Biaolie and object are determined by default Type mapping relation Type in second database, the storehouse corresponding with the type is created in second database Table is arranged and object.

4. data mover system as claimed in claim 3, it is characterised in that the system is further Including the second migration assessment unit,

When the migration task, which includes migration, to be assessed, the second migration assessment unit, receiving is described should With calling for Program Interfaces module, in the structural transfer module to the data institute to be migrated Before the Ku Biaolie and object used is migrated, exported according to first database and the second database Assessment report, wherein, the assessment report includes the space hold information of first database, storehouse table Row and object information, the SQL and process performance information of execution, the assessment report also include described the The incompatible characteristic information and improvement cost information of two databases.

5. data mover system as claimed in claim 1, it is characterised in that

The distributed scheduling module passes through the unit that is performed in each equipment in statistics migratory system The quantity and memory usage of change carries out the scheduling of migration task, with realize load between each equipment it is equal Weighing apparatus.

6. data mover system as claimed in claim 1, it is characterised in that

The full dose transferring module records written data when writing data to second database Site information, for breakpoint transmission.

7. data mover system as claimed in claim 1, it is characterised in that the full dose migrates mould Block is further used for,

8. data mover system as claimed in claim 1, it is characterised in that the full dose migrates mould Block is further used for,

9. data mover system as claimed in claim 1, it is characterised in that the full dose migrates mould Block is further used for,

10. data mover system as claimed in claim 1, it is characterised in that the full dose calibration mode Block is further used for,

11. a kind of data mover system, it is characterised in that including：Entrance (Portal) module, application Program Interfaces (API) module, distributed scheduling module, full dose transferring module, increment migration mould Block, full dose correction verification module；

The inlet module is used for the entrance for providing management migration task, receives the migration created by client The data to be migrated of task and the first database, the second database and first database specified, institute Stating migration task includes full dose migration, increment migration, full dose verification；

The application programming interface module is used to provide service interface to the client, with according to institute State the migration type in migration task and call corresponding full dose transferring module, increment transferring module, full dose school Test module；

The increment transferring module, will on the basis of the first moment during for by full dose migration beginning The data to be migrated produce the Data Migration of change to second data after first moment Storehouse, when first database and the second database isomery, by default intermediate form by the production The data type conversion of the data for changing is the data type of second database, after conversion The data for producing change are write second database by data type.

12. data mover system as claimed in claim 11, it is characterised in that the system enters one Step includes the first migration assessment unit,

13. data mover system as claimed in claim 11, it is characterised in that the system enters one Step includes structural transfer module,

14. data mover system as claimed in claim 13, it is characterised in that the system enters one Step includes the second migration assessment unit,

15. data mover system as claimed in claim 11, it is characterised in that

16. data mover system as claimed in claim 11, it is characterised in that

The full dose transferring module or the increment transferring module are writing data to second database When, the site information of written data is recorded, for breakpoint transmission.

17. data mover system as claimed in claim 11, it is characterised in that the full dose migration Module is further used for,

18. data mover system as claimed in claim 11, it is characterised in that the full dose migration Module is further used for,

19. data mover system as claimed in claim 11, it is characterised in that the full dose migration Module is further used for,

20. data mover system as claimed in claim 11, it is characterised in that the full dose verification Module is further used for,

21. data mover system as claimed in claim 11, it is characterised in that the increment migration Module is further used for,

Preserved using the daily record of first database, or using the trigger of first database Operation note the data of change are produced after first moment to obtain.

22. a kind of data mover system, it is characterised in that including：Entrance (Portal) module, application Program Interfaces (API) module, distributed scheduling module, full dose transferring module, increment migration mould Block, increment correction verification module；

The inlet module is used for the entrance for providing management migration task, receives the migration created by client The data to be migrated of task and the first database, the second database and first database specified, institute Stating migration task includes full dose migration, increment migration, increment verification；

The application programming interface module is used to provide service interface to the client, with according to institute State the migration type in migration task and call corresponding full dose transferring module, increment transferring module, increment school Test module；

The increment correction verification module, for when the increment transferring module migrated at first moment When the data for producing change afterwards have caught up with the data to be migrated current data variation, it is directed in real time The data that change is produced after first moment judge the number of first database and the second database According to whether consistent, and by inconsistent data Ji Lu the presets lists；Described in existing when described the presets list During inconsistent data, duplication check is carried out to the inconsistent data every preset duration, passed through When the inconsistent data still have after the duplication check of preset times, by the inconsistent number According to record in migration report.

23. data mover system as claimed in claim 22, it is characterised in that the system enters one Step includes the first migration assessment unit,

24. data mover system as claimed in claim 22, it is characterised in that the system enters one Step includes structural transfer module,

25. data mover system as claimed in claim 24, it is characterised in that the system enters one Step includes the second migration assessment unit,

26. data mover system as claimed in claim 22, it is characterised in that

27. data mover system as claimed in claim 22, it is characterised in that

28. data mover system as claimed in claim 22, it is characterised in that the full dose migration Module is further used for,

29. data mover system as claimed in claim 22, it is characterised in that the full dose migration Module is further used for,

30. data mover system as claimed in claim 22, it is characterised in that the full dose migration Module is further used for,

31. data mover system as claimed in claim 22, it is characterised in that the increment migration Module is further used for,

32. data mover system as claimed in claim 22, it is characterised in that

The increment transferring module utilizes the daily record of first database, or utilizes first data The operation note that the trigger in storehouse is preserved produces the data of change to obtain after first moment.

33. data mover system as claimed in claim 22, it is characterised in that the increment verification Module is further used for：

Pass through the operation of the daily record data for parsing the first database or the trigger for parsing the first database Record obtains the data of increment migration.

34. a kind of data mover system, it is characterised in that including：Entrance (Portal) module, application Program Interfaces (API) module, distributed scheduling module, full dose transferring module, increment migration mould Block, full dose correction verification module, increment correction verification module；

35. data mover system as claimed in claim 34, it is characterised in that

36. data mover system as claimed in claim 35, it is characterised in that

37. data mover system as claimed in claim 34, it is characterised in that the full dose migration Module is further used for,

38. data mover system as claimed in claim 34, it is characterised in that the full dose migration Module is further used for,

39. data mover system as claimed in claim 34, it is characterised in that the full dose migration Module is further used for,

40. data mover system as claimed in claim 34, it is characterised in that the full dose verification Module is further used for,

41. data mover system as claimed in claim 34, it is characterised in that the increment migration Module is further used for,

42. data mover system as claimed in claim 34, it is characterised in that the increment verification Module is further used for：

43. a kind of data migration method, it is characterised in that including：

Receive the migration task that is created by client and the first database specified, the second database and described The data to be migrated of first database, wherein, the migration task includes full dose migration, full dose verification, First database is in stopped status；

After by the data to be migrated from first database migration to second database, adjust The data after the migration are verified with full dose correction verification module, to judge first database and Whether the corresponding data of two databases is consistent, and by inconsistent data Ji Lu the presets lists；When described When there are inconsistent data in the presets list, the inconsistent data are weighed every preset duration Resume classes and test, will when the inconsistent data still have after the duplication check Jing Guo preset times In the inconsistent data Ji Lu migration reports.

44. data migration method as claimed in claim 43, it is characterised in that when the migration is appointed When business also includes migration assessment, before full dose transferring module is called, methods described also includes：

The first migration assessment unit is called, report is assessed according to first database and the output of the second database Accuse, wherein, the assessment report includes the space hold information of first database, Ku Biaolie and right Image information, the SQL and process performance information of execution, the assessment report also includes second data The incompatible characteristic information and improvement cost information in storehouse.

45. data migration method as claimed in claim 43, it is characterised in that when the migration is appointed When business also includes structural transfer, before full dose transferring module is called, methods described also includes：

Structural transfer module is called by Ku Biaolie used in the data to be migrated and object migration to institute The second database is stated, when first database and the second database isomery, is reflected by default type The relation of penetrating determines the type of the Ku Biaolie and object in second database, in second data Storehouse creates the Ku Biaolie and object corresponding with the type.

46. data migration method as claimed in claim 45, it is characterised in that when the migration is appointed When business also includes migration assessment, before structural transfer module is called, methods described also includes：

The second migration assessment unit is called, report is assessed according to first database and the output of the second database Accuse, wherein, the assessment report includes the space hold information of first database, Ku Biaolie and right Image information, the SQL and process performance information of execution, the assessment report also includes second data The incompatible characteristic information and improvement cost information in storehouse.

47. data migration method as claimed in claim 43, it is characterised in that call the full dose Transferring module, during by the data to be migrated from first database migration to second database, Further comprise：

The site information of written data is recorded, for breakpoint transmission.

48. data migration method as claimed in claim 43, it is characterised in that call the full dose The data to be migrated are write second database, further comprised by transferring module,

49. data migration method as claimed in claim 43, it is characterised in that call full dose to migrate Module from first database migration to second database, further wraps the data to be migrated Include,

50. data migration method as claimed in claim 43, it is characterised in that call full dose to migrate Module from first database migration to second database, further wraps the data to be migrated Include,

51. data migration method as claimed in claim 43, it is characterised in that call full dose to verify Module is verified to the data after the migration, is further comprised,

52. a kind of data migration method, it is characterised in that including：

Receive the migration task that is created by client and the first database specified, the second database and described The data to be migrated of first database, wherein, the migration task is migrated including full dose, increment is migrated, Full dose is verified, and first database is in open state；

Increment transferring module is called, will be described on the basis of the first moment for migrating when starting by the full dose Data to be migrated produce the Data Migration of change to second database after first moment, when When first database and the second database isomery, change is produced by described by default intermediate form Data data type conversion be second database data type, according to the data class after conversion The data for producing change are write second database by type.

53. data migration method as claimed in claim 52, it is characterised in that when the migration is appointed When business also includes migration assessment, before full dose transferring module is called, methods described also includes：

54. data migration method as claimed in claim 52, it is characterised in that when the migration is appointed When business also includes structural transfer, before full dose transferring module is called, methods described also includes：

55. data migration method as claimed in claim 54, it is characterised in that when the migration is appointed When business also includes migration assessment, before structural transfer module is called, methods described also includes：

56. data migration method as claimed in claim 52, it is characterised in that call the full dose Transferring module or the increment transferring module, by the data to be migrated from first database migration to During second database, further comprise：

The site information of written data is recorded, for breakpoint transmission.

57. data migration method as claimed in claim 52, it is characterised in that call the full dose The data to be migrated are write second database, further comprised by transferring module,

58. data migration method as claimed in claim 52, it is characterised in that call full dose to migrate Module from first database migration to second database, further wraps the data to be migrated Include,

59. data migration method as claimed in claim 52, it is characterised in that call full dose to migrate Module from first database migration to second database, further wraps the data to be migrated Include,

60. data migration method as claimed in claim 52, it is characterised in that call full dose to verify Module is verified to the data after the migration, is further comprised,

61. data migration method as claimed in claim 52, it is characterised in that call increment to migrate Module, is specifically included,

62. data migration method as claimed in claim 52, it is characterised in that methods described is also wrapped Include：

When the client has been switched to second database to the write operation of first database When, will on the basis of the second moment when writing data first to second database by the client Second database produces the Data Migration of change to first data after second moment Storehouse；

63. data migration method as claimed in claim 62, it is characterised in that methods described is also wrapped Include：

When the client occurs abnormal for the access of second database, by the client pin First database is switched back into the write operation of second database.

64. a kind of data migration method, it is characterised in that including：

Receive the migration task that is created by client and the first database specified, the second database and described The data to be migrated of first database, wherein, the migration task is migrated including full dose, increment is migrated, Increment is verified, and first database is in open state；

When the increment transferring module migrated at first moment after produced the data of change When catching up with the current data variation of the data to be migrated, methods described also includes：Call increment calibration mode Block is directed in real time to be produced the data of change after first moment and judges first database and the Whether the data of two databases consistent and data that migrated for the migration task judge described the Whether one database is consistent with the data of the second database, by inconsistent data Ji Lu the presets lists； When described the presets list has the inconsistent data, every preset duration to the inconsistent number According to duplication check is carried out, still had in data inconsistent described in the duplication check Jing Guo preset times When, by the inconsistent data Ji Lu migration reports.

65. the data migration method as described in claim 64, it is characterised in that when the migration is appointed When business also includes migration assessment, before full dose transferring module is called, methods described also includes：

66. the data migration method as described in claim 64, it is characterised in that when the migration is appointed When business also includes structural transfer, before full dose transferring module is called, methods described also includes：

67. the data migration method as described in claim 66, it is characterised in that when the migration is appointed When business also includes migration assessment, before structural transfer module is called, methods described also includes：

68. the data migration method as described in claim 66, it is characterised in that call the full dose Transferring module or the increment transferring module, by the data to be migrated from first database migration to During second database, further comprise：

The site information of written data is recorded, for breakpoint transmission.

69. the data migration method as described in claim 66, it is characterised in that call the full dose The data to be migrated are write second database, further comprised by transferring module,

70. the data migration method as described in claim 66, it is characterised in that call full dose to migrate Module from first database migration to second database, further wraps the data to be migrated Include,

71. the data migration method as described in claim 66, it is characterised in that call full dose to migrate Module from first database migration to second database, further wraps the data to be migrated Include,

72. the data migration method as described in claim 66, it is characterised in that call increment to migrate Module, is specifically included,

73. the data migration method as described in claim 66, it is characterised in that call increment to verify Module, is specifically included：

74. the data migration method as described in claim 66, it is characterised in that methods described is also wrapped Include：

75. the data migration method as described in claim 74, it is characterised in that methods described is also wrapped Include：

76. a kind of data migration method, it is characterised in that including：

77. the data migration method as described in claim 76, it is characterised in that call the full dose Transferring module or the increment transferring module, by the data to be migrated from first database migration to During second database, further comprise：

The site information of written data is recorded, for breakpoint transmission.

78. the data migration method as described in claim 76, it is characterised in that call the full dose The data to be migrated are write second database, further comprised by transferring module,

79. the data migration method as described in claim 76, it is characterised in that call full dose to migrate Module from first database migration to second database, further wraps the data to be migrated Include,

80. the data migration method as described in claim 76, it is characterised in that call full dose to migrate Module from first database migration to second database, further wraps the data to be migrated Include,

81. the data migration method as described in claim 76, it is characterised in that call full dose to verify Module is verified to the data after the migration, is further comprised,

82. the data migration method as described in claim 76, it is characterised in that call increment to migrate Module, is specifically included,

83. the data migration method as described in claim 76, it is characterised in that call increment to verify Module, is specifically included：

84. the data migration method as described in claim 76, it is characterised in that methods described is also wrapped Include：

85. the data migration method as described in claim 84, it is characterised in that methods described is also wrapped Include：