CN111177113B - Data migration method, device, computer equipment and storage medium - Google Patents
Data migration method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111177113B CN111177113B CN201911244417.1A CN201911244417A CN111177113B CN 111177113 B CN111177113 B CN 111177113B CN 201911244417 A CN201911244417 A CN 201911244417A CN 111177113 B CN111177113 B CN 111177113B
- Authority
- CN
- China
- Prior art keywords
- data
- migration
- relational database
- data migration
- parameters
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data migration method, a data migration device, computer equipment and a storage medium, wherein the method comprises the steps of running a data migration tool, wherein the data migration tool comprises a data migration front end and a data migration background; acquiring the type of a relational database corresponding to data to be migrated, and determining front-end page parameters according to the type of the relational database; loading the determined front-end page parameters and displaying the front-end page corresponding to the data migration front-end; acquiring data parameters and scheduling parameters corresponding to data to be migrated through the front-end page; and generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script so as to complete data migration by executing the migration script according to the scheduling task through the data migration background. The application relates to the field of data processing, and can improve data migration efficiency between databases of different types.
Description
Technical Field
The present disclosure relates to the field of data migration, and in particular, to a data migration method, apparatus, computer device, and storage medium.
Background
At present, in the Internet age, the database is widely applied, and plays a role in mainly helping users store and manage data, and related data can be quickly and effectively acquired and updated through operations such as inquiring, deleting or adding data newly. Although there are many types of databases available today, they mainly include two types, relational databases (Relational Database Management System, RDBMS) and non-relational databases (Not OnlyStructured Query Language, noSQL), respectively. Because the advantages and disadvantages of the relational databases and the non-relational databases can be complemented, there is often a need for data migration between different types of databases on many services.
Most of the data migration methods today utilize scheduling tools to realize migration data between databases of the same type, but migration data between databases of different types is difficult to realize or has low efficiency. For example, data in a relational database is migrated to a non-relational database, such data migration has no data migration tool with an open source, and often needs a developer to write codes to complete the data migration, which increases the workload of the developer undoubtedly and causes great trouble for users who only use the SQL language. Therefore, how to improve the data migration efficiency between the relational database and the non-relational database is a problem to be solved.
Disclosure of Invention
The application provides a data migration method, a data migration device, computer equipment and a storage medium, and aims to improve data migration efficiency between databases of different types.
In a first aspect, the present application provides a data migration method, including the steps of:
running a data migration tool, wherein the data migration tool comprises a data migration front end and a data migration background;
acquiring the type of a relational database corresponding to data to be migrated, determining front-end page parameters according to the type of the relational database, and enabling different types of relational databases to correspond to different front-end page parameters;
loading the determined front-end page parameters and displaying the front-end page corresponding to the data migration front-end;
acquiring data parameters and scheduling parameters corresponding to data to be migrated through the front-end page;
generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script;
and executing the migration script according to the scheduling task by the data migration background to complete data migration.
In a second aspect, the present application further provides a data migration apparatus, including:
the tool running module is used for running a data migration tool, and the data migration tool comprises a data migration front end and a data migration background;
the type acquisition module is used for acquiring the type of a relational database corresponding to the data to be migrated, determining front-end page parameters according to the type of the relational database, and enabling the relational databases of different types to correspond to different front-end page parameters;
the front-end display module is used for loading the determined front-end page parameters and displaying the front-end page corresponding to the data migration front-end;
the parameter acquisition module is used for acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page;
the script determining module is used for generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database and transmitting the data parameters to the migration script;
and the data migration module is used for executing the migration script according to the scheduling task through the data migration background to complete data migration.
In a third aspect, the present application also provides a computer device comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program when executed by the processor implements the steps of the data migration method as described above.
In a fourth aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a data migration method as described above.
The application provides a data migration method, a device, computer equipment and a storage medium, wherein the data migration method, the device, the computer equipment and the storage medium are used for migrating data of a relational database into a non-relational database, acquiring types of the relational database corresponding to the data to be migrated by operating a data migration tool, determining front page parameters according to the types of the relational database, loading the determined front page parameters to display the front page corresponding to the front end of data migration, acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front page, generating a scheduling task according to the scheduling parameters, determining corresponding migration scripts according to the types of the non-relational database, transmitting the data parameters to the migration scripts, and finally executing the migration scripts according to the scheduling task through a data migration background to complete data migration. The method and the device can realize rapid migration of the data from the relational database to the non-relational database, avoid complicated development process and effectively improve data migration efficiency.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data migration system according to an embodiment of the present application;
FIG. 2 is a flow chart of a data migration method according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating another data migration method according to an embodiment of the present application;
FIG. 4 is a schematic block diagram of a data migration method apparatus according to an embodiment of the present application;
FIG. 5 is a schematic block diagram of another data migration method apparatus provided in an embodiment of the present application;
fig. 6 is a schematic block diagram of a computer device according to an embodiment of the present application.
The realization, functional characteristics and advantages of the present application will be further described with reference to the embodiments, referring to the attached drawings.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The flow diagrams depicted in the figures are merely illustrative and not necessarily all of the elements and operations/steps are included or performed in the order described. For example, some operations/steps may be further divided, combined, or partially combined, so that the order of actual execution may be changed according to actual situations.
Some embodiments of the present application are described in detail below with reference to the accompanying drawings. The following embodiments and features of the embodiments may be combined with each other without conflict.
The embodiment of the application provides a data migration method, a data migration device, computer equipment and a storage medium, which are used for migrating data in a relational database to a non-relational database so as to improve data migration efficiency.
As shown in fig. 1, fig. 1 is a data migration system provided in an embodiment of the present application, where the system includes a terminal and a server, where the terminal is communicatively connected to the server, and the server is communicatively connected to a database, and the database includes a relational database and a non-relational database.
The terminal comprises electronic equipment such as a mobile phone, a tablet personal computer, a notebook computer, a desktop computer, a personal digital assistant, wearable equipment and the like.
Wherein the server comprises a single server or a cluster of servers.
Hereinafter, a data migration method provided by the embodiment of the present application will be described in detail based on the data migration system.
Referring to fig. 2, fig. 2 is a flow chart of a data migration method according to an embodiment of the present application. The data migration method can be used in a terminal of a data migration system.
As shown in fig. 2, the data migration method includes steps S101 to S105.
Step S101, a data migration tool is operated, wherein the data migration tool comprises a data migration front end and a data migration background.
Prior to running the data migration tool, it is necessary to pre-build the data migration tool, which includes a data migration front end and a data migration background. Specifically, the data migration background can be developed by utilizing the currently popular Java or Python language and based on the Django framework in Spring Boot or Python; meanwhile, a lightweight front-end framework, act or Vue, is used to develop a data migration front-end.
The data migration front end is specifically a Web front end, and is operated in the terminal to acquire corresponding information through interaction of a display Web page and a user, and the information is sent to a migration background for processing.
After the data migration tool is developed, the data migration tool is stored in a server, and when a user needs to migrate data in a relational database to a non-relational database, the pre-created data migration tool is started and operated, namely the Web front end of the data migration tool is opened to prepare data migration.
In an embodiment, when the Web front end of the data migration tool is opened and operated, the first front end page and the data migration background of the data migration tool are operated first, and the displayed first front end page may only include an input box, where the input box is used to facilitate a user to input a database identifier of a relational database corresponding to data to be migrated. Wherein the database identification may be a database name or a database ID.
S102, acquiring the type of a relational database corresponding to the data to be migrated, and determining a front-end page parameter according to the type of the relational database.
Wherein, different types of relational databases correspond to different front-end page parameters. Therefore, different pages are displayed according to different data types, a user can conveniently check or select data migration, and the display speed of the pages is increased.
In one embodiment, the type of a relational database corresponding to data to be migrated is obtained, and the front-end page parameter is determined according to the type of the relational database, which comprises the following specific processes: acquiring a database identifier corresponding to data to be migrated, inquiring the type of the relational database according to the database identifier, determining a display strategy, page content and a field analysis rule corresponding to the type according to the type of the relational database, and analyzing field information in a data table of the relational database according to the determined field analysis rule; and taking the display strategy, the page content and the field information obtained by analysis as front-end page parameters.
It should be noted that, different types of relational databases correspond to different front-end page parameters, because the different types of relational databases store different data formats, such as fields in a database table: the data table and the field information of the relational database can be displayed by utilizing the front-end page, and the specific content of the front-end page can be displayed quickly only by determining the type of the relational database.
S103, loading the determined front-end page parameters, and displaying the front-end page corresponding to the data migration front-end.
After the front-end page parameters are obtained, the front-end page parameters are loaded, page content and field information obtained through analysis are displayed according to a display strategy, namely the page content and the field information form a front-end page corresponding to the data migration front-end.
The specific display process is as follows: acquiring page content, a display strategy and a field analysis rule in front-end page parameters; inquiring and analyzing field information corresponding to the data to be migrated according to the field analysis rule; displaying the page content and the field information according to the display strategy; the field information is used for facilitating the user to determine the data to be migrated.
It should be noted that, since the different types of relational databases correspond to different front-end page parameters, the display policies, page contents and parsing rules contained in the different types of front-end page parameters may be the same or different. The last displayed front page may or may not be the same.
S104, acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page.
The method comprises the steps of obtaining data parameters and scheduling parameters corresponding to data to be migrated through a front-end page, wherein the data parameters and the scheduling parameters comprise a database for obtaining the data parameters, a data table name, a type of a non-relational database, and execution time and execution mode for obtaining the scheduling parameters.
In an embodiment, when data migration is performed, the acquired data parameters include field information of a table in the database to be migrated, and the field information is displayed through the front-end page, so that a user can configure a field of the table in the database to be migrated in the front-end page according to needs, that is, input a database of data to be migrated, a name of the data table, field information and the like in an input box of the front-end page.
The scheduling parameters include execution time and execution mode. The execution mode includes automatic execution or manual execution, wherein the automatic execution mode can be for example, when the time reaches the execution time set by the scheduling parameter, the data migration is automatically executed, and the manual execution can be for manually activating an activation button arranged on a data front page within the execution time according to the user requirement. For example, when it is detected that the user clicks the activation button, data migration may be performed. Alternatively, the activation state of the activation button may be indicated by a color change, for example, a gray to green change, or may be designed according to the user's needs, and the embodiment is not limited herein.
S105, generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script so as to complete data migration by executing the migration script through the data migration background according to the scheduling task.
Specifically, the scheduling task is generated according to the scheduling parameters, that is, the scheduling task is generated according to the execution time and the execution mode, that is, the Spark scheduling task, and meanwhile, the Spark scheduling task can be packaged into a scheduling Jar packet and sent to the data migration background, so that the data migration background executes the scheduling task according to the Jar packet.
Specifically, determining a corresponding migration script according to the type of the non-relational database, wherein the migration script is Shell script, and the specific process is as follows: and inquiring a mapping relation table preset between the type of the non-relational database and the migration script, and obtaining an inquiry result to determine the corresponding migration script.
Wherein, different Shell scripts correspond to different database types, including configuration and data format requirements related to the database types, and the like. Specifically, as shown in table 1, table 1 represents a mapping relationship table in which Shell scripts correspond to the types of the non-relational database.
Table 1 mapping table
| Types of non-relational databases | Shell script |
| HBase | Shell script I |
| MongoDB | Shell script II |
| ELASEARCH | Shell script III |
In addition, the Shell script also has a speed limiting function, and for a system with extremely large access quantity, data cannot be read and written without restriction, because the bandwidth of each system is limited, and the IO (input output) of a disk is also limited. Thus, the speed limit is required, specifically: monitoring the consumed time of the migration script when reading and writing data; the dwell time is determined based on the elapsed time and the dwell is based on the dwell time.
For example, the Shell script monitors the time spent reading and writing data, and starts to wait briefly if the corresponding time spent reading the corresponding data exceeds a preset threshold, for example, it takes more than 100ms to read 100 pieces of data, the Shell script pauses for 1s, and if the time spent reading the corresponding data exceeds 200ms, the Shell script pauses for 5s. Therefore, speed limiting is realized, and the data migration speed is improved.
After the Shell script is determined, data parameters corresponding to the data to be migrated are transmitted to the determined Shell script, and the transmitted parameters are different according to different non-relational databases. For example, the hive data of the relational database is migrated to a ELASTICSEARCH (abbreviated as ES) database of the non-relational database, the incoming data parameters are hive SQL statements, and when the data of the MySQL database is migrated to the HBase database, the incoming data parameters are MySQL data table names.
When the execution time in the scheduling task arrives, the data migration background calls the Shell script of the input parameters, and executes the Shell script to complete the data migration.
Specifically, the Shell script is called by using the spring boot of the data migration background, so that data scheduling among databases is realized, such as what time of day and what fields need to be scheduled, or what field scheduling is completed by starting by manual button clicking, thereby completing data migration from the relational database to the non-relational database.
According to the data migration method for migrating the data of the relational database to the non-relational database, the type determination front-end page of the relational database corresponding to the data to be migrated is obtained through the pre-created data migration tool, the data parameters of the data to be migrated and the type of the non-relational data to be migrated are obtained through the front-end page, the Shell script of the pair is determined according to the type of the non-relational database, the data migration is completed through the Shell script, the data can be quickly migrated from the relational database to the non-relational database, the complicated development process is avoided, and the data migration efficiency is improved.
Referring to fig. 3, fig. 3 is a flow chart of another data migration method according to an embodiment of the present application. The data migration method can be used in a data migration system.
As shown in fig. 3, the data migration method includes steps S201 to S207.
S201, running a data migration tool, wherein the data migration tool comprises a data migration front end and a data migration background;
starting and running a pre-created data migration tool, namely opening the Web front end of the data migration tool, running a first front end page of the data migration tool and a data migration background, wherein the displayed first front end page only comprises an input box, and the input box is used for facilitating a user to input data parameters of a relational database corresponding to data to be migrated.
S202, displaying an input box through a data migration front end, acquiring the type of a relational database corresponding to data to be migrated through the input box, and determining a front end page parameter according to the type of the relational database.
Specifically, after the data migration front end displays the input box, the user can input data parameters corresponding to data to be migrated, such as a database name, a data table and corresponding field information, corresponding to the data to be migrated, which are input, according to the input data parameters, obtain the database name corresponding to the data to be migrated, which is filled in by the user in the input box, and obtain the type of the relational database corresponding to the data to be migrated through the database name.
The front-end page parameters are determined according to the type of the relational database, and the specific process is as follows: determining a display strategy, page content and a field analysis rule corresponding to the type according to the type of the relational database, and analyzing field information in a data table of the relational database according to the determined field analysis rule; and taking the display strategy, the page content and the field information obtained by analysis as front-end page parameters.
S203, loading the determined front-end page parameters, and displaying the front-end page corresponding to the data migration front-end;
loading the determined front-end page parameters to display the front-end page corresponding to the data migration front-end, wherein the method specifically comprises the following steps: and loading the determined front-end page parameters to display a second front-end page corresponding to the data migration front-end on the first front-end page, wherein the second front-end page is related to the type of the relational database, the different types of relational data correspond to different front-end pages, and the first front-end page and the second front-end page jointly form the front-end page of the data migration front-end.
In an embodiment, the first front-end page obtains the page content, the display strategy and the field analysis rule in the page parameters, and the second front-end page queries and analyzes the field information corresponding to the data to be migrated according to the field analysis rule and displays the page content and the field information on the front-end page according to the display strategy. The field information is used for facilitating the user to determine the data to be migrated.
S204, acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page;
the method comprises the steps of obtaining data parameters and scheduling parameters corresponding to data to be migrated through a front-end page, wherein the data parameters and the scheduling parameters comprise a database for obtaining the data parameters, a data table name, a type of a non-relational database, and execution time and execution mode for obtaining the scheduling parameters.
In an embodiment, the data parameters acquired by the front-end page are data parameters corresponding to the data to be migrated by acquiring an import index or a namespace of the database. Specifically, databases of different types correspond to different naming manners of databases, wherein an import index refers to a concept similar to a database on an ES, and a namespace is a concept similar to an RDBMS database, and a plurality of data tables exist under each namespace, so that data to be migrated is stored in the data tables of the namespaces.
The scheduling parameters include execution time and execution mode. The execution mode includes automatic execution or manual execution, wherein the automatic execution mode can be for example, when the time reaches the execution time set by the scheduling parameter, the data migration is automatically executed, and the manual execution can be for manually activating an activation button arranged on a data front page within the execution time according to the user requirement. For example, when it is detected that the user clicks the activation button, data migration may be performed. Alternatively, the activation state of the activation button may be indicated by a color change, for example, a gray to green change, or may be designed according to the user's needs, and the embodiment is not limited herein.
S205, generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters into the migration script.
Specifically, a scheduling task is generated according to scheduling parameters, namely, the scheduling task is generated according to execution time and execution mode, namely, spark scheduling task, and a corresponding migration script is determined according to the type of the non-relational database, wherein the migration script is Shell script, and the specific process is as follows: inquiring a mapping relation table preset between the type of the non-relational database and the migration script, obtaining an inquiry result to determine a corresponding migration script, and transmitting data parameters corresponding to the data to be migrated to the determined migration script.
S206, executing the migration script according to the scheduling task through the data migration background to complete data migration.
When the execution time in the scheduling task arrives, the data migration background calls the Shell script of the input parameters, and executes the Shell script to complete the data migration.
Specifically, the Shell script is called by using the spring boot of the data migration background, so that data scheduling among databases is realized, such as what time of day and what fields need to be scheduled, or what field scheduling is completed by starting by manual button clicking, thereby completing data migration from the relational database to the non-relational database.
S207, sending a data verification instruction to a server to verify the data migrated to the non-relational database, and when verification is passed, sending a data deletion instruction to the relational database corresponding to the data to be migrated so as to delete the original data.
Specifically, after completing the data migration from the relational database to the non-relational data, the method further comprises verifying the data migrated to the non-relational database, and displaying the verification result at the data migration front end to determine the integrity and correctness of the migrated data.
And when the verification passes, sending a data deleting instruction to the relational database corresponding to the data to be migrated so as to delete the original data, thereby saving the storage space of the original relational database. The specific verification process is as follows: encrypting data to be migrated, and obtaining an encrypted character string; and meanwhile, the migrated data is also encrypted to obtain an encrypted character string, then whether the data to be migrated and the encrypted character string of the migrated data are consistent or not is compared, if the comparison result is consistent, the verification is confirmed to be passed, and otherwise, the verification is not passed. The manner of encrypting the data to be migrated is identical to the manner of encrypting the data after migration, and for example, the MD5 encryption manner is adopted.
The data migration method provided by the embodiment is used for migrating the data of the relational database to the non-relational database, acquiring the type, the data parameter and the scheduling parameter of the data to be migrated through the data migration front end, determining the corresponding migration script according to the type of the non-relational database, executing the determined migration script through the data migration background to complete data migration, and carrying out encryption verification on the migrated data to determine the integrity and the correctness of the migrated data, thereby improving the efficiency and the accuracy of data migration.
Referring to fig. 4, fig. 4 is a schematic block diagram of a data migration apparatus according to an embodiment of the present application, where the data migration apparatus is configured to perform the foregoing data migration method. The data migration device may be configured in the terminal.
As shown in fig. 4, the moving object detection device 300 includes: a tool running module 301, a type acquiring module 302, a front end display module 303, a parameter acquiring module 304 and a script determining module 305.
A tool running module 301, configured to run a data migration tool, where the data migration tool includes a data migration front end and a data migration background;
the type obtaining module 302 is configured to obtain a type of a relational database corresponding to data to be migrated, determine a front-end page parameter according to the type of the relational database, where different types of relational databases correspond to different front-end page parameters;
the front-end display module 303 is configured to load the determined front-end page parameter, and display a front-end page corresponding to the data migration front-end;
the parameter obtaining module 304 is configured to obtain, through the front-end page, a data parameter and a scheduling parameter corresponding to data to be migrated;
the script determining module 305 is configured to generate a scheduling task according to the scheduling parameter, determine a corresponding migration script according to a type of the non-relational database, and transfer the data parameter to the migration script so as to complete data migration by executing the migration script according to the scheduling task through the data migration background.
In an embodiment, the type obtaining module 302 is further configured to obtain a database identifier corresponding to the data to be migrated, and query the type of the relational database according to the database identifier; determining a display strategy, page content and a field analysis rule corresponding to the type according to the type of the relational database; analyzing field information in a data table of the relational database according to the determined field analysis rule; and taking the display strategy, the page content and the field information obtained by analysis as front-end page parameters.
In an embodiment, the front-end display module 303 is further configured to obtain page content, a display policy, and a field parsing rule in the front-end page parameter; inquiring and analyzing field information corresponding to the data to be migrated according to the field analysis rule; and displaying the page content and the field information according to the display strategy.
In an embodiment, the parameter obtaining module 304 is further configured to obtain, through a front-end page, a data parameter of data to be migrated, where the data parameter includes a database, a data table name, and a type of a non-relational database; and acquiring scheduling parameters of the data to be migrated through the front-end page, wherein the scheduling parameters comprise execution time and execution mode.
In an embodiment, the script determining module 305 is further configured to generate a scheduling task according to the execution time and the execution mode; and inquiring a mapping relation table preset between the type of the non-relational database and the migration script, and obtaining an inquiry result to determine the corresponding migration script.
Referring to fig. 5, fig. 5 is a schematic block diagram of another data migration apparatus according to an embodiment of the present application. The data migration device is used for executing the data migration method. The data migration device can be configured in a server and a terminal.
As shown in fig. 5, the face recognition apparatus 400 includes: a tool running module 401, a type obtaining module 402, a front end display module 403, a parameter obtaining module 404, a script determining module 405, a data migration module 406, and a data verification module 407.
A tool running module 401, configured to run a data migration tool, where the data migration tool includes a data migration front end and a data migration background;
the type acquisition module 402 is configured to display an input box through the data migration front end, acquire a type of a relational database corresponding to data to be migrated through the input box, and determine a front end page parameter according to the type of the relational database;
a front-end display module 403, configured to load the determined front-end page parameters, and display a front-end page corresponding to the data migration front-end
The parameter obtaining module 404 is configured to obtain, through the front-end page, a data parameter and a scheduling parameter corresponding to data to be migrated;
the script determining module 405 is configured to generate a scheduling task according to the scheduling parameter, determine a corresponding migration script according to a type of the non-relational database, and transmit the data parameter to the migration script;
and the data migration module 406 is configured to execute the migration script according to the scheduling task through the data migration background to complete data migration.
The data verification module 407 is configured to verify the data migrated to the non-relational database, and when the verification passes, send a data deletion instruction to the relational database corresponding to the data to be migrated to delete the original data.
It should be noted that, for convenience and brevity of description, specific working processes of the above-described apparatus and each module may refer to corresponding processes in the foregoing data migration method embodiment, which is not described herein again.
The apparatus provided by the above embodiments may be implemented in the form of a computer program which may be run on a computer device as shown in fig. 6.
Referring to fig. 6, fig. 6 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device may be a terminal.
With reference to FIG. 6, the computer device includes a processor, memory, and a network interface connected by a system bus, where the memory may include a non-volatile storage medium and an internal memory.
The non-volatile storage medium may store an operating system and a computer program. The computer program comprises program instructions that, when executed, cause a processor to perform any one of a number of data migration methods.
The processor is used to provide computing and control capabilities to support the operation of the entire computer device.
The internal memory provides an environment for the execution of a computer program in a non-volatile storage medium that, when executed by a processor, causes the processor to perform any one of a number of data migration methods.
The network interface is used for network communication such as transmitting assigned tasks and the like. It will be appreciated by those skilled in the art that the structure shown in fig. 6 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
It should be appreciated that the processor may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-Programmable gate arrays (FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. Wherein the general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Wherein in one embodiment the processor is configured to run a computer program stored in the memory to implement the steps of:
running a data migration tool, wherein the data migration tool comprises a data migration front end and a data migration background;
acquiring the type of a relational database corresponding to data to be migrated, determining front-end page parameters according to the type of the relational database, and enabling different types of relational databases to correspond to different front-end page parameters;
loading the determined front-end page parameters and displaying the front-end page corresponding to the data migration front-end;
acquiring data parameters and scheduling parameters corresponding to data to be migrated through the front-end page;
and generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script so as to complete data migration by executing the migration script according to the scheduling task through the data migration background.
In one embodiment, the processor is configured to, when implementing obtaining a type of a relational database corresponding to data to be migrated, determine a front-end page parameter according to the type of the relational database, implement:
acquiring a database identifier corresponding to data to be migrated, and inquiring the type of the relational database according to the database identifier; determining a display strategy, page content and a field analysis rule corresponding to the type according to the type of the relational database; analyzing field information in a data table of the relational database according to the determined field analysis rule; and taking the display strategy, the page content and the field information obtained by analysis as front-end page parameters.
In one embodiment, the processor is configured to, when acquiring a type of a relational database corresponding to the data to be migrated, implement:
and displaying an input box through the data migration front end, and acquiring the type of the relational database corresponding to the data to be migrated through the input box.
In one embodiment, when implementing the front-end page parameter determined by the loading and displaying the front-end page corresponding to the data migration front-end, the processor is configured to implement:
acquiring page content, a display strategy and a field analysis rule in front-end page parameters; inquiring and analyzing field information corresponding to the data to be migrated according to the field analysis rule; and displaying the page content and the field information according to the display strategy.
In one embodiment, when implementing the obtaining, by the processor, the data parameter and the scheduling parameter corresponding to the data to be migrated through the front-end page, the processor is configured to implement:
acquiring data parameters of data to be migrated through a front-end page, wherein the data parameters comprise a database, a data table name and the type of a non-relational database; and acquiring scheduling parameters of the data to be migrated through the front-end page, wherein the scheduling parameters comprise execution time and execution mode.
In an embodiment, when implementing the generating a scheduling task according to the scheduling parameter and determining a corresponding migration script according to a type of the non-relational database, the processor is configured to implement:
generating a scheduling task according to the execution time and the execution mode; and inquiring a mapping relation table preset between the type of the non-relational database and the migration script, and obtaining an inquiry result to determine the corresponding migration script.
In an embodiment, after implementing the data migration completed by the execution of the migration script by the data migration background according to the scheduling task, the processor is further configured to implement:
and checking the data migrated to the non-relational database, and when the checking is passed, sending a data deleting instruction to the relational database corresponding to the data to be migrated so as to delete the original data.
Embodiments of the present application further provide a computer readable storage medium, where the computer readable storage medium stores a computer program, where the computer program includes program instructions, and the processor executes the program instructions to implement any one of the data migration methods provided in the embodiments of the present application.
The computer readable storage medium may be an internal storage unit of the computer device according to the foregoing embodiment, for example, a hard disk or a memory of the computer device. The computer readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, which are provided on the computer device.
While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the invention. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (8)
1. A data migration method for migrating data of a relational database to a non-relational database, comprising:
running a data migration tool, wherein the data migration tool comprises a data migration front end and a data migration background;
acquiring the type of a relational database corresponding to data to be migrated, and determining a display strategy, page content and a field analysis rule corresponding to the type according to the type of the relational database; analyzing field information in a data table of the relational database according to the determined field analysis rule; taking the display strategy, the page content and the field information obtained by analysis as front-end page parameters; different types of relational databases correspond to different front-end page parameters;
acquiring page content, a display strategy and a field analysis rule in front-end page parameters; inquiring and analyzing field information corresponding to the data to be migrated according to the field analysis rule; displaying the page content and the field information according to the display strategy;
acquiring data parameters and scheduling parameters corresponding to data to be migrated through the front-end page; the data parameters comprise a database, a data table name and the type of a non-relational database; the scheduling parameters comprise execution time and execution mode;
and generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script so as to complete data migration by executing the migration script according to the scheduling task through the data migration background.
2. The data migration method as claimed in claim 1, wherein the obtaining the type of the relational database corresponding to the data to be migrated includes:
and acquiring a database identifier corresponding to the data to be migrated, and inquiring the type of the relational database according to the database identifier.
3. The data migration method as claimed in claim 2, wherein the obtaining the type of the relational database corresponding to the data to be migrated includes:
and displaying an input box through the data migration front end, and acquiring the type of the relational database corresponding to the data to be migrated through the input box.
4. The data migration method of claim 1, wherein the generating a scheduling task according to the scheduling parameter and determining a corresponding migration script according to a type of the non-relational database comprises:
generating a scheduling task according to the execution time and the execution mode;
and inquiring a mapping relation table preset between the type of the non-relational database and the migration script, and obtaining an inquiry result to determine the corresponding migration script.
5. The data migration method of claim 1, wherein after the data migration is completed by the data migration background executing the migration script according to the scheduled task, further comprising:
and checking the data migrated to the non-relational database, and when the checking is passed, sending a data deleting instruction to the relational database corresponding to the data to be migrated so as to delete the original data.
6. A data migration apparatus, comprising:
the tool running module is used for running a data migration tool, and the data migration tool comprises a data migration front end and a data migration background;
the type acquisition module is used for acquiring the type of a relational database corresponding to the data to be migrated, and determining a display strategy, page content and a field analysis rule corresponding to the type according to the type of the relational database; analyzing field information in a data table of the relational database according to the determined field analysis rule; taking the display strategy, the page content and the field information obtained by analysis as front-end page parameters; different types of relational databases correspond to different front-end page parameters;
the front-end display module is used for acquiring page content, a display strategy and a field analysis rule in the front-end page parameters; inquiring and analyzing field information corresponding to the data to be migrated according to the field analysis rule; displaying the page content and the field information according to the display strategy;
the parameter acquisition module is used for acquiring data parameters and scheduling parameters corresponding to the data to be migrated through the front-end page; the data parameters comprise a database, a data table name and the type of a non-relational database; the scheduling parameters comprise execution time and execution mode;
and the script determining module is used for generating a scheduling task according to the scheduling parameters, determining a corresponding migration script according to the type of the non-relational database, and transmitting the data parameters to the migration script so as to complete data migration by executing the migration script according to the scheduling task through the data migration background.
7. A computer device comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program when executed by the processor implements the steps of the data migration method of any one of claims 1 to 5.
8. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when being executed by a processor, implements the steps of the data migration method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911244417.1A CN111177113B (en) | 2019-12-06 | 2019-12-06 | Data migration method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911244417.1A CN111177113B (en) | 2019-12-06 | 2019-12-06 | Data migration method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111177113A CN111177113A (en) | 2020-05-19 |
| CN111177113B true CN111177113B (en) | 2023-07-25 |
Family
ID=70648755
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911244417.1A Active CN111177113B (en) | 2019-12-06 | 2019-12-06 | Data migration method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111177113B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112214473A (en) * | 2020-09-23 | 2021-01-12 | 航天信息股份有限公司 | Data migration method and system between databases |
| CN112181489B (en) * | 2020-09-28 | 2023-07-25 | 中国平安人寿保险股份有限公司 | Code migration method, device, computer equipment and storage medium |
| CN112131128B (en) * | 2020-09-29 | 2023-08-22 | 网易（杭州）网络有限公司 | Data testing method and device, storage medium and electronic device |
| CN115543969B (en) * | 2022-11-28 | 2023-03-10 | 建信金融科技有限责任公司 | Data migration method, device, equipment and medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20080084332A (en) * | 2007-03-16 | 2008-09-19 | (주)온디멘드 | Database migration apparatus and method |
| CN109684298A (en) * | 2018-09-07 | 2019-04-26 | 平安科技（深圳）有限公司 | Database migration method, apparatus, equipment and storage medium |
| CN110209652A (en) * | 2019-05-20 | 2019-09-06 | 平安科技（深圳）有限公司 | Tables of data moving method, device, computer equipment and storage medium |
| CN110297813A (en) * | 2019-05-22 | 2019-10-01 | 平安银行股份有限公司 | Data migration method, device, computer equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160188710A1 (en) * | 2014-12-29 | 2016-06-30 | Wipro Limited | METHOD AND SYSTEM FOR MIGRATING DATA TO NOT ONLY STRUCTURED QUERY LANGUAGE (NoSOL) DATABASE |
-
2019
- 2019-12-06 CN CN201911244417.1A patent/CN111177113B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20080084332A (en) * | 2007-03-16 | 2008-09-19 | (주)온디멘드 | Database migration apparatus and method |
| CN109684298A (en) * | 2018-09-07 | 2019-04-26 | 平安科技（深圳）有限公司 | Database migration method, apparatus, equipment and storage medium |
| CN110209652A (en) * | 2019-05-20 | 2019-09-06 | 平安科技（深圳）有限公司 | Tables of data moving method, device, computer equipment and storage medium |
| CN110297813A (en) * | 2019-05-22 | 2019-10-01 | 平安银行股份有限公司 | Data migration method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111177113A (en) | 2020-05-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111177113B (en) | Data migration method, device, computer equipment and storage medium | |
| US10366053B1 (en) | Consistent randomized record-level splitting of machine learning data | |
| US11100420B2 (en) | Input processing for machine learning | |
| WO2020151333A1 (en) | Page loading method, apparatus, computer device and storage medium | |
| US20150379425A1 (en) | Consistent filtering of machine learning data | |
| CN111767143A (en) | Transaction data processing method, device, equipment and system | |
| WO2019134340A1 (en) | Salary calculation method, application server, and computer readable storage medium | |
| CN108536745B (en) | Shell-based data table extraction method, terminal, equipment and storage medium | |
| CN109829287A (en) | Api interface permission access method, equipment, storage medium and device | |
| WO2021217846A1 (en) | Interface data processing method and apparatus, and computer device and storage medium | |
| US9971563B2 (en) | Systems and methods for low interference logging and diagnostics | |
| WO2021051624A1 (en) | Data acquisition method and apparatus, and electronic device and storage medium | |
| CN112817657A (en) | Application program starting item loading method, device and system and storage medium | |
| CN113032393A (en) | Method and device for binding associated object | |
| US20070180531A1 (en) | Analyzing interpretable code for harm potential | |
| WO2019071958A1 (en) | Cloud computing-based salary calculation method, application server, and computer readable storage medium | |
| CN113010542B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN109033456B (en) | Condition query method and device, electronic equipment and storage medium | |
| CN112883088B (en) | Data processing method, device, equipment and storage medium | |
| CN113760242A (en) | Data processing method, device, server and medium | |
| US9317553B2 (en) | Declarative partitioning for data collection queries | |
| US9201937B2 (en) | Rapid provisioning of information for business analytics | |
| CN111767498A (en) | Method and device for realizing file information sharing | |
| US9659041B2 (en) | Model for capturing audit trail data with reduced probability of loss of critical data | |
| CN110471708B (en) | Method and device for acquiring configuration items based on reusable components |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |