CN113297180A - Data migration method and device, electronic equipment and storage medium - Google Patents
Data migration method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113297180A CN113297180A CN202110607745.4A CN202110607745A CN113297180A CN 113297180 A CN113297180 A CN 113297180A CN 202110607745 A CN202110607745 A CN 202110607745A CN 113297180 A CN113297180 A CN 113297180A
- Authority
- CN
- China
- Prior art keywords
- data
- script
- database
- migrated
- generating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005012 migration Effects 0.000 title claims abstract description 126
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000013508 migration Methods 0.000 title claims abstract description 52
- 238000013515 script Methods 0.000 claims abstract description 124
- 238000012216 screening Methods 0.000 claims abstract description 11
- 238000003780 insertion Methods 0.000 claims description 34
- 230000037431 insertion Effects 0.000 claims description 34
- 238000004590 computer program Methods 0.000 claims description 5
- 238000012916 structural analysis Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 7
- 230000006870 function Effects 0.000 description 11
- 230000008569 process Effects 0.000 description 6
- 238000007726 management method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to big data technology, and discloses a data migration method, which comprises the following steps: connecting a source database, and screening effective data in the source database to obtain a data set to be migrated; generating a data migration program according to the structure of the data set to be migrated; generating a table building script and a data inserting script according to the data migration program; and connecting a target database, and generating a target data table containing the data in the data set to be migrated in the target database by using the table establishing script and the data inserting script. In addition, the invention also relates to a block chain technology, and the target data table can be stored in the node of the block chain. The invention also provides a data migration device, electronic equipment and a computer readable storage medium. The invention can solve the problems that the data migration efficiency is low and the data in different types of databases cannot be migrated.
Description
Technical Field
The present invention relates to the field of big data technologies, and in particular, to a data migration method and apparatus, an electronic device, and a computer-readable storage medium.
Background
After a service system is used for a certain period of time, due to the update of technology or the change of service, the performance of the system is reduced and becomes the bottleneck of service, in this case, a new system is usually used to replace an old system, and in the process of switching from the old system to the new system, the migration of a large amount of service data is often involved.
The current data migration mode mainly includes a data migration method and a data migration method based on a database. The method for manually transferring data is to execute the finishing script after manually writing the finishing script, and the method usually involves a complicated and tedious configuration process, and is long in time consumption and low in efficiency. The data migration method based on the database is processed through an import/export or backup recovery tool of the database, but the method only supports the database of the same type, and if the types of the databases of the new system and the old system are different, the data migration cannot be realized.
Disclosure of Invention
The invention provides a data migration method, a data migration device and a computer readable storage medium, and mainly aims to solve the problems that the data migration efficiency is low and data among different types of databases cannot be migrated.
In order to achieve the above object, the present invention provides a data migration method, including:
connecting a source database, and screening effective data in the source database to obtain a data set to be migrated;
generating a data migration program according to the structure of the data set to be migrated;
generating a table building script and a data inserting script according to the data migration program;
and connecting a target database, and generating a target data table containing effective data in the data set to be migrated in the target database by using the table establishing script and the data inserting script.
Optionally, the connection source database includes:
determining the type of the source database;
loading a driver corresponding to the source database according to the type of the source database, and creating a blank connection object based on the driver;
acquiring a path, a user name and a password of the source database;
changing the attribute of the blank connection object according to the path, the user name and the password of the source database to obtain a database connection object;
and connecting the source database by using the database connection object.
Optionally, the screening out valid data from the source database to obtain a to-be-migrated data set includes:
generating a query script of corresponding content according to the type of the source database;
executing the query script in the source database, and querying to obtain all data tables stored in the source database;
and deleting invalid data in the historical data table to obtain a data set to be migrated.
Optionally, the generating a data migration program according to the structure of the data set to be migrated includes:
performing structural analysis on each data table in the data set to be migrated to obtain a data table structural element set;
generating a table building command which accords with the type of the target database according to the data table structure element set;
performing all data query on each data table in the data set to be migrated to obtain a detailed data set;
generating a data insertion command according to the table building command corresponding to the detailed data set and the data in the detailed data set;
and integrating the table establishing command and the data inserting command to obtain a data migration program.
Optionally, the generating a table building script and a data inserting script according to the data migration program includes:
obtaining table names of all data tables in the data set to be migrated, transmitting the table names as parameters to the data migration program and executing the data migration program;
generating a table building script according to an execution result of a table building command in the data migration program;
and generating a data insertion script according to an execution result of the data insertion command in the data migration program.
Generating a table building script according to an execution result of a table building command in the data migration program, wherein the table building script comprises:
obtaining a table name based on an execution result of a table establishing command in the data migration program to obtain a data table name set;
acquiring a preset script command for establishing a blank data table;
and creating a table building circulation command based on the script command, and taking the table names and the total number in the data table name set as circulation execution parameters to obtain a table building script.
The connecting the target database, and generating a target data table containing effective data in the data set to be migrated in the target database by using the table establishing script and the data inserting script, including:
connecting the target database by using the preset connection method according to the type of the target database;
executing the table establishing script to generate a blank data table consistent with the data structure of the data table in the data set to be migrated;
executing the data insertion script, and writing the effective data in the data set to be migrated into the blank data table to obtain a target data table;
and storing the target data table into the target database.
In order to solve the above problem, the present invention also provides a data migration apparatus, including:
the data acquisition module is used for connecting a source database and screening effective data in the source database to obtain a data set to be migrated;
the migration program generation module is used for generating a data migration program according to the structure of the data set to be migrated;
the script generation module is used for generating a table building script and a data insertion script according to the data migration program;
and the data generation module is used for connecting a target database and generating a target data table containing the effective data in the data set to be migrated in the target database by using the table establishing script and the data inserting script.
In order to solve the above problem, the present invention also provides an electronic device, including:
a memory storing at least one instruction; and
and the processor executes the instructions stored in the memory to realize the data migration method.
In order to solve the above problem, the present invention further provides a computer-readable storage medium, which stores at least one instruction, where the at least one instruction is executed by a processor in an electronic device to implement the data migration method described above.
The embodiment of the invention connects the source database, screens out effective data in the source database to obtain a data set to be migrated, wherein the data set to be migrated comprises all historical data, and the comprehensiveness of the migrated data is ensured; generating a data migration program according to the structure of the data set to be migrated, and converting data in a source database into data conforming to the storage structure of a target database by using the data migration program, so that the working efficiency is improved; meanwhile, a target data table is generated by using the table establishing script and the data inserting script, so that a target database can process data in batches, and the execution speed of the database is improved; and generating a target data table containing the data in the data set to be migrated in a target database through the data migration program, the table building script and the data inserting script, so that the data migration between different database types can be realized more efficiently. Therefore, the data migration method, the data migration device, the electronic equipment and the computer readable storage medium provided by the invention can solve the problems that the data migration efficiency is low and the data among different types of databases cannot be migrated.
Drawings
Fig. 1 is a schematic flow chart of a data migration method according to an embodiment of the present invention;
FIG. 2 is a functional block diagram of a data migration apparatus according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device for implementing the data migration method according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the application provides a data migration method. The execution subject of the data migration method includes, but is not limited to, at least one of electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiments of the present application. In other words, the data migration method may be performed by software or hardware installed in the terminal device or the server device, and the software may be a block chain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.
Fig. 1 is a schematic flow chart of a data migration method according to an embodiment of the present invention.
In this embodiment, the data migration method includes:
and S1, connecting the source database, and screening effective data in the source database to obtain a data set to be migrated.
In the embodiment of the invention, the source database refers to a database of an old service system and comprises a large amount of historical service data; and the database into which the historical service data is migrated is the target database, namely the database for receiving the historical service data. Meanwhile, the types of the source database and the target database are different, for example, the source database may be an SQL server database, the target database may be an oracle database, the storage structures of different types of databases are different, the data types are also different, and the used SQL statements are also different.
In detail, the connection source database includes:
determining the type of the source database;
loading a driver corresponding to the source database according to the type of the source database, and creating a blank connection object based on the driver;
acquiring a path, a user name and a password of the source database;
changing the attribute of the blank connection object according to the path, the user name and the password of the source database to obtain a database connection object;
and connecting the source database by using the database connection object.
The driver is an interface for connecting a computer and a database, and the drivers of different database types are different; the path is a character string which can provide a database position and comprises a protocol name, a domain name, a port and a database name; the user name and the password refer to the user name and the password used when logging in the database.
The embodiment of the invention is used for connecting the source Database based on a Java Database connection technology (JDBC), wherein the JDBC is a Java API used for executing SQL statements, can provide uniform access for various relational databases and consists of a group of classes and interfaces written by Java language. If the source database type is an oracle database, the source database type can be obtained through a configuration file, a driver, namely a jar packet, of the oracle database is loaded, an interface in the driver can be directly called to create a connection object after the driver is loaded, a path, a user name and a password of the source database can be obtained through the configuration file, the connection attribute of the blank connection object is called, the obtained path, the user name and the password are used as parameters of the attribute to be changed, the database connection object of the source database is obtained, and the source database is connected through a self-contained connection method in the database connection object.
In detail, the screening out valid data from the source database to obtain a to-be-migrated data set includes:
generating a query script of corresponding content according to the type of the source database;
executing the query script in the source database, and querying to obtain all data tables stored in the source database;
and deleting invalid data in the historical data table to obtain a data set to be migrated.
Further, the query script is an executable file for querying a data table stored in a database based on the SQL language, and the query script content corresponding to different database types is different. For example, when the type of the source database is an oracle database, the query script content is a selection table _ name from dba _ all _ tables where owner ═ username'; when the type of the source database is the sql server database, the query script content is select from sys.
Optionally, some historical data may be too lengthy and may not be used subsequently, or some historical data may become invalid data because some functions are deleted in the new system, and therefore, the embodiment of the present invention may filter out the invalid data to obtain a data set to be migrated, so as to save computer resources and improve data migration efficiency.
And S2, generating a data migration program according to the structure of the data set to be migrated.
In detail, the generating a data migration program according to the structure of the data set to be migrated includes:
performing structural analysis on each data table in the data set to be migrated to obtain a data table structural element set;
generating a table building command which accords with the type of the target database according to the data table structure element set;
performing all data query on each data table in the data set to be migrated to obtain a detailed data set;
generating a data insertion command according to the table building command corresponding to the detailed data set and the data in the detailed data set;
and integrating the table establishing command and the data inserting command to obtain a data migration program.
The table building command is based on a structured query language and used for realizing generation of a method code capable of creating a data table SQL statement function, and the data inserting command is based on the structured query language and used for realizing generation of a method code capable of inserting specified data into an existing data table SQL statement function.
The data migration program is a database object which realizes a certain complex function in a database so as to be called by an external program, and can be a storage process in the database. The data migration program in the embodiment of the invention can convert the table structure of one database into the table structure of another database, for example, convert the sqlServer data table structure into the oracle data table structure.
Optionally, the data migration program is stored in the database, and is permanently effective after once compiling, and is compiled once every time a general SQL statement is executed, and the data migration program only needs to be compiled during creation, and does not need to be recompiled every time of later execution, so that computer resources can be saved and the execution speed of the database can be improved.
And S3, generating a table building script and a data inserting script according to the data migration program.
In detail, the generating a table building script and a data inserting script according to the data migration program includes:
obtaining table names of all data tables in the data set to be migrated, transmitting the table names as parameters to the data migration program and executing the data migration program;
generating a table building script according to an execution result of a table building command in the data migration program;
and generating a data insertion script according to an execution result of the data insertion command in the data migration program.
Wherein the build script is an executable file for creating a blank data table; the data insertion script is an executable file for inserting specified data into an existing data table.
The scripts are database script files, tasks of users can be processed in batch by executing the scripts, the tasks include data table establishment, data query, custom function execution and the like, and specific tasks are determined by the content of the scripts.
Further, the generating a table creation script according to an execution result of the table creation command in the data migration program includes:
obtaining a table name based on an execution result of a table establishing command in the data migration program to obtain a data table name set;
acquiring a preset script command for establishing a blank data table;
and creating a table building circulation command based on the script command, and taking the table names and the total number in the data table name set as circulation execution parameters to obtain a table building script.
Further, the generating a data insertion script according to an execution result of the data insertion command in the data migration program includes:
acquiring insertion data based on an execution result of a data insertion command in the data migration program to obtain a data set;
acquiring a script command of preset inserted data;
and creating a data insertion cycle command based on the script command, and taking the data and the total number in the data set as cycle execution parameters to obtain a data insertion script.
And S4, connecting a target database, and generating a target data table containing effective data in the data set to be migrated in the target database by using the table establishing script and the data inserting script.
In detail, the S4 includes:
connecting the target database by using the preset connection method according to the type of the target database;
executing the table establishing script to generate a blank data table consistent with the data structure of the data table in the data set to be migrated;
executing the data insertion script, and writing the effective data in the data set to be migrated into the blank data table to obtain a target data table;
and storing the target data table into the target database.
Further, the connecting the target database by using the preset connection method according to the type of the target database includes:
loading a driver corresponding to the target database according to the type of the target database, and creating a blank connection object based on the driver;
acquiring a path, a user name and a password of the target database;
changing the attribute of the blank connection object according to the path, the user name and the password of the target database to obtain a database connection object;
and connecting the target database by using the database connection object.
Similarly, in the embodiment of the present invention, the connection of the target Database is also based on Java Database connection technology (JDBC).
Optionally, to further ensure the privacy and security of the target data table, the target data table may also be stored in a node of a block chain
In the embodiment of the invention, the table establishing script is executed in the target database to generate the target data table consistent with the data structure of all the data tables in the data set to be migrated, and the data inserting script is executed to generate the data with the same data content as the data content of the data set to be migrated, so that the data migration from the source database to the target database is completed.
The embodiment of the invention connects the source database, screens out effective data in the source database to obtain a data set to be migrated, wherein the data set to be migrated comprises all historical data, and the comprehensiveness of the migrated data is ensured; generating a data migration program according to the structure of the data set to be migrated, and converting data in a source database into data conforming to the storage structure of a target database by using the data migration program, so that the working efficiency is improved; meanwhile, a target data table is generated by using the table establishing script and the data inserting script, so that a target database can process data in batches, and the execution speed of the database is improved; and generating a target data table containing the data in the data set to be migrated in a target database through the data migration program, the table building script and the data inserting script, so that the data migration between different database types can be realized more efficiently. Therefore, the data migration method, the data migration device, the electronic equipment and the computer readable storage medium provided by the invention can solve the problems that the data migration efficiency is low and the data among different types of databases cannot be migrated.
Fig. 2 is a functional block diagram of a data migration apparatus according to an embodiment of the present invention.
The data migration apparatus 100 of the present invention may be installed in an electronic device. According to the implemented functions, the data migration apparatus 100 may include a data acquisition module 101, a migration program generation module 102, a script generation module 103, and a data generation module 104. The module of the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the data acquisition module 101 is configured to connect to a source database, and screen out valid data in the source database to obtain a to-be-migrated data set.
In detail, when connecting to a source database, the data obtaining module 101 specifically executes the following operations:
determining the type of the source database;
loading a driver corresponding to the source database according to the type of the source database, and creating a blank connection object based on the driver;
acquiring a path, a user name and a password of the source database;
changing the attribute of the blank connection object according to the path, the user name and the password of the source database to obtain a database connection object;
and connecting the source database by using the database connection object.
In detail, when effective data is screened from the source database to obtain a to-be-migrated data set, the data obtaining module 101 specifically executes the following operations:
generating a query script of corresponding content according to the type of the source database;
executing the query script in the source database, and querying to obtain all data tables stored in the source database;
and deleting invalid data in the historical data table to obtain a data set to be migrated.
The migration program generating module 102 is configured to generate a data migration program according to the structure of the data set to be migrated.
In detail, the migration program generation module 102 is specifically configured to:
performing structural analysis on each data table in the data set to be migrated to obtain a data table structural element set;
generating a table building command which accords with the type of the target database according to the data table structure element set;
performing all data query on each data table in the data set to be migrated to obtain a detailed data set;
generating a data insertion command according to the table building command corresponding to the detailed data set and the data in the detailed data set;
and integrating the table establishing command and the data inserting command to obtain a data migration program.
The script generating module 103 is configured to generate a table building script and a data inserting script according to the data migration program.
In detail, the script generating module 103 is specifically configured to:
obtaining table names of all data tables in the data set to be migrated, transmitting the table names as parameters to the data migration program and executing the data migration program;
generating a table building script according to an execution result of a table building command in the data migration program;
and generating a data insertion script according to an execution result of the data insertion command in the data migration program.
Wherein the build script is an executable file for creating a blank data table; the data insertion script is an executable file for inserting specified data into an existing data table.
Further, the generating a table creation script according to an execution result of the table creation command in the data migration program includes:
obtaining a table name based on an execution result of a table establishing command in the data migration program to obtain a data table name set;
acquiring a preset script command for establishing a blank data table;
and creating a table building circulation command based on the script command, and taking the table names and the total number in the data table name set as circulation execution parameters to obtain a table building script.
Further, the generating a data insertion script according to an execution result of the data insertion command in the data migration program includes:
acquiring insertion data based on an execution result of a data insertion command in the data migration program to obtain a data set;
acquiring a script command of preset inserted data;
and creating a data insertion cycle command based on the script command, and taking the data and the total number in the data set as cycle execution parameters to obtain a data insertion script.
The data generating module 104 is configured to connect to a target database, and generate a target data table containing valid data in the data set to be migrated in the target database by using the table creating script and the data inserting script.
In detail, the data generating module 104 is specifically configured to:
connecting the target database by using the preset connection method according to the type of the target database;
executing the table establishing script to generate a blank data table consistent with the data structure of the data table in the data set to be migrated;
executing the data insertion script, and writing the effective data in the data set to be migrated into the blank data table to obtain a target data table;
and storing the target data table into the target database.
Further, the connecting the target database by using the preset connection method according to the type of the target database includes:
loading a driver corresponding to the target database according to the type of the target database, and creating a blank connection object based on the driver;
acquiring a path, a user name and a password of the target database;
changing the attribute of the blank connection object according to the path, the user name and the password of the target database to obtain a database connection object;
and connecting the target database by using the database connection object.
Optionally, to further ensure the privacy and security of the target data table, the target data table may also be stored in a node of a block chain.
Fig. 3 is a schematic structural diagram of an electronic device for implementing a data migration method according to an embodiment of the present invention.
The electronic device 1 may comprise a processor 10, a memory 11 and a bus 12, and may further comprise a computer program, such as a data migration program, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device 1, such as a removable hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device 1. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 may be used not only to store application software installed in the electronic device 1 and various types of data, such as codes of a data migration program, but also to temporarily store data that has been output or is to be output.
The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device 1 by running or executing programs or modules (e.g., data migration programs, etc.) stored in the memory 11 and calling data stored in the memory 11.
The bus 12 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like.
Fig. 3 shows only an electronic device with components, and it will be understood by those skilled in the art that the structure shown in fig. 3 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than those shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device 1 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so as to implement functions of charge management, discharge management, power consumption management, and the like through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 1 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the electronic device 1 may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used for establishing a communication connection between the electronic device 1 and other electronic devices.
Optionally, the electronic device 1 may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the electronic device 1 and for displaying a visualized user interface, among other things.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The data migration program stored in the memory 11 of the electronic device 1 is a combination of a plurality of instructions, and when running in the processor 10, can realize:
connecting a source database, and screening effective data in the source database to obtain a data set to be migrated;
generating a data migration program according to the structure of the data set to be migrated;
generating a table building script and a data inserting script according to the data migration program;
and connecting a target database, and generating a target data table containing effective data in the data set to be migrated in the target database by using the table establishing script and the data inserting script.
Specifically, the specific implementation method of the processor 10 for the instruction may refer to the description of the relevant steps in the embodiments corresponding to fig. 1 to fig. 3, which is not repeated herein.
Further, the integrated modules/units of the electronic device 1, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. The computer readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
The present invention also provides a computer-readable storage medium, storing a computer program which, when executed by a processor of an electronic device, may implement:
connecting a source database, and screening effective data in the source database to obtain a data set to be migrated;
generating a data migration program according to the structure of the data set to be migrated;
generating a table building script and a data inserting script according to the data migration program;
and connecting a target database, and generating a target data table containing effective data in the data set to be migrated in the target database by using the table establishing script and the data inserting script.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method of data migration, the method comprising:
connecting a source database, and screening effective data in the source database to obtain a data set to be migrated;
generating a data migration program according to the structure of the data set to be migrated;
generating a table building script and a data inserting script according to the data migration program;
and connecting a target database, and generating a target data table containing effective data in the data set to be migrated in the target database by using the table establishing script and the data inserting script.
2. The data migration method of claim 1, wherein connecting the source database comprises:
determining the type of the source database;
loading a driver corresponding to the source database according to the type of the source database, and creating a blank connection object based on the driver;
acquiring a path, a user name and a password of the source database;
changing the attribute of the blank connection object according to the path, the user name and the password of the source database to obtain a database connection object;
and connecting the source database by using the database connection object.
3. The data migration method according to claim 1, wherein the step of screening the source database for valid data to obtain a to-be-migrated data set comprises:
generating a query script of corresponding content according to the type of the source database;
executing the query script in the source database, and querying to obtain all data tables stored in the source database;
and deleting invalid data in the historical data table to obtain a data set to be migrated.
4. The data migration method according to claim 1, wherein the generating a data migration program according to the structure of the data set to be migrated comprises:
performing structural analysis on each data table in the data set to be migrated to obtain a data table structural element set;
generating a table building command which accords with the type of the target database according to the data table structure element set;
performing all data query on each data table in the data set to be migrated to obtain a detailed data set;
generating a data insertion command according to the table building command corresponding to the detailed data set and the data in the detailed data set;
and integrating the table establishing command and the data inserting command to obtain a data migration program.
5. The data migration method of claim 4, wherein said generating a build script and a data insertion script according to said data migration program comprises:
obtaining table names of all data tables in the data set to be migrated, transmitting the table names as parameters to the data migration program and executing the data migration program;
generating a table building script according to an execution result of a table building command in the data migration program;
and generating a data insertion script according to an execution result of the data insertion command in the data migration program.
6. The data migration method according to claim 5, wherein the generating a table creation script according to the execution result of the table creation command in the data migration program comprises:
obtaining a table name based on an execution result of a table establishing command in the data migration program to obtain a data table name set;
acquiring a preset script command for establishing a blank data table;
and creating a table building circulation command based on the script command, and taking the table names and the total number in the data table name set as circulation execution parameters to obtain a table building script.
7. The data migration method according to claim 5, wherein the connecting the target database and generating the target data table containing valid data in the data set to be migrated in the target database by using the table creation script and the data insertion script comprises:
connecting the target database by using the preset connection method according to the type of the target database;
executing the table establishing script to generate a blank data table consistent with the data structure of the data table in the data set to be migrated;
executing the data insertion script, and writing the effective data in the data set to be migrated into the blank data table to obtain a target data table;
and storing the target data table into the target database.
8. An apparatus for data migration, the apparatus comprising:
the data acquisition module is used for connecting a source database and screening effective data in the source database to obtain a data set to be migrated;
the migration program generation module is used for generating a data migration program according to the structure of the data set to be migrated;
the script generation module is used for generating a table building script and a data insertion script according to the data migration program;
and the data generation module is used for connecting a target database and generating a target data table containing the effective data in the data set to be migrated in the target database by using the table establishing script and the data inserting script.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a data migration method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out a data migration method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110607745.4A CN113297180A (en) | 2021-06-01 | 2021-06-01 | Data migration method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110607745.4A CN113297180A (en) | 2021-06-01 | 2021-06-01 | Data migration method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113297180A true CN113297180A (en) | 2021-08-24 |
Family
ID=77326630
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110607745.4A Pending CN113297180A (en) | 2021-06-01 | 2021-06-01 | Data migration method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113297180A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114553958A (en) * | 2022-01-04 | 2022-05-27 | 深圳绿米联创科技有限公司 | Data migration processing method and device, electronic equipment and storage medium |
| CN115292289A (en) * | 2022-08-09 | 2022-11-04 | 华润数字科技有限公司 | Data filtering method, device, equipment and medium based on database migration |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102156699A (en) * | 2010-02-11 | 2011-08-17 | 陈巍 | Data migration method based on JDOM revolving technology |
| CN110019140A (en) * | 2017-12-29 | 2019-07-16 | 华为技术有限公司 | Data migration method, device, equipment and computer readable storage medium |
| CN110275914A (en) * | 2019-04-26 | 2019-09-24 | 武汉众邦银行股份有限公司 | Data migration method, device and the storage medium of integration across database |
| CN112749151A (en) * | 2021-01-13 | 2021-05-04 | 叮当快药科技集团有限公司 | Data migration method and device among different types of databases and storage medium |
-
2021
- 2021-06-01 CN CN202110607745.4A patent/CN113297180A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102156699A (en) * | 2010-02-11 | 2011-08-17 | 陈巍 | Data migration method based on JDOM revolving technology |
| CN110019140A (en) * | 2017-12-29 | 2019-07-16 | 华为技术有限公司 | Data migration method, device, equipment and computer readable storage medium |
| CN110275914A (en) * | 2019-04-26 | 2019-09-24 | 武汉众邦银行股份有限公司 | Data migration method, device and the storage medium of integration across database |
| CN112749151A (en) * | 2021-01-13 | 2021-05-04 | 叮当快药科技集团有限公司 | Data migration method and device among different types of databases and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114553958A (en) * | 2022-01-04 | 2022-05-27 | 深圳绿米联创科技有限公司 | Data migration processing method and device, electronic equipment and storage medium |
| CN114553958B (en) * | 2022-01-04 | 2024-07-05 | 深圳绿米联创科技有限公司 | Data migration processing method and device, electronic equipment and storage medium |
| CN115292289A (en) * | 2022-08-09 | 2022-11-04 | 华润数字科技有限公司 | Data filtering method, device, equipment and medium based on database migration |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112948427B (en) | Data query method, device, equipment and storage medium | |
| CN111538594B (en) | Order ID generation method, device, equipment and medium based on distributed system | |
| CN113297180A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN113239106A (en) | Excel file export method and device, electronic equipment and storage medium | |
| CN112685117A (en) | System language internationalization maintenance method, device and computer readable storage medium | |
| CN114185895A (en) | Data import and export method and device, electronic equipment and storage medium | |
| CN111538573A (en) | Asynchronous task processing method and device and computer readable storage medium | |
| CN114398194A (en) | Data collection method and device, electronic equipment and readable storage medium | |
| CN112580079A (en) | Authority configuration method and device, electronic equipment and readable storage medium | |
| CN113468175A (en) | Data compression method and device, electronic equipment and storage medium | |
| CN112685384A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN112948380A (en) | Data storage method and device based on big data, electronic equipment and storage medium | |
| CN114895942A (en) | Application skin changing method, device, equipment and storage medium | |
| CN114911479A (en) | Interface generation method, device, equipment and storage medium based on configuration | |
| CN114510400A (en) | Task execution method and device, electronic equipment and storage medium | |
| CN114625376A (en) | Data transfer mode conversion method and device, electronic equipment and readable storage medium | |
| CN114003661A (en) | Offline data entry method and device, electronic equipment and storage medium | |
| WO2022105494A1 (en) | Automatic data backfilling method and apparatus, electronic device, and computer storage medium | |
| CN113076069A (en) | Double-cache method and device, electronic equipment and computer readable storage medium | |
| CN113254446A (en) | Data fusion method and device, electronic equipment and medium | |
| CN113485711A (en) | Task triggering method and device based on dependency relationship, electronic equipment and medium | |
| CN114185522A (en) | Page theme customizing method and device, electronic equipment and storage medium | |
| CN112527443A (en) | Prompt box display method and device, electronic equipment and computer readable storage medium | |
| CN112686759A (en) | Account checking monitoring method, device, equipment and medium | |
| CN112328656A (en) | Service query method, device, equipment and storage medium based on middle platform architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210824 |