CN113849482A - Data migration method and device and electronic equipment - Google Patents
Data migration method and device and electronic equipment Download PDFInfo
- Publication number
- CN113849482A CN113849482A CN202111128570.5A CN202111128570A CN113849482A CN 113849482 A CN113849482 A CN 113849482A CN 202111128570 A CN202111128570 A CN 202111128570A CN 113849482 A CN113849482 A CN 113849482A
- Authority
- CN
- China
- Prior art keywords
- data
- piece
- target data
- database
- table partition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data migration method, a device and electronic equipment, which relate to the field of data storage, and the method is applied to a data migration tool and comprises the following steps: reading each piece of target data to be subjected to data migration in a centralized database; obtaining table partition metadata of each piece of target data; the table partition metadata of any target data is used for representing the table partition in which the target data is located in the centralized database; and storing the table partition metadata of each piece of target data into a distributed database, and storing each piece of target data into the distributed database in a balanced storage mode. Compared with the prior art, the scheme provided by the embodiment of the invention can realize the advantages of distributed storage of the distributed database when data migration is carried out from the centralized database to the distributed database, and can ensure effective reverse migration after the data migration.
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a data migration method and apparatus, and an electronic device.
Background
Currently, for reasons such as data access efficiency, data stored in a centralized database is generally migrated to a distributed database, and then data access is responded based on the distributed database.
In the related art, during data migration, the table partitioning characteristic of the centralized database is often lost or modified, so that it is difficult to recover the table partitioning characteristic of the centralized database during the reverse migration process of migrating the entry index data from the distributed database back to the centralized database.
Therefore, when data migration is performed from a centralized database to a distributed database, how to consider the advantages of distributed storage of the distributed database and ensure that the data can be effectively migrated in a reverse direction after the data migration is a problem to be solved urgently.
Disclosure of Invention
The embodiment of the invention aims to provide a data migration method, a data migration device and electronic equipment, so that the advantages of distributed storage of a distributed database are taken into consideration when data migration is performed from a centralized database to the distributed database, and effective reverse migration after the data migration can be guaranteed. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a data migration method, which is applied to a data migration tool, and the method includes:
reading each piece of target data to be subjected to data migration in a centralized database;
obtaining table partition metadata of each piece of target data; the table partition metadata of any target data is used for representing the table partition in which the target data is located in the centralized database;
and storing the table partition metadata of each piece of target data into a distributed database, and storing each piece of target data into the distributed database in a balanced storage mode.
Optionally, in a specific implementation manner, the method further includes:
reading each item label data and table partition metadata of each target data from the distributed database;
storing each piece of target data into a target table partition in the centralized database; wherein the target table partition is: the table partition characterized by the table partition metadata for each piece of target data.
Optionally, in a specific implementation manner, the step of storing each piece of target data in the distributed database in a balanced storage manner includes:
for each piece of target data, determining a preset key value in the entry mark data, and performing hash transformation on the preset key value to obtain a target hash value;
and determining a database instance for storing the target data in the distributed database based on the target hash value, and storing the entry mark data into the determined database instance.
Optionally, in a specific implementation manner, the step of obtaining table partition metadata of each piece of target data includes:
when reading each piece of target data, the storage address of the entry tag data is acquired, and the table partition metadata of the entry tag data is determined based on the determined storage address.
Optionally, in a specific implementation manner, a database instance in the distributed database, which is used to store the table partition metadata of each piece of target data, is different from a database instance used to store each piece of target data.
In a second aspect, an embodiment of the present invention provides a data migration apparatus, which is applied to a data migration tool, and the apparatus includes:
the first data reading module is used for reading each piece of target data to be subjected to data migration in the centralized database;
the metadata acquisition module is used for acquiring the table partition metadata of each piece of target data; the table partition metadata of any target data is used for representing the table partition in which the target data is located in the centralized database;
and the data storage module is used for storing the table partition metadata of each piece of target data into the distributed database and storing each piece of target data into the distributed database in a balanced storage mode.
Optionally, in a specific implementation manner, the apparatus further includes:
the second data reading module is used for reading each item label data and the table partition metadata of each target data from the distributed database;
the data recovery module is used for storing each piece of target data into a target table partition in the centralized database; wherein the target table partition is: the table partition characterized by the table partition metadata for each piece of target data.
Optionally, in a specific implementation manner, the data storage module is specifically configured to:
for each piece of target data, determining a preset key value in the entry mark data, and performing hash transformation on the preset key value to obtain a target hash value; and determining a database instance for storing the target data in the distributed database based on the target hash value, and storing the entry mark data into the determined database instance.
Optionally, in a specific implementation manner, the metadata obtaining module is specifically configured to:
when reading each piece of target data, the storage address of the entry tag data is acquired, and the table partition metadata of the entry tag data is determined based on the determined storage address.
Optionally, in a specific implementation manner, a database instance in the distributed database, which is used to store the table partition metadata of each piece of target data, is different from a database instance used to store each piece of target data.
In a third aspect, an embodiment of the present invention provides an electronic device, which is provided with a data migration tool, and includes a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;
a memory for storing a computer program;
a processor, configured to implement the steps of any of the data migration methods provided in the first aspect when executing the program stored in the memory.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements the steps of any one of the data migration methods provided in the first aspect.
In a fifth aspect, an embodiment of the present invention provides a computer program product containing instructions, which when run on a computer, causes the computer to perform the steps of any of the data migration methods provided in the first aspect.
The embodiment of the invention has the following beneficial effects:
as can be seen from the above, with the adoption of the scheme provided by the embodiment of the present invention, when data stored in the centralized database is migrated to the distributed database, the data migration tool can read each piece of target data to be subjected to data migration in the centralized database, and obtain table partition metadata of each piece of target data. Furthermore, the data migration tool can store the read entry mark data into the distributed database in a load balancing mode, and can also store the table partition metadata of each target data into the distributed database.
In this way, in the reverse migration process of migrating each entry mark data from the distributed database back to the centralized database, the data migration tool may read the table partition metadata of each target data in the distributed database, so as to migrate each target data back to the partition indicated by the table partition metadata of the target data in the centralized database, thereby implementing effective reverse migration after data migration.
Based on the scheme provided by the embodiment of the invention, the distributed storage advantage of the distributed database can be taken into consideration when data migration is carried out from the centralized database to the distributed database, and the effective reverse migration after the data migration can be ensured.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by referring to these drawings.
Fig. 1 is a schematic diagram of an embodiment of a data migration method according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a data migration method according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating another data migration method according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived from the embodiments given herein by one of ordinary skill in the art, are within the scope of the invention.
In the related art, during data migration, the table partitioning characteristic of the centralized database is often lost or modified, so that it is difficult to recover the table partitioning characteristic of the centralized database during the reverse migration process of migrating the entry index data from the distributed database back to the centralized database. Therefore, how to consider the advantages of distributed storage of the distributed database when data migration is performed from the centralized database to the distributed database, and ensure that the data can be effectively migrated in a reverse direction after the data migration, is a problem to be solved urgently.
In order to solve the above technical problem, an embodiment of the present invention provides a data migration method.
The method is suitable for various application scenarios in which data needs to be migrated from the centralized database to the distributed database, for example, enterprise machine room relocation, physical machine room-to-cloud platform conversion, and the like, but is not limited thereto.
The method is applied to a Data Transfer Service (DTS) tool, wherein the DTS tool is a Data conversion Service and can realize the movement of Data from one storage position to another storage position. In a specific application, the data migration tool for executing the method is a data migration tool preset in various electronic devices such as a notebook computer and a desktop computer, and of course, the data migration tool for executing the method may also be a data migration tool preset in a device cluster.
The data migration method provided by the embodiment of the invention can comprise the following steps:
reading each piece of target data to be subjected to data migration in a centralized database;
obtaining table partition metadata of each piece of target data; the table partition metadata of any target data is used for representing the table partition in which the target data is located in the centralized database;
and storing the table partition metadata of each piece of target data into a distributed database, and storing each piece of target data into the distributed database in a balanced storage mode.
As can be seen from the above, with the adoption of the scheme provided by the embodiment of the present invention, when data stored in the centralized database is migrated to the distributed database, the data migration tool can read each piece of target data to be subjected to data migration in the centralized database, and obtain table partition metadata of each piece of target data. Furthermore, the data migration tool can store the read entry mark data into the distributed database in a load balancing mode, and can also store the table partition metadata of each target data into the distributed database.
In this way, in the reverse migration process of migrating each entry mark data from the distributed database back to the centralized database, the data migration tool may read the table partition metadata of each target data in the distributed database, so as to migrate each target data back to the partition indicated by the table partition metadata of the target data in the centralized database, thereby implementing effective reverse migration after data migration.
Based on the scheme provided by the embodiment of the invention, the distributed storage advantage of the distributed database can be taken into consideration when the data is migrated from the centralized database to the distributed database, and the effective reverse migration after the data migration can be ensured.
Before a data migration method provided by the embodiment of the present invention is specifically described, a storage manner of data in a centralized database and a distributed database is briefly described.
The data stored in the centralized database are stored in the same data storage space, so that the centralized database externally presents a uniform data logic table, that is, when a user responds to data access based on the centralized database, the user views the uniform data logic table. However, inside the data storage space of the centralized database, the data storage space is divided into a plurality of data partitions, i.e. into a plurality of table partitions, so that the pieces of data belonging to the same logical table of data are actually stored in the respective table partitions in the data storage space of the centralized database.
For convenience of description, each table partition in the data storage space of the centralized database is simply referred to as each table partition of the centralized database.
That is, in the centralized database, various pieces of data are stored in various table partitions of the centralized database, and the centralized database externally presents a unified logical table of data.
For example, as shown in fig. 1, a table partition may also be referred to as a table space partition, and the TableA is a unified data logical table presented to the outside by the centralized database, and each piece of data belonging to the TableA is stored in each table partition space such as table partition space 1 and table partition space 2 of the centralized database.
Accordingly, the distributed database has a plurality of independent database instances, each piece of data is stored in each different database instance, and the data stored in each database instance is stored according to the designated data sub-table, that is, each piece of data stored in the same database instance is stored in each different sub-table in the database instance. Therefore, each database instance presents the data sub-tables to which the stored data belongs and the data logic table to which each sub-table belongs separately, wherein the data sub-tables belonging to the same data logic table can be located in different database instances.
For example, as shown in fig. 1, TableA is a data logic table, sub-table X and sub-table Y are data sub-tables belonging to TableA in database instance 1(DB1), and sub-table M and sub-table N are data sub-tables belonging to TableA in database instance N (dbn). Among them, DB is a shorthand for Database.
Alternatively, the distributed database may generally be a distributed data database system including a plurality of data storage nodes, where each data storage node is a respective database instance.
Generally, a node may be set in the distributed database as a management node of the distributed database, where the management node may be a separate node separated from each database instance, or may be a database instance in each data storage node, and then the database instance may be used for managing the distributed database system and also may be used for data storage.
For example, as shown in fig. 1, the distributed database is a distributed database system, the DBs 1-DBN are N data storage nodes, and a Global Management node (GMS) is a Management node.
A data migration method provided in an embodiment of the present invention is specifically described below with reference to the accompanying drawings.
Fig. 2 is a schematic flowchart of a data migration method according to an embodiment of the present invention, and as shown in fig. 2, the method may include the following steps S201 to S203.
S201: reading each piece of target data to be subjected to data migration in a centralized database;
when migrating data stored in a centralized database to a distributed database, a data migration tool may first determine each piece of target data to be subjected to data migration in the centralized database.
For example, the data migration tool may use all data stored in the centralized database as target data to be subjected to data migration; for another example, the data migration tool may receive a data migration instruction sent by a user, where the data migration instruction may indicate target data to be subjected to data migration in the centralized database, and exemplarily indicate that data of which the data type is the target type in the centralized database is the target data to be subjected to data migration, indicate that data stored in the centralized database within the target time range is the target data to be subjected to data migration, and the like. The embodiment of the invention does not specifically limit the way of determining each piece of target data to be subjected to data migration in the centralized database by the data migration tool.
In this way, after determining each piece of target data to be subjected to data migration in the centralized database, the data migration tool may read each piece of target data to be subjected to data migration in the centralized database.
For example, when all data stored in the centralized database are target data to be subjected to data migration, the data migration tool may read each piece of data stored in the centralized database one by one; for another example, in each piece of data stored in the centralized database, each piece of target data to be migrated indicated by the data migration instruction is screened, and thus each piece of screened data is read.
Based on this, the embodiment of the present invention does not limit the specific implementation manner of step S201.
S202: obtaining table partition metadata of each piece of target data;
the table partition metadata of any target data is used for representing the table partition in which the target data is located in the centralized database;
since in the centralized database, each piece of data may be stored in a different table partition of the centralized database, when each piece of target data is read, the data migration tool may obtain the table partition metadata of the piece of target data accordingly.
The table partition metadata of any target data is used for representing the table partition in which the target data is located in the centralized database, so that the data migration tool can determine the table partition in which each target data is located in the centralized database by reading the table partition metadata of each target data.
Optionally, in a specific implementation manner, the table partition metadata of each piece of target data may be recorded in the centralized database, so that when the data migration tool reads each piece of target data, the table partition metadata of the piece of target data recorded in the centralized database may be read therewith.
Optionally, in another specific implementation manner, each table partition in the centralized database occupies a part of a storage space in the centralized database, so that a corresponding relationship exists between each table partition in the centralized database and a storage address of the data storage space in the centralized database, that is, each table partition in the centralized database corresponds to one address range of all storage addresses of the data storage space in the centralized database, and thus, for each data stored in the centralized database, the table partition in which the data is located may be determined according to the address range in which the storage address of the data is located.
Based on this, in this specific implementation manner, the step S202 may include the following step 11:
step 11: when reading each piece of target data, the storage address of the entry tag data is acquired, and the table partition metadata of the entry tag data is determined based on the determined storage address.
In this specific implementation manner, when each piece of target data is read, the data migration tool may obtain the storage address of the entry mark data, so that the table partition where the entry mark data is located may be determined based on the storage address of the piece of target data, and further determine the table partition metadata of the piece of target data.
S203: and storing the table partition metadata of each piece of target data into a distributed database, and storing each piece of target data into the distributed database in a balanced storage mode.
After the target data and the table partition metadata of the target data are obtained, the data migration tool can store the table partition metadata of the target data into the distributed database, and store the target data into the distributed database in a balanced storage mode.
The distributed database comprises a plurality of database instances, so that the data migration tool can store the acquired metadata of each table partition in a certain database instance and store the data of each entry mark in each database instance.
Thus, the table partition metadata and at least one piece of target data can be stored on the same database instance, and can also be not stored on the same database instance with each piece of entry mark data.
Based on this, optionally, in a specific implementation manner, the database instance in the distributed database for storing the table partition metadata of each piece of target data is different from the database instance for storing each piece of target data.
In addition, optionally, when there is a separate management node separate from each database instance in the distributed database, in order to not make the above-mentioned each table partitioning data occupy the storage space in each database instance in the distributed database, so as to achieve storage balance between each database instance, each table partitioning metadata may be stored in the management node, so that each entry mark data is stored in each database instance.
Optionally, in a specific implementation manner, in the step S203, storing each piece of target data in the distributed database in a balanced storage manner may include the following step 21:
step 21: determining a preset key value in the entry mark data aiming at each piece of target data, and carrying out hash transformation on the preset key value to obtain a target hash value; and determining a database instance for storing the target data in the distributed database based on the target hash value, and storing the entry mark data into the determined database instance.
In this specific implementation manner, when storing each piece of target data, the data migration tool may determine a preset key value in the piece of entry mark data, and further may perform Hash (Hash) transformation on the preset key value to obtain a target Hash value, so that a database instance for storing the piece of target data may be determined in the distributed database based on the target Hash value, and the piece of entry mark data is stored in the determined database instance.
Optionally, a remainder obtained by dividing the target hash value by the database instance in the distributed database may be calculated, so that the database instance for storing the target data is determined in the distributed database according to the remainder.
It should be noted that, the above step 21 is only an example of the implementation of the above balanced storage, and is not limited, and any manner capable of implementing balanced storage of each database instance in the distributed database is within the scope of the present invention.
Optionally, the data migration tool may store the table partition metadata of each piece of target data into the distributed database after reading each piece of target data and acquiring the table partition metadata of the piece of target data, and store the piece of target data into the distributed database in a balanced storage manner. Thus, the data migration tool may store the target data and the table partition metadata of the target data item by item.
Optionally, the data migration tool may store all the acquired table partition metadata into the distributed database at one time after acquiring all the target data and the table partition metadata of each target data, and store all the target data into the distributed database at one time according to a balanced storage manner.
Based on the scheme provided by the embodiment of the invention, the distributed storage advantage of the distributed database can be taken into consideration when data migration is carried out from the centralized database to the distributed database, and effective reverse migration after the data migration can be ensured.
Each target data is stored in the distributed database in a balanced storage mode, each database instance in the distributed database can receive data writing tasks with balanced quantity, the quantity of the data stored in each database instance can meet the requirement of balanced data quantity, and each database instance can respond to data access requests with balanced quantity when responding to data access based on the distributed database, so that the data access efficiency of the distributed database is guaranteed.
In each of the above specific implementation manners, after the data of each entry mark to be subjected to data migration in the centralized database is stored in the distributed database, the centralized database deletes the stored data of each entry mark, that is, after the target data is migrated from the centralized database to the distributed database, the centralized database does not store any target data.
In some practical applications, the target data needs to be migrated back from the distributed database to the centralized database, and for each target data, after the migration back, the entry mark data is located in the table partition in the centralized database, which is the same as the table partition in the centralized database where the target data is located before the data migration.
Based on this, optionally, in a specific implementation manner, as shown in fig. 3, the data migration method provided in the embodiment of the present invention may further include the following steps S204 to S205.
S204: reading each item label data and table partition metadata of each target data from a distributed database;
s205: storing each piece of target data into a target table partition in a centralized database;
wherein, the target table partition is: the table partition characterized by the table partition metadata for each piece of target data.
In this specific implementation manner, the table partition metadata of any target data is used to characterize the table partition in which the target data is located in the centralized database, and the distributed database stores the table partition metadata of each target data, where the table partition metadata of each target data stored in the distributed database characterizes the table partition in which the target data is located in the centralized database before data migration is performed, so that the table partition metadata can characterize the target table partition in the centralized database to which the entry index data is to be stored when data is migrated back at this time.
In this way, when target data is migrated back, the data migration tool may read each entry index data and the table partition metadata of each piece of target data from the distributed database, and further, for each piece of target data, the data migration tool may determine, according to the table partition metadata of the target data, a target table partition where the entry index data is located in the centralized database, so as to store the entry index data into the target table partition in the centralized database.
Based on this, after all the target data are stored in the target table partition in the centralized database, the reverse migration of each piece of target data is completed, and for each piece of target data, after the migration, the table partition in which the entry mark data is located in the centralized database is the same as the table partition in which the piece of target data is located in the centralized database before the data migration, that is, effective reverse migration is realized.
And after all the target data are stored in the target table partition in the centralized database, the distributed database does not store the target data and the table partition metadata of the target data any more.
Therefore, the distributed storage advantage of the distributed database can be taken into consideration when data migration is carried out from the centralized database to the distributed database, and effective reverse migration after the data migration can be guaranteed.
In order to facilitate understanding of the data migration method provided by the embodiment of the present invention, the data migration method provided by the embodiment of the present invention is described below by using a specific example shown in fig. 1.
As shown in fig. 1, TableA is stored in the centralized database, and each piece of data in TableA is stored in a plurality of tablespace partitions such as a centralized and distributed tablespace partition 1 and a tablespace partition 2.
In the forward migration process of migrating data in the centralized database to the distributed database, the data migration tool reads each piece of data in TableA from the centralized database and obtains table partition metadata of each piece of data. Furthermore, the data migration tool stores each piece of data in the TableA read to the sub-tables X to N in each database instance DB1-DBN in the distributed database in a balanced storage manner, and stores each table partition metadata to the global management unit in the distributed database.
After the forward migration is completed, the centralized database stores no more pieces of data in TableA.
In a reverse migration process of migrating data in the distributed database back to the centralized database, the data migration tool reads each piece of data in TableA from the sub-table X to the sub-table N in each database instance DB1-DBN in the distributed database, and reads table partition metadata of each piece of data from the global management unit. Further, for each piece of data, a table partition in which the piece of data is located in the centralized database is determined according to the table partition metadata of the piece of data, and thus, the piece of data is stored into the determined table partition in the centralized database.
After the reverse migration is completed, the distributed database no longer stores the data in TableA, and the table partition in which the data in TableA is located in the centralized database is the same as the table partition in which the data in TableA is located before the forward migration is performed.
Corresponding to the data migration method provided by the embodiment of the invention, the embodiment of the invention also provides a data migration device, and the device is applied to a data migration tool.
Fig. 4 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present invention, and as shown in fig. 4, the apparatus may include the following modules:
a first data reading module 410, configured to read each piece of target data to be subjected to data migration in the centralized database;
a metadata obtaining module 420, configured to obtain table partition metadata of each piece of target data; the table partition metadata of any target data is used for representing the table partition in which the target data is located in the centralized database;
the data storage module 430 is configured to store the table partition metadata of each piece of target data in the distributed database, and store each piece of target data in the distributed database in a balanced storage manner.
Based on the scheme provided by the embodiment of the invention, the distributed storage advantage of the distributed database can be taken into consideration when data migration is carried out from the centralized database to the distributed database, and effective reverse migration after the data migration can be ensured.
Optionally, in a specific implementation manner, the apparatus further includes:
the second data reading module is used for reading each item label data and the table partition metadata of each target data from the distributed database;
the data recovery module is used for storing each piece of target data into a target table partition in the centralized database; wherein the target table partition is: the table partition characterized by the table partition metadata for each piece of target data.
Optionally, in a specific implementation manner, the data storage module 430 is specifically configured to:
for each piece of target data, determining a preset key value in the entry mark data, and performing hash transformation on the preset key value to obtain a target hash value; and determining a database instance for storing the target data in the distributed database based on the target hash value, and storing the entry mark data into the determined database instance.
Optionally, in a specific implementation manner, the metadata obtaining module 410 is specifically configured to:
when reading each piece of target data, the storage address of the entry tag data is acquired, and the table partition metadata of the entry tag data is determined based on the determined storage address.
Optionally, in a specific implementation manner, a database instance in the distributed database, which is used to store the table partition metadata of each piece of target data, is different from a database instance used to store each piece of target data.
Corresponding to the data migration method provided by the above embodiment of the present invention, an embodiment of the present invention further provides an electronic device, as shown in fig. 5, which is provided with a data migration tool, including a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 complete mutual communication through the communication bus 504,
a memory 503 for storing a computer program;
the processor 501 is configured to implement the steps of any data migration method provided in the above embodiments of the present invention when executing the program stored in the memory 503.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In another embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program, when executed by a processor, implements the steps of any of the data migration methods provided in the above embodiments of the present invention.
In another embodiment, the present invention further provides a computer program product containing instructions, which when run on a computer, causes the computer to perform the steps of any of the data migration methods provided in the above embodiments of the present invention.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, apparatus embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments are described with relative simplicity as they are substantially similar to method embodiments, where relevant only as described in portions of the method embodiments.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (12)
1. A data migration method, applied to a data migration tool, the method comprising:
reading each piece of target data to be subjected to data migration in a centralized database;
obtaining table partition metadata of each piece of target data; the table partition metadata of any target data is used for representing the table partition in which the target data is located in the centralized database;
and storing the table partition metadata of each piece of target data into a distributed database, and storing each piece of target data into the distributed database in a balanced storage mode.
2. The method of claim 1, further comprising:
reading each item label data and table partition metadata of each target data from the distributed database;
storing each piece of target data into a target table partition in the centralized database; wherein the target table partition is: the table partition characterized by the table partition metadata for each piece of target data.
3. The method of claim 1, wherein the step of storing each piece of target data in the distributed database in a balanced storage manner comprises:
for each piece of target data, determining a preset key value in the entry mark data, and performing hash transformation on the preset key value to obtain a target hash value;
and determining a database instance for storing the target data in the distributed database based on the target hash value, and storing the entry mark data into the determined database instance.
4. The method of claim 1, wherein the step of obtaining table partition metadata for each piece of target data comprises:
when reading each piece of target data, the storage address of the entry tag data is acquired, and the table partition metadata of the entry tag data is determined based on the determined storage address.
5. The method according to any one of claims 1 to 4, wherein the database instance for storing the table partition metadata of each piece of target data in the distributed database is different from the database instance for storing each piece of target data.
6. A data migration apparatus, applied to a data migration tool, the apparatus comprising:
the first data reading module is used for reading each piece of target data to be subjected to data migration in the centralized database;
the metadata acquisition module is used for acquiring the table partition metadata of each piece of target data; the table partition metadata of any target data is used for representing the table partition in which the target data is located in the centralized database;
and the data storage module is used for storing the table partition metadata of each piece of target data into the distributed database and storing each piece of target data into the distributed database in a balanced storage mode.
7. The apparatus of claim 6, further comprising:
the second data reading module is used for reading each item label data and the table partition metadata of each target data from the distributed database;
the data recovery module is used for storing each piece of target data into a target table partition in the centralized database; wherein the target table partition is: the table partition characterized by the table partition metadata for each piece of target data.
8. The apparatus of claim 6, wherein the data storage module is specifically configured to:
for each piece of target data, determining a preset key value in the entry mark data, and performing hash transformation on the preset key value to obtain a target hash value; and determining a database instance for storing the target data in the distributed database based on the target hash value, and storing the entry mark data into the determined database instance.
9. The apparatus of claim 6, wherein the metadata acquisition module is specifically configured to:
when reading each piece of target data, the storage address of the entry tag data is acquired, and the table partition metadata of the entry tag data is determined based on the determined storage address.
10. The apparatus according to any one of claims 6-9, wherein the database instance for storing the table partition metadata of each piece of target data in the distributed database is different from the database instance for storing each piece of target data.
11. The electronic equipment is characterized by being provided with a data migration tool, wherein the data migration tool comprises a processor, a communication interface, a memory and a communication bus, and the processor, the communication interface and the memory are communicated with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 5 when executing a program stored in the memory.
12. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of the claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111128570.5A CN113849482A (en) | 2021-09-26 | 2021-09-26 | Data migration method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111128570.5A CN113849482A (en) | 2021-09-26 | 2021-09-26 | Data migration method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113849482A true CN113849482A (en) | 2021-12-28 |
Family
ID=78979504
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111128570.5A Pending CN113849482A (en) | 2021-09-26 | 2021-09-26 | Data migration method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113849482A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115328924A (en) * | 2022-10-17 | 2022-11-11 | 北京奥星贝斯科技有限公司 | Method and device for creating table space, storage medium and electronic equipment |
-
2021
- 2021-09-26 CN CN202111128570.5A patent/CN113849482A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115328924A (en) * | 2022-10-17 | 2022-11-11 | 北京奥星贝斯科技有限公司 | Method and device for creating table space, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10296498B2 (en) | Coordinated hash table indexes to facilitate reducing database reconfiguration time | |
| US11030247B2 (en) | Layered graph data structure | |
| US10157214B1 (en) | Process for data migration between document stores | |
| CN107103011B (en) | Method and device for realizing terminal data search | |
| US9514170B1 (en) | Priority queue using two differently-indexed single-index tables | |
| CN113448938A (en) | Data processing method and device, electronic equipment and storage medium | |
| US11544229B1 (en) | Enhanced tracking of data flows | |
| CN102959548A (en) | Data storage method, search method and device | |
| US9111114B1 (en) | Method of transforming database system privileges to object privileges | |
| CN113849482A (en) | Data migration method and device and electronic equipment | |
| US7487172B2 (en) | Three-dimensional data structure for storing data of multiple domains and the management thereof | |
| US10521398B1 (en) | Tracking version families in a file system | |
| CN110837499B (en) | Data access processing method, device, electronic equipment and storage medium | |
| CN109542860B (en) | Service data management method based on HDFS and terminal equipment | |
| CN111522626A (en) | Virtual machine list generation method and device and electronic equipment | |
| US12013861B2 (en) | Method and apparatus for retrieving and enumerating object metadata in distributed storage system | |
| CN111209304B (en) | Data processing method, device and system | |
| US10114864B1 (en) | List element query support and processing | |
| CN113760854A (en) | Method for identifying data in HDFS memory and related equipment | |
| CN113392067A (en) | Data processing method, device and system for distributed database | |
| CN110968267A (en) | Data management method, device, server and system | |
| CN115718571B (en) | Data management method and device based on multidimensional features | |
| CN114328755B (en) | Data writing method, data reading device and electronic equipment | |
| CN114661249B (en) | Data storage method and device, computer equipment and storage medium | |
| US11451627B2 (en) | System and method for content management with intelligent data store access across distributed stores |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |